[uWSGI] Flat file caching in subdirectories

John Burk jburk at pipelinefx.com
Mon Jul 11 02:44:44 CEST 2016

Not a direct answer to your question, but another thing to avoid; with that
many files, be careful that your filesystem doesn't run out of inodes.  Had
that happen in scenarios with many many small tables in MySQL.

On Sunday, July 10, 2016, Tim van der Linden <tim at shisaa.jp> wrote:

> Hi folks
> I have a caching/router based question.
> I need to cache a site with a large number of pages (think 20 million+).
> These pages are generated by a Python application with a fair amount of
> database calls, data crunching and template rendering, hence the need to
> cache the responses.
> First I looked into uWSGI's Cache2 setup, which is awesome but "only"
> memory based. With a potential of 20 million pages at ~30 Kb each ... well
> I do not have that kind of memory :)
> File based caching it is. As far as I could find in the docs Cache2 does
> not support dumping responses into files, only into memory (tell me if I'm
> wrong here...). The only way to accomplish file based caching is through
> the router and the "transformation_tofile" plugin as demonstrated in the
> caching cookbook.
> I have tested this and it works very well. Not as fast as memory based
> caching, but still a considerable speed boost.
> Here comes the crux: If more than 20 million files could potentially be
> dumped into a single cache directory on disk this could wreak havoc on
> filesystem read times in that specific directory.
> So here is what I try to do: I would like to create subdirectories inside
> the main cache directory based on the URL path. Each path in the
> application points to a different type of page so I'm looking at the
> following cache setup:
> - If the path is: "/path-a/some-title/" the cache on disk
> "/main-cache-dir/path-a/some-title/file.html
> In essence this is no problem and can be achieved by the following router
> rule:
> route-if = isfile:/disk1/uwsgi_cache/files${PATH_INFO}file.html
> static:/disk1/uwsgi_cache/files${PATH_INFO}file.html
> However, uWSGI needs to create this subdirectory structure (and the
> arbitrary named "file.html" file) first. I know uWSGI can do this with the
> "exec-asap" directive:
> exec-asap = mkdir -p <some-path>
> Yet "exec-asap" is not part of the router setup and does not fire during
> routing nor does it carry the needed routing variables (such as
> ${PATH_INFO}).
> Does this sound like a scenario for uWSGI "flat file" caching ... can it
> cache to subdirectories instead of flat all in one?
> Is it possible to create directories (and their parents) from within the
> router?
> Cheers,
> Tim
> _______________________________________________
> uWSGI mailing list
> uWSGI at lists.unbit.it <javascript:;>
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi



John Burk
jburk at pipelinefx.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.unbit.it/pipermail/uwsgi/attachments/20160710/b0a5715b/attachment.html>

More information about the uWSGI mailing list