[uWSGI] Flat file caching in subdirectories

Tim van der Linden tim at shisaa.jp
Sun Jul 10 10:12:55 CEST 2016


Hi folks

I have a caching/router based question.

I need to cache a site with a large number of pages (think 20 million+). These pages are generated by a Python application with a fair amount of database calls, data crunching and template rendering, hence the need to cache the responses.

First I looked into uWSGI's Cache2 setup, which is awesome but "only" memory based. With a potential of 20 million pages at ~30 Kb each ... well I do not have that kind of memory :)

File based caching it is. As far as I could find in the docs Cache2 does not support dumping responses into files, only into memory (tell me if I'm wrong here...). The only way to accomplish file based caching is through the router and the "transformation_tofile" plugin as demonstrated in the caching cookbook.

I have tested this and it works very well. Not as fast as memory based caching, but still a considerable speed boost.

Here comes the crux: If more than 20 million files could potentially be dumped into a single cache directory on disk this could wreak havoc on filesystem read times in that specific directory.

So here is what I try to do: I would like to create subdirectories inside the main cache directory based on the URL path. Each path in the application points to a different type of page so I'm looking at the following cache setup:

- If the path is: "/path-a/some-title/" the cache on disk "/main-cache-dir/path-a/some-title/file.html

In essence this is no problem and can be achieved by the following router rule:

route-if = isfile:/disk1/uwsgi_cache/files${PATH_INFO}file.html static:/disk1/uwsgi_cache/files${PATH_INFO}file.html

However, uWSGI needs to create this subdirectory structure (and the arbitrary named "file.html" file) first. I know uWSGI can do this with the "exec-asap" directive:

exec-asap = mkdir -p <some-path>

Yet "exec-asap" is not part of the router setup and does not fire during routing nor does it carry the needed routing variables (such as ${PATH_INFO}).

Does this sound like a scenario for uWSGI "flat file" caching ... can it cache to subdirectories instead of flat all in one?
Is it possible to create directories (and their parents) from within the router?

Cheers,
Tim



More information about the uWSGI mailing list