[uWSGI] Flat file caching in subdirectories

Roberto De Ioris roberto at unbit.it
Sun Jul 10 16:43:49 CEST 2016


> Hi folks
>
> I have a caching/router based question.
>
> I need to cache a site with a large number of pages (think 20 million+).
> These pages are generated by a Python application with a fair amount of
> database calls, data crunching and template rendering, hence the need to
> cache the responses.
>
> First I looked into uWSGI's Cache2 setup, which is awesome but "only"
> memory based. With a potential of 20 million pages at ~30 Kb each ... well
> I do not have that kind of memory :)
>
> File based caching it is. As far as I could find in the docs Cache2 does
> not support dumping responses into files, only into memory (tell me if I'm
> wrong here...). The only way to accomplish file based caching is through
> the router and the "transformation_tofile" plugin as demonstrated in the
> caching cookbook.
>
> I have tested this and it works very well. Not as fast as memory based
> caching, but still a considerable speed boost.
>
> Here comes the crux: If more than 20 million files could potentially be
> dumped into a single cache directory on disk this could wreak havoc on
> filesystem read times in that specific directory.
>
> So here is what I try to do: I would like to create subdirectories inside
> the main cache directory based on the URL path. Each path in the
> application points to a different type of page so I'm looking at the
> following cache setup:
>
> - If the path is: "/path-a/some-title/" the cache on disk
> "/main-cache-dir/path-a/some-title/file.html
>
> In essence this is no problem and can be achieved by the following router
> rule:
>
> route-if = isfile:/disk1/uwsgi_cache/files${PATH_INFO}file.html
> static:/disk1/uwsgi_cache/files${PATH_INFO}file.html
>
> However, uWSGI needs to create this subdirectory structure (and the
> arbitrary named "file.html" file) first. I know uWSGI can do this with the
> "exec-asap" directive:
>
> exec-asap = mkdir -p <some-path>
>
> Yet "exec-asap" is not part of the router setup and does not fire during
> routing nor does it carry the needed routing variables (such as
> ${PATH_INFO}).
>
> Does this sound like a scenario for uWSGI "flat file" caching ... can it
> cache to subdirectories instead of flat all in one?
> Is it possible to create directories (and their parents) from within the
> router?
>
> Cheers,
> Tim
>
> _______________________________________________
> uWSGI mailing list
> uWSGI at lists.unbit.it
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>


Hi, write plugins when performance are the main requirement:

https://gist.github.com/rdeioris/d1ba5b143ce54a0d58233d835fe24653

uwsgi --build-plugin router_mkdir.c

then:

[uwsgi]
; load the plugin
plugin = path_to_router_mkdir.so
; example
route = ^/foo mkdir:/tmp/foo/${SCRIPT_NAME}


-- 
Roberto De Ioris
http://unbit.com


More information about the uWSGI mailing list