[uWSGI] Flat file caching in subdirectories
Tim van der Linden
tim at shisaa.jp
Mon Jul 11 04:46:13 CEST 2016
On Mon, 11 Jul 2016 11:02:44 +1000
Curtis Maloney <curtis at tinbrain.net> wrote:
> Two things come to mind to contribute to this discussion.
Thanks for contributing :)
> 1. Early on, you talk of how many millions of pages you have...
> typically, caching is the tradeoff of only holding the most in-demand
> pages, not all of them... so don't worry if you can't manage to store
> all of them.
Have been thinking about this as well, but I still would like to try and "cache them all" in this first iteration. If not all should be cached a memory based approach with Cache2 would be the way to go, it has all the bells and whistles I need.
> 2. Most file systems don't handle a large number of file in one dir
> efficiently [some use btrees or other indices, but still], so using a
> tree of directories based on either (a) a fixed prefix of the
> key/filename or (b) a hash of that, can often significantly speed things.
Correct, that is also what I try to do with my approach: using the applications URI as a directory structure to split out files over various directories *and* to make it easier to locate a certain page within the cache.
> And another thought occurs... if you compress the content as you write
> it, uwsgi may be able to ship if compressed for you without any work.
> In fact, if you cache into the right place you can have static-serve do
> all the lookup for you [but not expiration...]
Compressing did not come to mind yet (Nginx is in front of this setup and does Gzipping at the moment), but can make for a nice addition later.
One thing that you made me notice, however, was expiration. Caching to static files is all fine and fast, but with the current approach (static:) I have no way of expiring a page after a set time ... have I?
More information about the uWSGI