[uWSGI] Possible setup race condition between uwsgi-plugin-python and python-flask

Roberto De Ioris roberto at unbit.it
Wed Apr 2 00:17:51 CEST 2014


> Hi guys,
>
> I've been struggling with a weird situation (I have to assume a bug) and
> cannot find any references to this on the Internet. If you could chime in
> with some advice, that'd be wonderful :)
>
> Please note that this whole situation is related to python-flask, and I
> have contacted their mailing list to see if anyone has seen this behavior
> before. They didn't and suggested to see if you guys know more :-/
>
> The problem is that I'm not sure at what point uwsgi-plugin-python starts
> spawning new threads for handling requests. Maybe it's too early and
> python-flask does some optimizations that result in a bug.
>
> Here is the situation: We host a web-service through nginx, serving
> requests to uwsgi-plugin-python (version 1.0.3+dfsg-1ubuntu0.1) to our
> nice
> Python (2.7.3) flask app. Since we had garbage-collection issues with many
> small objects on the heap in the past, we decided to respawn the uwsgi
> service every few k requests and use multiple processes to avoid a
> down-time in the respawn phase (and every process has multiple threads of
> course).
>
> Another important note is that we use the "lazy" option, because we use
> MySQL connections which cannot be forked after Python initialization
> (because all DB connections would share the same connection ID, which is
> not supported by the MySQL client).
>
> Now, it seems like each time the Python process respawns, that flask sets
> up our blueprints (mappings between Python handlers and URLs in case
> you're
> not familiar with flask) and then starts accepting requests, but
> *sometimes* responds with a 404 to clients early during this phase. This
> happens very rarely, but since there is considerable load on the server,
> it
> actually starts adding up so that we need to something about it.
>
> I have spent quite some time debugging the flask blueprints, and my
> findings - so far - show that at app creation time, only one thread is
> running in Python *after* the blueprints have been registered. However,
> when I add code to log details about the blueprints / URL-rules when the
> 404 happens, then the blueprints are not all/fully loaded yet.
>
> It seems to me like new threads are created and start accepting
> connections
> before all blueprints/rules have been fully registered. Here is some code
> to show what I mean:
>
> def create_app(...):
>     app = flask.Flask(__name__)
>     [...]
>
>     @app.after_request
>     def after_request(response):
>         if response.status_code == 404:
>             # DBG2
>             print 'dbg', response.status_code, request.base_url
>             print threading.active_count(), "active threads"
>             print app.url_map.__dict__.get('foo/mytesturl')
>         return response
>
>     app.register_blueprint(mymodel, url_prefix="/foo")
>     # DBG1
>     print 'done loading', app.url_map.__dict__
>     print threading.active_count(), "active threads"
>     return app
>
> While the DBG1 code shows everything correctly, the DBG2 code shows a
> sub-set of my URL-handlers (which is why the 404 gets executed).
>
> BTW, at DBG1 point, the logs usually show "1 active threads", while at
> DBG2
> many more threads are loaded, making me assume there is some sort of
> lazy/on-demand loading of the rules within flask.
>
> Again, the code works, it's just that there is this race where I get 404s.
>
> Is this a known bug in uwsgi / uwsgi-plugin-python? If so, it'd be worth
> investigating on a newer version, or is there a specific way to tell flask
> to fully load all blueprints/rules before accepting connections?
>
> Sorry for the overly long email, I wanted to get all the details right :)
>
> Thanks!
> -Clemens
>
> --
> Clemens Kolbitsch
> Security Researcher
> kolbitsch at lastline.com
> Mobile +1 (206) 356-7745
> Land +1 (805) 456-7076
>
> Lastline, Inc.
> 6950 Hollister Avenue, Suite 101
> Goleta, CA 93117
>
>

Hi Clemens, sorry but uWSGI 1.0 is unsupported since 2012 and know to be
broken (as well as 1.2). Only 1.4.x and 2.0.x are supported and actively
maintained.

Upgrade to a stable version, and if you still have problems feel free to
repost.


-- 
Roberto De Ioris
http://unbit.it


More information about the uWSGI mailing list