[uWSGI] Possible setup race condition between uwsgi-plugin-python and python-flask

Clemens Kolbitsch kolbitsch at lastline.com
Tue Apr 1 20:16:42 CEST 2014


Hi guys,

I've been struggling with a weird situation (I have to assume a bug) and
cannot find any references to this on the Internet. If you could chime in
with some advice, that'd be wonderful :)

Please note that this whole situation is related to python-flask, and I
have contacted their mailing list to see if anyone has seen this behavior
before. They didn't and suggested to see if you guys know more :-/

The problem is that I'm not sure at what point uwsgi-plugin-python starts
spawning new threads for handling requests. Maybe it's too early and
python-flask does some optimizations that result in a bug.

Here is the situation: We host a web-service through nginx, serving
requests to uwsgi-plugin-python (version 1.0.3+dfsg-1ubuntu0.1) to our nice
Python (2.7.3) flask app. Since we had garbage-collection issues with many
small objects on the heap in the past, we decided to respawn the uwsgi
service every few k requests and use multiple processes to avoid a
down-time in the respawn phase (and every process has multiple threads of
course).

Another important note is that we use the "lazy" option, because we use
MySQL connections which cannot be forked after Python initialization
(because all DB connections would share the same connection ID, which is
not supported by the MySQL client).

Now, it seems like each time the Python process respawns, that flask sets
up our blueprints (mappings between Python handlers and URLs in case you're
not familiar with flask) and then starts accepting requests, but
*sometimes* responds with a 404 to clients early during this phase. This
happens very rarely, but since there is considerable load on the server, it
actually starts adding up so that we need to something about it.

I have spent quite some time debugging the flask blueprints, and my
findings - so far - show that at app creation time, only one thread is
running in Python *after* the blueprints have been registered. However,
when I add code to log details about the blueprints / URL-rules when the
404 happens, then the blueprints are not all/fully loaded yet.

It seems to me like new threads are created and start accepting connections
before all blueprints/rules have been fully registered. Here is some code
to show what I mean:

def create_app(...):
    app = flask.Flask(__name__)
    [...]

    @app.after_request
    def after_request(response):
        if response.status_code == 404:
            # DBG2
            print 'dbg', response.status_code, request.base_url
            print threading.active_count(), "active threads"
            print app.url_map.__dict__.get('foo/mytesturl')
        return response

    app.register_blueprint(mymodel, url_prefix="/foo")
    # DBG1
    print 'done loading', app.url_map.__dict__
    print threading.active_count(), "active threads"
    return app

While the DBG1 code shows everything correctly, the DBG2 code shows a
sub-set of my URL-handlers (which is why the 404 gets executed).

BTW, at DBG1 point, the logs usually show "1 active threads", while at DBG2
many more threads are loaded, making me assume there is some sort of
lazy/on-demand loading of the rules within flask.

Again, the code works, it's just that there is this race where I get 404s.

Is this a known bug in uwsgi / uwsgi-plugin-python? If so, it'd be worth
investigating on a newer version, or is there a specific way to tell flask
to fully load all blueprints/rules before accepting connections?

Sorry for the overly long email, I wanted to get all the details right :)

Thanks!
-Clemens

-- 
Clemens Kolbitsch
Security Researcher
kolbitsch at lastline.com
Mobile +1 (206) 356-7745
Land +1 (805) 456-7076

Lastline, Inc.
6950 Hollister Avenue, Suite 101
Goleta, CA 93117

www.lastline.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.unbit.it/pipermail/uwsgi/attachments/20140401/599c3685/attachment.html>


More information about the uWSGI mailing list