[uWSGI] Possible setup race condition between uwsgi-plugin-python and python-flask

Clemens Kolbitsch kolbitsch at lastline.com
Wed Apr 2 00:48:43 CEST 2014


Hi Roberto!

Thanks for picking up this thread. I was afraid that's what I'd hear -
we're running on Ubuntu LTS 12.04.1, which is why we are using such an old
version. I'll try to upgrade and re-test.

-Clemens


On Tue, Apr 1, 2014 at 3:17 PM, Roberto De Ioris <roberto at unbit.it> wrote:

>
> > Hi guys,
> >
> > I've been struggling with a weird situation (I have to assume a bug) and
> > cannot find any references to this on the Internet. If you could chime in
> > with some advice, that'd be wonderful :)
> >
> > Please note that this whole situation is related to python-flask, and I
> > have contacted their mailing list to see if anyone has seen this behavior
> > before. They didn't and suggested to see if you guys know more :-/
> >
> > The problem is that I'm not sure at what point uwsgi-plugin-python starts
> > spawning new threads for handling requests. Maybe it's too early and
> > python-flask does some optimizations that result in a bug.
> >
> > Here is the situation: We host a web-service through nginx, serving
> > requests to uwsgi-plugin-python (version 1.0.3+dfsg-1ubuntu0.1) to our
> > nice
> > Python (2.7.3) flask app. Since we had garbage-collection issues with
> many
> > small objects on the heap in the past, we decided to respawn the uwsgi
> > service every few k requests and use multiple processes to avoid a
> > down-time in the respawn phase (and every process has multiple threads of
> > course).
> >
> > Another important note is that we use the "lazy" option, because we use
> > MySQL connections which cannot be forked after Python initialization
> > (because all DB connections would share the same connection ID, which is
> > not supported by the MySQL client).
> >
> > Now, it seems like each time the Python process respawns, that flask sets
> > up our blueprints (mappings between Python handlers and URLs in case
> > you're
> > not familiar with flask) and then starts accepting requests, but
> > *sometimes* responds with a 404 to clients early during this phase. This
> > happens very rarely, but since there is considerable load on the server,
> > it
> > actually starts adding up so that we need to something about it.
> >
> > I have spent quite some time debugging the flask blueprints, and my
> > findings - so far - show that at app creation time, only one thread is
> > running in Python *after* the blueprints have been registered. However,
> > when I add code to log details about the blueprints / URL-rules when the
> > 404 happens, then the blueprints are not all/fully loaded yet.
> >
> > It seems to me like new threads are created and start accepting
> > connections
> > before all blueprints/rules have been fully registered. Here is some code
> > to show what I mean:
> >
> > def create_app(...):
> >     app = flask.Flask(__name__)
> >     [...]
> >
> >     @app.after_request
> >     def after_request(response):
> >         if response.status_code == 404:
> >             # DBG2
> >             print 'dbg', response.status_code, request.base_url
> >             print threading.active_count(), "active threads"
> >             print app.url_map.__dict__.get('foo/mytesturl')
> >         return response
> >
> >     app.register_blueprint(mymodel, url_prefix="/foo")
> >     # DBG1
> >     print 'done loading', app.url_map.__dict__
> >     print threading.active_count(), "active threads"
> >     return app
> >
> > While the DBG1 code shows everything correctly, the DBG2 code shows a
> > sub-set of my URL-handlers (which is why the 404 gets executed).
> >
> > BTW, at DBG1 point, the logs usually show "1 active threads", while at
> > DBG2
> > many more threads are loaded, making me assume there is some sort of
> > lazy/on-demand loading of the rules within flask.
> >
> > Again, the code works, it's just that there is this race where I get
> 404s.
> >
> > Is this a known bug in uwsgi / uwsgi-plugin-python? If so, it'd be worth
> > investigating on a newer version, or is there a specific way to tell
> flask
> > to fully load all blueprints/rules before accepting connections?
> >
> > Sorry for the overly long email, I wanted to get all the details right :)
> >
> > Thanks!
> > -Clemens
> >
> > --
> > Clemens Kolbitsch
> > Security Researcher
> > kolbitsch at lastline.com
> > Mobile +1 (206) 356-7745
> > Land +1 (805) 456-7076
> >
> > Lastline, Inc.
> > 6950 Hollister Avenue, Suite 101
> > Goleta, CA 93117
> >
> >
>
> Hi Clemens, sorry but uWSGI 1.0 is unsupported since 2012 and know to be
> broken (as well as 1.2). Only 1.4.x and 2.0.x are supported and actively
> maintained.
>
> Upgrade to a stable version, and if you still have problems feel free to
> repost.
>
>
> --
> Roberto De Ioris
> http://unbit.it
> _______________________________________________
> uWSGI mailing list
> uWSGI at lists.unbit.it
> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>



-- 
Clemens Kolbitsch
Security Researcher
kolbitsch at lastline.com
Mobile +1 (206) 356-7745
Land +1 (805) 456-7076

Lastline, Inc.
6950 Hollister Avenue, Suite 101
Goleta, CA 93117

www.lastline.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.unbit.it/pipermail/uwsgi/attachments/20140401/71b9da18/attachment-0001.html>


More information about the uWSGI mailing list