[uWSGI] Problem with multiple processes with httplib.HTTPConnection?

Alan Castro alanclic at gmail.com
Wed Nov 10 02:09:38 CET 2010


Roberto,

I posted this situation on the sqlalchemy google group and I got the
following answer:

"DBAPI connections don't travel well over process boundaries.  You need to
call create_engine() inside each new child process."

I saw that the create_engine gets called only once and then the workers
spawns... Is this a simple thing to do in uWSGI, like a 'process start
callback' or 'per process app initialization'.

Since I didn't see a fast way to develop this, i created a uwsgi cluster in
nginx and tested it again. And guess what? It worked as a charm!
Well this was a quick setup, tomorrow I'll do more tests on this.

Thanks Roberto!

Regards

Alan

On Tue, Nov 9, 2010 at 7:54 PM, Alan Castro <alanclic at gmail.com> wrote:

> Roberto,
>
> I got the following error:
>
> OperationalError: (OperationalError) (2013, 'Lost connection to MySQL
> server during query') 'SELECT `PROGRAM`.id AS `PROGRAM_id`, `PROGRAM`.title
> AS `PROGRAM_title`, `PROGRAM`.type AS `PROGRAM_type` \nFROM `PROGRAM`' ()
>
> Check if you have the same problem there.
>
> Alan
>
>
> On Tue, Nov 9, 2010 at 7:09 PM, Alan Castro <alanclic at gmail.com> wrote:
>
>> Roberto,
>>
>> I've created a very simple pylons application as fast as possible that
>> reproduces the error with mysql adapter.
>>
>> You can clone it at:
>> git clone git://github.com/alancastro/simpleapp.git
>>
>> There is a simple readme there too:
>> http://github.com/alancastro/simpleapp
>>
>> Thank you
>>
>> Alan
>>
>>
>> On Tue, Nov 9, 2010 at 4:52 PM, Alan Castro <alanclic at gmail.com> wrote:
>>
>>> The -i didn't solve it..
>>>
>>> I'll work on a simple app and reproduce the problem from there and I'll
>>> send it to you.
>>>
>>> Thanks alot for the attention
>>>
>>> Alan
>>>
>>>
>>> On Tue, Nov 9, 2010 at 4:35 PM, Roberto De Ioris <roberto at unbit.it>wrote:
>>>
>>>>
>>>> > Hi Roberto,
>>>> >
>>>> > I tried uwsgi last last version with --threads and the current stable
>>>> with
>>>> > -T and the problem still occurs..
>>>> >
>>>> > The real problem is SQLAlchemy + Pylons + Multi-process.
>>>> >
>>>> > Summaryzing:
>>>> >
>>>> > There are 2 situations where everything works:
>>>> >
>>>> > * when I comment out the every db related code and deploy on a
>>>> > 'multi-process environment'.
>>>> >
>>>> > * and if I use a '1 process environment' with db related code.
>>>> >
>>>> > And 1 situation (the one I really wanted to work) that does not work:
>>>> >
>>>> > * 'multi-process environment' with db related code.
>>>>
>>>>
>>>> Can you change the db adapter (something like postgres or sqlite) to see
>>>> if the problem is oracle related ?
>>>>
>>>> Another problem can be (i have never used the oracle adaptor) with
>>>> multiple interpreters, try passing the -i option to a stable release.
>>>>
>>>> If both still do not works, can you send a skeletal app that i can use
>>>> for
>>>> tests ?
>>>>
>>>> Thanks a lot
>>>>
>>>>
>>>> >
>>>> > Thanks
>>>> >
>>>> > Alan
>>>> >
>>>> > On Tue, Nov 9, 2010 at 3:55 PM, Roberto De Ioris <roberto at unbit.it>
>>>> wrote:
>>>> >
>>>> >>
>>>> >> > I just deployed with the current version of mercurial and when i
>>>> use
>>>> >> -T
>>>> >> > option my app won't work. (without -T it works)
>>>> >> >
>>>> >> > The "ready_to_reload..." is a normal debug info?
>>>> >>
>>>> >>
>>>> >>
>>>> >> Please try the -T with the current stable releases, and --threads <n>
>>>> >> (use a low number like 4) with the mercurial one
>>>> >>
>>>> >> >
>>>> >> > Thanks
>>>> >> >
>>>> >> > Alan
>>>> >> >
>>>> >> > On Tue, Nov 9, 2010 at 3:05 PM, Roberto De Ioris <roberto at unbit.it
>>>> >
>>>> >> wrote:
>>>> >> >
>>>> >> >>
>>>> >> >> Il giorno 09/nov/2010, alle ore 17.39, Alan Castro ha scritto:
>>>> >> >>
>>>> >> >> > Roberto,
>>>> >> >> >
>>>> >> >> > I'm still facing this problem.
>>>> >> >> >
>>>> >> >> > Let me context my problem: I developed a Pylons application
>>>> (which
>>>> >> is
>>>> >> >> multithreaded, each request fires up a new thread) and is working
>>>> >> fine
>>>> >> >> (deployed with uwsgi with one process). When I run it on a
>>>> >> multiprocess
>>>> >> >> with
>>>> >> >> uwsgi eveything goes out of control and I keep getting this Oracle
>>>> >> error
>>>> >> >> TNS: Bad packet, which looks like problem with my sqlalchemy
>>>> session
>>>> >> >> object.
>>>> >> >> >
>>>> >> >> > I know this runs a bit out of uwsgi context, but do you know how
>>>> >> much
>>>> >> >> uwsgi multiprocess environment affects pylons multithreaded? In my
>>>> >> head
>>>> >> >> everything should work fine.
>>>> >> >> >
>>>> >> >>
>>>> >> >> I suppose that your app generate a new python thread at every
>>>> request
>>>> >> >> that
>>>> >> >> do something in the background and pass data to the "main" thread
>>>> >> that
>>>> >> >> will
>>>> >> >> send the response to the webserver; in this case, have you enabled
>>>> >> the
>>>> >> >> -T
>>>> >> >> option ?
>>>> >> >> Without it the GIL is not initialized and i do not know what could
>>>> >> >> happen
>>>> >> >> :)  If instead your app generate a new thread that "alone" manages
>>>> >> the
>>>> >> >> whole
>>>> >> >> request
>>>> >> >> (so the thread itself generate the wsgi response) this will cause
>>>> all
>>>> >> >> sort
>>>> >> >> of problems, as the current uWSGI stable releases are not
>>>> >> multithreaded
>>>> >> >> (you
>>>> >> >> have to use
>>>> >> >> a 0.9.7-dev release from mercurial repository)
>>>> >> >>
>>>> >> >> --
>>>> >> >> Roberto De Ioris
>>>> >> >> http://unbit.it
>>>> >> >>
>>>> >> >> _______________________________________________
>>>> >> >> uWSGI mailing list
>>>> >> >> uWSGI at lists.unbit.it
>>>> >> >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>>>> >> >>
>>>> >> > _______________________________________________
>>>> >> > uWSGI mailing list
>>>> >> > uWSGI at lists.unbit.it
>>>> >> > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>>>> >> >
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Roberto De Ioris
>>>> >> http://unbit.it
>>>> >> _______________________________________________
>>>> >> uWSGI mailing list
>>>> >> uWSGI at lists.unbit.it
>>>> >> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>>>> >>
>>>> > _______________________________________________
>>>> > uWSGI mailing list
>>>> > uWSGI at lists.unbit.it
>>>> > http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>>>> >
>>>>
>>>>
>>>> --
>>>> Roberto De Ioris
>>>> http://unbit.it
>>>> _______________________________________________
>>>> uWSGI mailing list
>>>> uWSGI at lists.unbit.it
>>>> http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.unbit.it/pipermail/uwsgi/attachments/20101109/364dc29c/attachment.htm 


More information about the uWSGI mailing list