[uWSGI] call a "streaming" LUA RPC from python

Riccardo Magliocchetti riccardo.magliocchetti at gmail.com
Tue Mar 29 17:45:13 CEST 2016


Hello Davide,

Il 29/03/2016 17:19, Davide Setti ha scritto:
> Hi,
> I have a python web service that does an HTTP request to ElasticSearch and
> generates a JSON with all the results. The problem is that the result set can be
> quite big, therefore we switched to ijson for streaming JSON decoding and
> chunked encoding for the output.
>
> But now it's slow, because ijson is: the whole thing is now CPU bound.
>
> One possible solution would be to write the JSON parsing in C, but I can't write
> proper C code anymore... :(

Have you tried ujson? maybe it's fast enough and does not consume too much memory

> Another solution would be to call a LUA RPC, but we would have memory problems
> because uWSGI RPC only supports strings as an exchange format, a single string I
> think. Are there tricks to easily dump to file/unix pipes or something?

If you want to go down this route maybe you can send to lua just the query url, 
process the file in lua and just return the processed output position instead of 
its full content?

-- 
Riccardo Magliocchetti
@rmistaken

http://menodizero.it


More information about the uWSGI mailing list