Python multiprocessing Pool hangs on ubuntu server

By : apardes
Source: Stackoverflow.com
Question!

I'm running Django on an Ubuntu server with nginx and gunicorn. I'm trying to do some multiprocessing which is working on my local machine but hangs until the gunicorn worker times out on my server.

cpu_count = int(multiprocessing.cpu_count())
pool = Pool(processes = cpu_count)
result = pool.map_async(apiSimulAvail, rate_ranges)
result.wait()

...do some more stuff once all processes return

It hangs at pool = Pool(processes = cpu_count). I don't get any errors, the gunicorn worker just times out and reboots.

Any indication as to why this is happening and/or how I can solve it is greatly appreciated. Thanks.

By : apardes


Answers

Change

pool = Pool(processes = cpu_count)

to

pool = Pool(cpu_count)

This assumes that you've imported Pool from multiprocessing otherwise you'll need to do

multiprocessing.Pool(cpu_count)


Are you using some async Gunicorn worker? If so, try the default sync worker and see if you can reproduce the problem.

If the problem can only be reproduced when using async workers, you should make sure the multiprocessing module is patched correctly.

By : satoru


This video can help you solving your question :)
By: admin