Python Global Interpreter Lock

April 27, 2009

Some friends and I have been playing around trying to write a game engine in Python. In an effort to avoid premature optimization, the whole thing has been single threaded up until this point. We're all still learning Python and hadn't used it for multithreading anything, but we figured that because it was Python, multithreading would be a simple task. We were right, to an extent.

Threading in Python really is a breeze. You simply import threading, create some threads, and start them up. Unfortunately, I coded that up and the game ran slower, even on multicore machines. After looking around the Python docs, I discovered that Python uses a global interpreter lock. This means that each Python interpreter can only execute one thread at a time. Threading is basically implemented via time sharing and not real threads. This means that threading doesn't actually give you any concurrency which makes programming much simpler, but it kills performance on multicore machines.

The truly unfortunate problem with the GIL is that you can't turn it off. There is no way to tell Python that I really do want it to run multiple interpreter threads and that I am willing to accept the overhead of performing locking on objects. The best workaround for this is to spawn multiple interpreter processes and communicate between them, but this incurs significantly more overhead than threading, and it is more difficult to implement.

After posting about this on a tech news site that I use, I learned about unladen-swallow which is a project that plans to greatly improve the performance of the Python interpreter. Among other things, the unladen-swallow team would like to "remove the GIL and fix the state of multithreading in Python". I hope that this projects goes somewhere and eventually becomes a part of Python. Until then, I'm going back to C++.

Check out my other pages tagged "blog".