So What is a GIL In Python ? Multiprocessing in Python.

No matter who you are or where you come from, one thing we can all agree upon is that Finding Nemo is an absolute masterpiece (10/10). if you have not watched it yet, then I would highly recommend watching it right away, of course after going through this article :)

As you might recall, Finding Nemo had a character called GILL (Moorish Idol fish). In the movie, GILL had a leadership role inside his fish tank. GILL made all the important decisions and planned the future for the betterment of his fish tank.

The GIL (Global Interpreter Lock) that we are discussing today, also has a very similar role in Python, a mechanism that has been developed around the memory management system of Python. GIL keeps the stuff (in this case avoiding memory leaks and race conditions) in order, and makes a lot of things simpler and cleaner. However, just like all that glitters are not gold, GIL also has some problems especially if want to write something which is compute-intensive (CPU-bound).

Let's dive a little deep (Python's GIL Mechanism)

In programming languages, if you want to do compute-intensive work, the first solution which pops up in your head is Threads. In python, while you can utilize the threading module, it is only suitable if the task that you are performing is I/O bound (reading and writing to disk, performing network operations). This is because GIL ( Global Interpreter Lock) only allows one thread to be in the state of execution at any given time. So, if your machine has let's say 8 cores, then you can not take advantage of all the cores for computation.

At this point, you might be asking yourself, why GIL has been designed like this. Python utilizes a reference counting mechanism to manage memory for you, this means if the reference count of a variable reaches zero, then it is automatically cleaned up. In many ways, this makes the life of a developer so much simpler.

# Variable name is cleaned up immediately after the 
# execution of function scope, as reference count reaches zero
def the_coolest_function_ever():
   name = "Aamir"


# ==== Let's take a slightly involved example =====


# In the following function planet_name will not be 
# cleaned, as at the end of the function scope the 
# reference count of planet_name will be greater than 0
def boot_planet(planet_name):
   print("Booting....")

   # planet_name Ref-count => 3
   print(planet_name)



def main():

   # planet_name Ref-count => 1
   planet_name = "Mars"

   # planet_name Ref-count => 2
   boot_planet(planet_name)

It can be seen in the above code snippet, that the reference counting system made things so much simpler when it comes to memory management, allowing you the developer to focus more on the app, instead of worrying about acquiring and releasing memory.

However, if more than one thread tries to access a variable simultaneously then the reference counting mechanism might wrongly increment or decrement the reference count of a variable resulting in thread race conditions. Because of this, a variable might be left uncleaned causing leaked memory, or even worse if a variable is cleaned at the wrong time our application might crash.

This is where Python's GIL mechanism comes in, it is the job of GIL (Global Interpreter Lock) to make sure that at any given time, only a single thread has access to variables and data structures. This avoids all the nasty thread race conditions and false modification of the reference counting variables.

A point to be noted here is that GIL (Global interpreter Lock), is different from other locking mechanisms used by various programming languages for thread-safe data access. Unlike other locking mechanisms which lock individual variables and data structures, GIL is the lock around the Python's interpreter itself, meaning any python code which requires execution has to explicitly acquire the GIL lock. The effectively results in single-threaded execution of any CPU-bound application.

Living happily with GIL (Bypassing GIL)

There are multiple ways to bypass GIL, firstly the GIL mechanism only exists in the original CPython implementation, and does not exist in Jython and IronPython, if your application along with its dependent libraries is available in some other Python implementation, then you can use that interpreter without worrying about the GIL.

Secondly, we can also utilize Python's Multiprocessing module, which lunches multiple Python processes in the memory, each with its own interpreter. However, creating and launching processes is still heavier than launching multiple threads.

Nothing in life is to be feared, it is only to be understood. Now is the time to understand more, so that we may fear less. Marie Curie