DEV Community

loading...

Here is how Python list's memory size changes w.r.t it's length

Dilli Babu R
I am working on RPA, love Python, Julia, JavaScript. Happy to contribute to opensource community in anyway I can.
・3 min read

We use python's Lists pretty much everywhere. Since we don't declare it's size and we just add items, Ever wondered how and when the memory allocation and consumption is done under the hood?

Lets analyze the same now, for that we need...

  1. sys package to get the list size
  2. matplotlib for plotting graphs
    1. Lets start writing code by importing the packages.
      import sys
      from matplotlib import pyplot as plt
      

      Lets print and see the length and memory of our list for 10 items.

      l = [] # our list
      size = 10
      
      # here we are just adding some values to our list
      
      for counter in range(size):
          l.append(counter)
          print("length {}, memory in bytes {}".format(
              length[counter], memory[counter]))
      

      We can see the following in console.

      length 1, memory in bytes 96
      length 2, memory in bytes 96
      length 3, memory in bytes 96
      length 4, memory in bytes 96
      length 5, memory in bytes 128
      length 6, memory in bytes 128
      length 7, memory in bytes 128
      length 8, memory in bytes 128
      length 9, memory in bytes 192
      length 10, memory in bytes 192
      

      We can see that after about 4 iterations each time, the memory is increased.
      Now lets increase the loop counter to 1000 and graph it out...

      size = 1000 # increasing the size
      length = [] # to hold lengths at each iteration
      memory = [] # to hold memory values at each iteration
      

      And then change the for loop

      for counter in range(size):
          l.append(counter)
          # capture and store the length
          length.append(len(l)) 
          # Capture and store the memory in bytes
          memory.append(sys.getsizeof(l))
      

      Now lets plot the graph using the captured values and see what is shows.

      plt.plot(length, memory)
      plt.title("Python List length - Memory comparison")
      plt.xlabel("Length")
      plt.ylabel("Memory in bytes")
      plt.show()
      

      Alt Text

      We can see that when ever list is running out of size, its getting added with additional memory.

      Now, Lets change the code to capture the difference in memory only when there is a change.

      # declaring two more lists
      counter_at_which_memory_changed = []
      changed_memory_difference = []
      

      and changing the for loop as follows

      for counter in range(size):
          l.append(counter)
          # capture and store the length
          length.append(len(l))
          # Capture and store the memory in bytes
          memory.append(sys.getsizeof(l))
      
          if memory[counter] != memory[counter - 1]:
              counter_at_which_memory_changed.append(counter)
              changed_memory_difference.append(
                  memory[counter] - memory[counter - 1])
      
      # Lets try checking the first 10 values of length and difference in memory
      
      for counter in range(10):
          print("at list length {}, memory increase in bytes {}".format(
              counter_at_which_memory_changed[counter],
              changed_memory_difference[counter]))
      

      The first 10 values of memory changes is as follows.

      at list length 4, memory increase in bytes 32
      at list length 8, memory increase in bytes 64
      at list length 16, memory increase in bytes 72
      at list length 25, memory increase in bytes 80
      at list length 35, memory increase in bytes 88
      at list length 46, memory increase in bytes 96
      at list length 58, memory increase in bytes 112
      at list length 72, memory increase in bytes 128
      at list length 88, memory increase in bytes 144
      at list length 106, memory increase in bytes 160
      
      # And the graph for index - memory difference is as follows
      plt.plot(counter_at_which_memory_changed, changed_memory_difference)
      plt.title("Counter - Memory difference")
      plt.xlabel("Counter")
      plt.ylabel("Memory difference")
      plt.show()
      

      Alt Text

      So, by checking the difference we can see that the memory increase is follows..
      8, 8, 8, 8, 16, 16, 16, 16, 16, 24, 24, 32, 32, 32, 40, 48, 48, 56, 64, 72, 88, 88, 104, 112.

      Thanks for your time, this is my first post. Comments are welcome :)

Discussion (0)