Python – Save Memory With Generators
Python
Python – Save Memory With Generators
Python

Python – Save Memory With Generators

First, let’s see two pieces of code, looks like no much difference, except the bracket () and square bracket [] inside sum function.

>>> sum([i * i for i in range(1, 1001)])
333833500
>>> sum((i * i for i in range(1, 1001)))
333833500

It’s really making no much difference for above code, at least from user’s point of view (feeling). Let’s increase the for loop range to 100 million.

>>> sum([i * i for i in range(1, 100000001)])
333333338333333350000000
>>> sum((i * i for i in range(1, 100000001)))
333333338333333350000000
List ComprehensionsGenerators
Performance comparison

See the difference? The memory. For the running time, it makes no much difference, but for the memory usage, it makes huge difference, 3.74GB vs 4.5 MB.

BTW, the tests run on my MacBookPro (2.2 GHz Quad-Core Intel Core i7, 16 GB 1600 MHz DDR3, SSD).

So what caused the difference? The generator expression. Generator expressions are perfect for when you know you want to retrieve data from a sequence, but you don’t need to access all of it at the same time.

Instead of creating a list, the generator expression returns a generator object. That object knows where it is in the current state (for example, i = 49) and only calculates the next value when it’s asked for.

So when sum iterates over the generator object by calling .__next__() repeatedly, the generator checks what i equals, calculates i * i, increments i internally, and returns the proper value to sum. The design allows generators to be used on massive sequences of data, because only one element exists in memory at a time.

Let’s explore the object size, here is the difference.

gc_obj = (x * x for x in range(1, 100000001))
print(sys.getsizeof(gc_obj))
835128600

lc_obj = [x * x for x in range(1, 100000001)]
print(sys.getsizeof(lc_obj))
208

Looks familiar, right? Yes, it’s kind of like when parsing a large file, instead of loading the entire file into memory at the same time, we are using a small buffer to repeatedly read data from a file descriptor.

Refs:

https://realpython.com/python-coding-interview-tips/#save-memory-with-generators

https://realpython.com/introduction-to-python-generators/#understanding-generators