Generator Functions in Python

Generators are special functions in which execution does not happen all at once like in traditional functions. Instead,  generators can pause the execution and resume later from the same point.  

By definition, generators are functions that contains one or more yield statements. The yield statement is similar to the return statement except that when it is encountered,  it does not immediately cause the function to exit . Instead, it causes the function to suspend its execution and returns the given value(if any) to the caller. The execution can then be resumed by calling the next() function on the generator.

Consider the following example:

def func():
    print('Hello, World!')
    yield 1
    print('You are at Pynerds')
    yield 2
    print('Enjoy Your stay')
    yield 3

gen = func()
print(next(gen))
print('....................')
print(next(gen))
print('....................')
print(next(gen))

As shown above, when a generator function is called it returns a generator object which is a typical iterator. Being an iterator, the object supports the iterator protocol, which means that it can be used with the next() function and other iteration constructs such as for loops.

The next() function makes the generator to start or resume its execution until another yield statement is encountered or the execution is complete. We can also use the generator with a for loop, which automatically calls the next() function on the generator until it reaches the end of the execution. 

def func():
   L = [10, 20, 30, 40, 50, 60]
   for i in L:
       yield i

for i in func():
    print(i)

As shown in the above example, the yield statement  can be used anywhere in the function's body like in loops, condition blocks, etc.

How generators work

Consider what happens when we call a normal function i.e those using the return statement. Whenever the return statement is encountered, the function simply returns the value and terminates without retaining any information on its internal state, any subsequent calls to the function will start fresh and execute the same code again.

On the other hand, whenever a generator object encounters a yield statement, it temporarily suspends/pauses the function and saves its state. This allows the generator to "remember" where it left off when it is called again, and it can resume execution from that point on rather than  starting all over again. This makes generators very efficient when dealing with large datasets or long-running calculations, since it allows you to get than results in chunks rather than loading the entire dataset into memory or waiting for a long-running calculation to finish before returning the results.

def my_gen():
   data = range(10)
   print("started")
   for i in data:
       if i == 5:
           print('We are halfway.')
       yield i
   print('Done')

gen = my_gen()
for i in gen:
    print(i)

Generator with a return statement

When a return statement is encountered in a generator object, the  generator gets closed and any subsequent calls will result in a StopIteration exception.

def func():
   data = range(10)
   for i in data:
       yield i
       if i == 5:
           return

for i in func():
    print(i)

The next function

The builtin  next() function is a useful feature when working with iterators. It is used to retrieve the next item from an iterator object. The function raises a StopIteration exception if the iterator is empty.

When called on a generator object, the next() function makes the the generator to proceed with the execution from where it had stopped until the next yield statement is reached or the execution is done.

def func():
   L = range(10)
   for i in L:
       yield i

gen = func()
L = list(gen)

print(*L)

#The generator object is now empty
next(gen)

As shown above, trying to access the next value in an empty generator object will raise a StopIteration exception. We can use the try blocks to catch the StopIteration exception. Example:

def func():
   L = [1, 4, 8, 9]
   for i in L: 
      yield i

gen = func()
try:
    while True:
        value = next(gen)
        print(value)
except StopIteration:
    print("Nothing else")

Closing the generator 

The generator objects contains the close() method which can be used to pre-maturely close the generator and free up any resources being used.

def func():
    data = ['Python', 'Java', 'C++', 'Ruby', 'Swift']
    for i in data:
        yield i

gen = func()
print(next(gen))
print(next(gen))
print(next(gen))

#close the generator
gen.close()

#subsuent calls will raise a StopIteration exception
next(gen)

Controlling the generator from outside

When we are performing subsequent calls to the generator, we can use the send() method to send a value to the generator. This method can be thought partly as the opposite of the yield statement. It resumes the function execution and receives the yielded value just like the next() function, but also sends an external value back to the yield statement.

send(obj)

The object sent is received by the yield statement.  We can only send a single object.

def square():
    recieved = None
    while True:
        if recieved is None: 
            recieved = yield "No value given."
        else:
            recieved = yield f"{recieved} ^ 2 = {recieved ** 2}"

gen = square()
print(gen.send(None))
print(gen.send(5))
print(gen.send(10))
print(gen.send(21))
print(gen.send(25))
print(gen.send(30))
print(gen.send(50))

gen.close()

Note that we cannot send a value(other than None) in the first yield call.

Get the Internal state of the generator object

Generator objects contains some attributes which we can use to get its internal state. For example the gi_suspended and the gi_running attributes returns a boolean value indicating whether the generator object is currently paused or running, respectively.

def func():
    data = [1, 2, 3, 4, 5]
    for i in data:
        yield i

gen = func()

print(next(gen))

print('running: ', gen.gi_running)
print('suspended: ', gen.gi_suspended)

1
running:  False
suspended:  True 

Lazy Evaluation in Generator Functions

Lazy evaluation is an optimization technique in which the evaluation of an expression is delayed until the value is needed. Generator functions use this technique such that the values returned are not loaded in memory all at once, instead, the items are efficiently processed one at a time as needed.This makes them capable of efficiently handling large data sets which would otherwise take up a lot of memory space.

Python Functions