Generators are special functions in which execution does not happen all at once like in traditional functions. Instead,  generators can pause the execution and resume later from the same point.  

By definition, generators are functions that contains one or more yield statements. The yield statement is similar to the return statement except that when it is encountered,  it does not immediately cause the function to exit . Instead, it causes the function to suspend its execution and returns the given value(if any) to the caller. The execution can then be resumed by calling the next() function on the generator.

Consider the following example:

ExampleEdit & Run
def func():
    print('Hello, World!')
    yield 1
    print('You are at Pynerds')
    yield 2
    print('Enjoy Your stay')
    yield 3

gen = func()
print(next(gen))
print('....................')
print(next(gen))
print('....................')
print(next(gen))
 Visualize 
copy
Output:
Hello, World!
1
....................
You are at Pynerds
2
....................
Enjoy Your stay
3
[Finished in 0.020751188043504953s]

As shown above, when a generator function is called it returns a generator object which is a typical iterator. Being an iterator, the object supports the iterator protocol, which means that it can be used with the next() function and other iteration constructs such as for loops.

The next() function makes the generator to start or resume its execution until another yield statement is encountered or the execution is complete. We can also use the generator with a for loop, which automatically calls the next() function on the generator until it reaches the end of the execution. 

ExampleEdit & Run
def func():
   L = [10, 20, 30, 40, 50, 60]
   for i in L:
       yield i

for i in func():
    print(i)
 Visualize 
copy
Output:
10
20
30
40
50
60
[Finished in 0.013683976139873266s]

As shown in the above example, the yield statement  can be used anywhere in the function's body like in loops, condition blocks, etc.

How generators work

Consider what happens when we call a normal function i.e those using the return statement. Whenever the return statement is encountered, the function simply returns the value and terminates without retaining any information on its internal state, any subsequent calls to the function will start fresh and execute the same code again.

On the other hand, whenever a generator object encounters a yield statement, it temporarily suspends/pauses the function and saves its state. This allows the generator to "remember" where it left off when it is called again, and it can resume execution from that point on rather than  starting all over again. This makes generators very efficient when dealing with large datasets or long-running calculations, since it allows you to get than results in chunks rather than loading the entire dataset into memory or waiting for a long-running calculation to finish before returning the results.

ExampleEdit & Run
def my_gen():
   data = range(10)
   print("started")
   for i in data:
       if i == 5:
           print('We are halfway.')
       yield i
   print('Done')

gen = my_gen()
for i in gen:
    print(i)
 Visualize 
copy
Output:
started
0
1
2
3
4
We are halfway.
5
6
7
8
9
Done
[Finished in 0.017291093710809946s]

Generator with a return statement

When a return statement is encountered in a generator object, the  generator gets closed and any subsequent calls will result in a StopIteration exception.

ExampleEdit & Run
def func():
   data = range(10)
   for i in data:
       yield i
       if i == 5:
           return

for i in func():
    print(i)
 Visualize 
copy
Output:
0
1
2
3
4
5
[Finished in 0.012430797796696424s]

The next function

The builtin  next() function is a useful feature when working with iterators. It is used to retrieve the next item from an iterator object. The function raises a StopIteration exception if the iterator is empty.

When called on a generator object, the next() function makes the the generator to proceed with the execution from where it had stopped until the next yield statement is reached or the execution is done.

ExampleEdit & Run
def func():
   L = range(10)
   for i in L:
       yield i

gen = func()
L = list(gen)

print(*L)

#The generator object is now empty
next(gen)
 Visualize 
copy
Output:
0 1 2 3 4 5 6 7 8 9

Traceback (most recent call last):
  File "<string>", line 12, in <module>
StopIteration

[Finished in 0.01268698601052165s]

As shown above, trying to access the next value in an empty generator object will raise a StopIteration exception. We can use the try blocks to catch the StopIteration exception. Example:

ExampleEdit & Run
def func():
   L = [1, 4, 8, 9]
   for i in L: 
      yield i

gen = func()
try:
    while True:
        value = next(gen)
        print(value)
except StopIteration:
    print("Nothing else")
 Visualize 
copy
Output:
1
4
8
9
Nothing else
[Finished in 0.015511297155171633s]

Closing the generator 

The generator objects contains the close() method which can be used to pre-maturely close the generator and free up any resources being used.

ExampleEdit & Run
def func():
    data = ['Python', 'Java', 'C++', 'Ruby', 'Swift']
    for i in data:
        yield i

gen = func()
print(next(gen))
print(next(gen))
print(next(gen))

#close the generator
gen.close()

#subsuent calls will raise a StopIteration exception
next(gen)
 Visualize 
copy
Output:
Python
Java
C++

Traceback (most recent call last):
  File "<string>", line 15, in <module>
StopIteration

[Finished in 0.012801798060536385s]

Controlling the generator from outside

When we are performing subsequent calls to the generator, we can use the send() method to send a value to the generator. This method can be thought partly as the opposite of the yield statement. It resumes the function execution and receives the yielded value just like the next() function, but also sends an external value back to the yield statement.

Syntax: 
send(obj)
copy

The object sent is received by the yield statement.  We can only send a single object.

ExampleEdit & Run
def square():
    recieved = None
    while True:
        if recieved is None: 
            recieved = yield "No value given."
        else:
            recieved = yield f"{recieved} ^ 2 = {recieved ** 2}"

gen = square()
print(gen.send(None))
print(gen.send(5))
print(gen.send(10))
print(gen.send(21))
print(gen.send(25))
print(gen.send(30))
print(gen.send(50))

gen.close()
 Visualize 
copy
Output:
No value given.
5 ^ 2 = 25
10 ^ 2 = 100
21 ^ 2 = 441
25 ^ 2 = 625
30 ^ 2 = 900
50 ^ 2 = 2500
[Finished in 0.013761338777840137s]

Note that we cannot send a value(other than None) in the first yield call.

Get the Internal state of the generator object

Generator objects contains some attributes which we can use to get its internal state. For example the gi_suspended and the gi_running attributes returns a boolean value indicating whether the generator object is currently paused or running, respectively.

def func():
    data = [1, 2, 3, 4, 5]
    for i in data:
        yield i

gen = func()

print(next(gen))

print('running: ', gen.gi_running)
print('suspended: ', gen.gi_suspended)
copy
Output:
1
running:  False
suspended:  True 

Lazy Evaluation in Generator Functions

Lazy evaluation is an optimization technique in which the evaluation of an expression is delayed until the value is needed. Generator functions use this technique such that the values returned are not loaded in memory all at once, instead, the items are efficiently processed one at a time as needed.This makes them capable of efficiently handling large data sets which would otherwise take up a lot of memory space.