"async for" expressions in Python

In Python, we can have asynchronous iterators. If you are familiar with iterators, you know that they are required to implement __iter__() and __next__() methods. Similarly,  asynchronous iterators are required to implement __aiter__() and __anext__() methods. The __anext__() method must return an awaitable object, usually the current iterator itself.

Asynchronous generators are the most basic forms of asynchronous iterators as  the __aiter__() and __anext__() methods are implemented automatically. An asynchronous generator is simply a coroutine with one or more yield statements. 

import asyncio

#an async generator
async def get_evens(n):
   for i in range(n):
      if i % 2 == 0: 
         yield i

async def main():
   evens = get_evens(10)
   print(await anext(evens))
   print(await anext(evens))
   print(await anext(evens))
   print(await anext(evens))
   print(await anext(evens))
   print(await anext(evens))

asyncio.run(main())
 Visualize 

Output:
0
2
4
6
8

StopAsyncIteration
[Finished in 0.05639263289049268s]

The get_evens() function above is an example of a simple asynchronous generator. It takes an integer n as argument, then yields all the even numbers from 0 to n.

The builtin anext() function retrieves the next object in an asynchronous iterator it is analogous to the next() function in regular iterators.

Note that unlike in regular generators where calling the next() function on it will return the actual yielded value, with the anext() function, we have to await the returned object in order to retrieve the actual value.

basic "async for" usage

Consider if we wanted to repeatedly iterate through the elements of the previous asynchronous generator(get_evens()) in a loop so that we do not need to await the values manually. We cannot do this with regular loops because for one reason, asynchronous generators are not iterable.

import asyncio

async def get_evens(n):
   for i in range(n):
      if i % 2 == 0: 
         yield i

async def main():
   for i in get_evens(10):
      print(i)

asyncio.run(main())
 Visualize 
copied!

Output:

TypeError: 'async_generator' object is not iterable
[Finished in 0.056022260803729296s]

As shown above, a TypeError exception gets raised because the asynchronous generators are considered not iterable, this is why we need async for expressions. The above example will work as intended if we use async for instead of regular for loops.

import asyncio

async def get_evens(n):
   for i in range(n):
      if i % 2 == 0: 
         yield i

async def main():
   async for i in get_evens(10):
      print(i)

asyncio.run(main())
 Visualize 
copied!

Output:
0
2
4
6
8

[Finished in 0.055968062952160835s]

As shown above, the yielded values are correctly retrieved when we use an asynchronous generator with an async for expression. Note that the async for expression automatically awaits the yielded awaitables. It also must be used inside of a coroutine function, we cannot use them as standalone statements as this will lead to a SyntaxError exception being raised, this is shown below.

async def get_evens(n):
   for i in range(n):
      if i % 2 == 0: 
         yield i

async for i in get_evens(10):
      print(i)
 Visualize 
copied!

Output:

SyntaxError: 'async for' outside async function
[Finished in 0.01014859788119793s]

async for vs regular for loops

In order to better understand how the async for expressions works, let us compare them with their counterparts, the for loops.

Simply put, for loops works with regular iterators and iterables while async for will only work with asynchronous iterators. Asynchronous iterators are not necessarily iterables and they cannot be used in regular for loops. 

The name "async for" may be misunderstood to mean that it is a parallelized for loop and that the elements of the target asynchronous iterator are being traversed asynchronously i.e in parallel, but as we have already seen, this is not so. The async for expression simply iterates sequentially through the items of an async source.

Let us understand this better with an example that is closely related to our previous example:

import asyncio

async def get_odds(n):
   for i in range(n):
      if i % 2 == 1: 
         yield i

async def main():
   async for i in get_odds(10):
      print(i)

asyncio.run(main())
 Visualize 
copied!

Output:
1
3
5
7
9

[Finished in 0.05854252818971872s]

The get_odds()  above is an asynchronous generator that yields odd numbers from 0 to n. We used the async for expression inside main() to retrieve the yielded values. 

The closest way of doing the same with a regular for loop would be to create a list of the awaitables and then await each in a for loop as shown below:

import asyncio

async def get_odds(n):
   for i in range(n):
      if i % 2 == 1: 
         yield i

async def main():
   odds = get_odds(10)
   awaitable_list = [anext(odds), anext(odds), anext(odds), anext(odds)]
   for awaitable in awaitable_list:
      print(await awaitable)

asyncio.run(main())
 Visualize 
copied!

Output:
1
3
5
7

[Finished in 0.056094068102538586s]

As shown, with regular loops, you have to manually await the yielded awaitables but with async for expression, they are awaited automatically.

Another common error is to assume that async for will work on iterables such as lists that contains awaitable objects. However, this will blatantly fail.

import asyncio

async def get_odds(n):
   for i in range(n):
      if i % 2 == 1: 
         yield i

async def main():
   odds = get_odds(10)
   odds =[ anext(odds), anext(odds)]
   async for i in odds: #this fails
      print(i)

asyncio.run(main())
 Visualize 
copied!

Output:

TypeError: 'async for' requires an object with __aiter__ method, got list
[Finished in 0.055924112908542156s]

As shown above, the async for expression requires the target object to have the __aiter__() method, if not so a TypeError exception is raised.

async for with custom asynchronous iterators

So far we have been using asynchronous generators in which the __aiter__() and __anext__() methods are implemented automatically. With custom asynchronous iterators, the two methods have to be implemented manually in the class definition. In this section we will work with such objects.

Each of the __aiter__() and __anext__() methods have its own purpose. The __aiter__() method returns an instance of the iterator, while the __anext__() method returns the next awaitable in the sequence.

After implementing the two method, the objects of the class become eligible for use in async for expressions.

Let us create the equivalent iterator to the get_evens() generator in our previous examples. The iterator will generate even numbers starting from 0 up to n.

import asyncio

class EvensGenerator:
   def __init__(self, n):
      self.n = n  
      self.current = 0

   def __aiter__(self):
      return self

   async def __anext__(self):
      if self.current >= self.n:
         raise StopAsyncIteration
      value = self.current
      self.current += 2
      return value

async def main():
   async for i in EvensGenerator(20):
      print(i)

asyncio.run(main())
 Visualize 
copied!

Output:
0
2
4
6
8
10
12
14
16
18

[Finished in 0.055947919841855764s]

Calling EvensGenerator(n) creates an EvensGenerator object. Since the Evensgenerator class implements __aiter__() and __anext__(), we can use the returned object in an async for expression.

In our example above, the __aiter__() method simply returns the current object i.e self. The __anext__() method implements the logic for moving from one even number to the next until n is reached, in which case the StopAsycIteration exception is raised. The StopAsyncIteration is analogous to the StopIteration exception in regular iterators.

Note that unlike __aiter__(),  the __anext__() method is defined as a coroutine (with async def)  as it is supposed to return an awaitable object.  Failure to define __anext__() as a coroutine will raise a TypeError exception, as demonstrated below.

import asyncio

class EvensGenerator:
   def __init__(self, n):
      self.n = n  
      self.current = 0

   def __aiter__(self):
      return self

   def __anext__(self):
      if self.current >= self.n:
         raise StopAsyncIteration
      value = self.current
      self.current += 2
      return value

async def main():
   async for i in EvensGenerator(20):
      print(i)

asyncio.run(main())
 Visualize 
copied!

Output:

TypeError: 'async for' received an invalid object from __anext__: int
[Finished in 0.05674647679552436s]

The TypeError exception is raised because the __anext__() method is supposed to return an awaitable object but in the above case, it returned an integer.