The gather() function in the asyncio library is used to group multiple tasks/coroutines into a single awaitable object. When this object is awaited, the execution will wait until each of  the grouped coroutines is fully executed.

The gathered tasks are executed simultaneously in a non-blocking manner,  this means that if a delay is encountered in one task, execution can move on to the other task and then return later on to the coroutine where the delay was encountered.

Let us consider a basic example:

import asyncio

async def display_nums():
   for n in range(5):
      print(n)
      await asyncio.sleep(0.001) #simulate a small delay

async def display_letters():
   for let in "ABCDE":
      print(let)
      await asyncio.sleep(0.001) #simulate a small delay

async def main():
   print("Started")
   await asyncio.gather(display_nums(), display_letters()) #call the gather function with the two tasks

asyncio.run(main())

If you observe the output of the above snippet, you will see that the small delay caused by the asyncio.sleep() calls is enough to make the execution to alternate between the two gathered coroutines.

The asyncio.gather() function takes an arbitrary number of coroutines as arguments. It creates a task for each coroutine in an event loop and then returns an awaitable object which we can await inside of an async function. As we have seen above, the gathered tasks gets executed simultaneously in a non-blocking manner.

How asyncio.gather() works

The function takes one or more awaitables as argument(s). Thus the basic syntax is:

asyncio.gather(coro1(), coro2(), coro3(), ...)

The function also takes one optional argument called return_exceptions, thus the extended syntax is:

asyncio.gather(*coros, return_exceptions = False)

The function returns a Future object, which is itself awaitable.

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   result = asyncio.gather(square(10), cube(10), square(11), cube(11), return_exceptions = True)
   print(result)

asyncio.run(main())

Note that in the above example, we did not await the Future object and therefore, the grouped tasks are not executed.   When the object is awaited,  it ensures that all the gathered tasks are fully executed and it finally returns a list with their return values in the order of input.

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   print("Started")
   return_values = await asyncio.gather(square(10), cube(10), square(11), cube(11))
   print(return_values)
   print('Finished')

asyncio.run(main())

As shown above, the awaited Future object returns a list whose elements are the the values returned by the  coroutines.

If an error is encountered in the execution of a coroutine, an exception is raised and the execution is terminated. However, if the return_exceptions is set to True, the raised exception will be added in the results in the place of the involved coroutine's return value,  and the execution will continue with the remaining coroutines.  

with return_exceptions set to False

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   print("Started")
   return_values = await asyncio.gather(square(10), cube(10), square('eleven'), cube(11))
   print(return_values)
   print('Finished')

asyncio.run(main())

with return_exception set to True

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   print("Started")
   return_values = await asyncio.gather(square(10), cube(10), square('eleven'), cube(11), return_exceptions = True)
   print(return_values)
   print('Finished')

asyncio.run(main())

In the above examples, the third coroutine in the gather() function raises an exception because 'eleven' cannot be cast into an integer.

In the first snippet, the capture_exceptions parameter is False and hence the exception is raised an the execution is halted.

In the second case, the capture_exceptions is set to True and hence the exception is captured and entered in place of the last coroutine's return value. The execution exits successfully without the exception being propagated to the callers.

gather() with other awaitables

We have already seen how we can use asyncio.gather() with coroutine. Apart from coroutines, asyncio.gather() can work with any other awaitable object such as tasks and future objects.

This means that we can also use an object returned by gather() inside of another asyncio.gather() call. Consider the following example:

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   print("Started")
   group1 = asyncio.gather(square(10), cube(10))
   group2 = asyncio.gather(square(20), cube(20))
   
   return_values = await asyncio.gather(group1, group2)
   print(return_values)
   print('Finished')

asyncio.run(main())

We can as well combine coroutines with gather objects and any other awaitable object.

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   print("Started")
   group1 = asyncio.gather(square(10), cube(10))

   return_values = await asyncio.gather(group1, square(20), square(30))
   print(return_values)
   print('Finished')

asyncio.run(main())

The Future objects

The Future object returned by asyncio.gather() is a typical asyncio.Future object. These objects contains some methods which we can call to get the state and generally interact with the object.

The Future.cancel() method removes the Future object from execution schedule. After calling this method, the object becomes unusable as we cannot await it after.

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   obj = asyncio.gather(square(10), cube(10))
   obj.cancel()
   await obj  #this will raise an error

asyncio.run(main())

The Future.done() method can be used to tell whether the Future object is done with the execution.

import asyncio

async def square(num):
   return int(num) ** 2
   await asyncio.sleep(0.001) #simulate a small delay

async def cube(num):
   return int(num) ** 3
   await asyncio.sleep(0.001) #simulate a small delay

async def main():
   obj = asyncio.gather(square(10), cube(10))
   print('done: ', obj.done())
   print(await obj)
   print('done: ', obj.done())

asyncio.run(main())