The gather()
function in the asyncio
library is used to group multiple tasks/coroutines into a single awaitable object. When this object is awaited, the execution will wait until each of the grouped coroutines is fully executed.
The gathered tasks are executed simultaneously in a non-blocking manner, this means that if a delay is encountered in one task, execution can move on to the other task and then return later on to the coroutine where the delay was encountered.
Let us consider a basic example:
import asyncio
async def display_nums():
for n in range(5):
print(n)
await asyncio.sleep(0.001) #simulate a small delay
async def display_letters():
for let in "ABCDE":
print(let)
await asyncio.sleep(0.001) #simulate a small delay
async def main():
print("Started")
await asyncio.gather(display_nums(), display_letters()) #call the gather function with the two tasks
print('Finished')
asyncio.run(main())
If you observe the output of the above snippet, you will see that the small delay caused by the asyncio.sleep()
calls is enough to make the execution to alternate between the two gathered coroutines.
The asyncio.gather()
function takes an arbitrary number of coroutines as arguments. It creates a task for each coroutine in an event loop and then returns an awaitable object which we can await inside of an async function. As we have seen above, the gathered tasks gets executed simultaneously in a non-blocking manner.
How asyncio.gather() works
The function takes one or more awaitables as argument(s). Thus the basic syntax is:
asyncio.gather(coro1(), coro2(), coro3(), ...)
The function also takes one optional argument called return_exceptions
, thus the extended syntax is:
asyncio.gather(*coros, return_exceptions = False)
The function returns a Future
object, which is itself awaitable.
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
result = asyncio.gather(square(10), cube(10), square(11), cube(11), return_exceptions = True)
print(result)
asyncio.run(main())
Note that in the above example, we did not await the Future
object and therefore, the grouped tasks are not executed. When the object is awaited, it ensures that all the gathered tasks are fully executed and it finally returns a list with their return values in the order of input.
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
print("Started")
return_values = await asyncio.gather(square(10), cube(10), square(11), cube(11))
print(return_values)
print('Finished')
asyncio.run(main())
As shown above, the awaited Future
object returns a list whose elements are the the values returned by the coroutines.
If an error is encountered in the execution of a coroutine, an exception is raised and the execution is terminated. However, if the return_exceptions
is set to True
, the raised exception will be added in the results in the place of the involved coroutine's return value, and the execution will continue with the remaining coroutines.
with return_exceptions set to False
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
print("Started")
return_values = await asyncio.gather(square(10), cube(10), square('eleven'), cube(11))
print(return_values)
print('Finished')
asyncio.run(main())
with return_exception set to True
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
print("Started")
return_values = await asyncio.gather(square(10), cube(10), square('eleven'), cube(11), return_exceptions = True)
print(return_values)
print('Finished')
asyncio.run(main())
In the above examples, the third coroutine in the gather()
function raises an exception because 'eleven' cannot be cast into an integer.
In the first snippet, the capture_exceptions
parameter is False
and hence the exception is raised an the execution is halted.
In the second case, the capture_exceptions
is set to True
and hence the exception is captured and entered in place of the last coroutine's return value. The execution exits successfully without the exception being propagated to the callers.
gather() with other awaitables
We have already seen how we can use asyncio.gather()
with coroutine. Apart from coroutines, asyncio.gather()
can work with any other awaitable object such as tasks and future objects.
This means that we can also use an object returned by gather()
inside of another asyncio.gather()
call. Consider the following example:
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
print("Started")
group1 = asyncio.gather(square(10), cube(10))
group2 = asyncio.gather(square(20), cube(20))
return_values = await asyncio.gather(group1, group2)
print(return_values)
print('Finished')
asyncio.run(main())
We can as well combine coroutines with gather objects and any other awaitable object.
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
print("Started")
group1 = asyncio.gather(square(10), cube(10))
return_values = await asyncio.gather(group1, square(20), square(30))
print(return_values)
print('Finished')
asyncio.run(main())
The Future objects
The Future
object returned by asyncio.gather()
is a typical asyncio.Future
object. These objects contains some methods which we can call to get the state and generally interact with the object.
The Future.cancel()
method removes the Future
object from execution schedule. After calling this method, the object becomes unusable as we cannot await it after.
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
obj = asyncio.gather(square(10), cube(10))
obj.cancel()
await obj #this will raise an error
asyncio.run(main())
The Future.done()
method can be used to tell whether the Future
object is done with the execution.
import asyncio
async def square(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 2
async def cube(num):
await asyncio.sleep(0.001) #simulate a small delay
return int(num) ** 3
async def main():
obj = asyncio.gather(square(10), cube(10))
print('done: ', obj.done())
print(await obj)
print('done: ', obj.done())
asyncio.run(main())