asynciorun cannot be called from a running event loop when using Jupyter Notebook

Navigating asynchronous programming in Python can be incredibly powerful, especially for I/O-bound tasks that require efficient handling of many operations concurrently. However, a common roadblock many developers encounter, particularly when working in interactive environments like Jupyter Notebooks, is the error message: “asyncio.run() cannot be called from a running event loop”. This seemingly cryptic message often signals a fundamental misunderstanding of how asyncio manages its event loop and how that conflicts with the persistent nature of Jupyter’s execution model. This article will demystify this error, explain the underlying mechanisms, and provide robust, practical solutions to ensure your asynchronous code runs smoothly within your Jupyter Notebook sessions, empowering you to leverage the full potential of non-blocking I/O without the frustration.

Understanding the Asyncio Event Loop and Its Role

At the heart of Python’s asyncio library lies the event loop, a central orchestrator that manages and executes asynchronous tasks. Think of it as a single thread that continuously monitors for events – such as data becoming available on a socket, a file being ready for reading, or a timer expiring – and then dispatches the corresponding asynchronous operations (coroutines) to handle them. This non-blocking approach allows your program to perform many tasks concurrently without resorting to traditional multi-threading, which can introduce significant overhead and complexity due to the Global Interpreter Lock (GIL).

When you define an async def function, you’re creating a coroutine, a special type of function that can be paused and resumed. The await keyword is crucial here; it tells the event loop that the current coroutine can yield control to another coroutine while it waits for an operation (like a network request) to complete. Once the awaited operation is done, the event loop resumes the paused coroutine from where it left off. This cooperative multitasking is what makes asyncio so efficient for I/O-bound operations, allowing a single thread to handle thousands of concurrent connections.

asyncio.run() is the primary function used to execute the top-level entry point of an asyncio program. It’s designed to manage the event loop from start to finish: it creates a new event loop, runs the specified coroutine until it completes, and then closes the loop. This lifecycle is intended for standalone scripts where the event loop has a clear beginning and end. According to the official Python documentation, “asyncio.run() simplifies the common use case of running an asyncio program, handling the creation and closing of the event loop automatically.” This explicit lifecycle management is where the conflict with interactive environments like Jupyter Notebooks arises, as we’ll explore next.

Why Jupyter Notebooks Conflict with asyncio.run()

The core reason you encounter the “asyncio.run() cannot be called from a running event loop” error in Jupyter Notebooks stems from the fundamental difference in their execution models. Jupyter Notebooks operate on a persistent kernel that maintains state across cell executions. When you run a cell, that code executes within the existing environment. If you run asynchronous code that implicitly or explicitly starts an asyncio event loop, that loop continues to run in the background after the cell finishes. The problem then arises when you try to call asyncio.run() in a subsequent cell.

asyncio.run() is designed to create and manage its own fresh event loop. If an event loop is already active in the current thread (as is often the case in a persistent Jupyter kernel), asyncio.run() detects this and throws the infamous RuntimeError. This isn’t a bug; it’s a safety mechanism to prevent unexpected behavior from trying to manage multiple concurrent event loops in a way asyncio.run() isn’t designed for. Jupyter’s IPython kernel itself often uses an event loop internally for its own operations, further complicating matters.

Consider a scenario where you’re building a web scraper or an API client using asyncio. In a standalone script, you’d call asyncio.run(main_coroutine()) once. In Jupyter, if your first cell initiates an async operation, it might start an implicit event loop. Then, if your second cell tries to run another async function using asyncio.run(), the clash occurs because the kernel’s event loop is still active. This creates a challenging situation for developers who want to experiment with or iterate on asynchronous code incrementally within the interactive environment.

The key takeaway is that Jupyter’s interactive nature means the environment (including any existing event loops) persists between cell executions. asyncio.run() expects a clean slate, leading to the conflict. Understanding this fundamental difference is the first step towards implementing effective solutions.

Practical Solutions for Asynchronous Code in Jupyter ----------------------------------------------------

When faced with the “asyncio.run() cannot be called from a running event loop” error, there are several effective strategies to integrate asyncio seamlessly into your Jupyter Notebook workflow. The most widely adopted and recommended solution for allowing nested event loops, especially in interactive environments, is the nest_asyncio library. This package patches asyncio to permit calling asyncio.run() (or other loop-starting functions) even when an event loop is already running.

To use nest_asyncio, simply install it via pip (pip install nest_asyncio) and then apply it at the beginning of your notebook session, typically in the first cell:

import nest_asyncio nest_asyncio.apply()

Once nest_asyncio.apply() has been called, you can use asyncio.run() in any subsequent cell without encountering the RuntimeError. This elegant solution makes your Jupyter experience with asyncio much smoother. For example, if you have an asynchronous function:

async def fetch_data(): Simulate an I/O bound operation print("Fetching data...") await asyncio.sleep(2) print("Data fetched!") return "Some data" Now you can run it directly: result = asyncio.run(fetch_data()) print(result)

This will now execute correctly. Another approach, if you prefer not to use nest_asyncio, is to directly obtain and manage the currently running event loop, if one exists. You can check if a loop is running using asyncio.get_event_loop().is_running() and then use loop.run_until_complete() to execute your coroutine. However, nest_asyncio simplifies this process significantly, making it the preferred method for most users.

Here’s a breakdown of the steps for using nest_asyncio:

Install nest_asyncio: Open your terminal or command prompt and run pip install nest_asyncio.
Import and Apply: In the very first cell of your Jupyter Notebook, add these two lines: ``` import nest_asyncio nest_asyncio.apply()
Run Your Async Code: You can now use asyncio.run(your_coroutine()) or await coroutines directly if you’re in an async-aware environment (like IPython’s top level await functionality) in any subsequent cell without issues.

This strategy allows for seamless integration and debugging of asynchronous Python code within the interactive and stateful environment of Jupyter Notebooks, transforming a frustrating error into a minor configuration step. More details on nest_asyncio can be found on its PyPI project page.

Best Practices for Asynchronous Development in Interactive Environments

While nest_asyncio offers an excellent solution for the immediate problem, adopting a few best practices can further enhance your asynchronous development workflow in Jupyter Notebooks and other interactive environments. These practices focus on code organization, resource management, and understanding the nuances of asynchronous execution. Question & Answer :

I would like to use asyncio to get webpage html.

I run the following code in jupyter notebook:

import aiofiles import aiohttp from aiohttp import ClientSession async def get_info(url, session): resp = await session.request(method="GET", url=url) resp.raise_for_status() html = await resp.text(encoding='GB18030') with open('test_asyncio.html', 'w', encoding='utf-8-sig') as f: f.write(html) return html async def main(urls): async with ClientSession() as session: tasks = [get_info(url, session) for url in urls] return await asyncio.gather(*tasks) if __name__ == "__main__": url = ['http://huanyuntianxiazh.fang.com/house/1010123799/housedetail.htm', 'http://zhaoshangyonghefu010.fang.com/house/1010126863/housedetail.htm'] result = asyncio.run(main(url))

However, it returns RuntimeError: asyncio.run() cannot be called from a running event loop

What is the problem?

How to solve it?

The asyncio.run() documentation says:

This function cannot be called when another asyncio event loop is running in the same thread.

In your case, jupyter (IPython ≥ 7.0) is already running an event loop:

You can now use async/await at the top level in the IPython terminal and in the notebook, it should — in most of the cases — “just work”. Update IPython to version 7+, IPykernel to version 5+, and you’re off to the races.

Therefore you don’t need to start the event loop yourself and can instead call await main(url) directly, even if your code lies outside any asynchronous function.

Modern Jupyter lab/notebook

Use the following for newer versions of Jupyter (IPython ≥ 7.0):

async def main(): print(1) await main()

Python or older IPython

If you are using Python ≥ 3.7 or IPython < 7.0, use the following:

import asyncio async def main(): print(1) asyncio.run(main())

That’s also that form you should use if you are running this in a python REPL or in an independent script (a bot, a web scrapper, etc.).

If you are using an older version of python (< 3.7), the API to run asynchronous code was a bit less elegant:

import asyncio async def main(): print(1) loop = asyncio.get_event_loop() loop.run_until_complete(hello_world())

Using `await` in your code

In your case, you can call await main(url) as follows:

url = ['url1', 'url2'] result = await main(url) for text in result: pass # text contains your html (text) response

This change to recent versions of IPython makes notebook code simpler and more intuitive for beginers.

Further notices

Few remarks that might help you in different use cases.

Jupyter vs. IPython caution

There is a slight difference on how Jupyter uses the loop compared to IPython.

[…] IPykernel having a persistent asyncio loop running, while Terminal IPython starts and stops a loop for each code block.

This can lead to unexpected issues.

Google Colab

In the past, Google colab required you to do more complex loop manipulations like presented in some other answers here. Now plain await main() should just work like in IPython ≥ 7.0 (tested on Colab version 2023/08/18).

Python REPL

You can also run the python REPL using the asyncio concurrent context. As explained in asyncio’s documentation:

$ python -m asyncio asyncio REPL ... Use "await" directly instead of "asyncio.run()". >>> import asyncio >>> await asyncio.sleep(10, result='hello') 'hello'

The asyncio REPL should be available for python ≥ 3.8.1.

When does `asyncio.run` matters and why?

Older versions of IPython were running in a synchronous context, which is why calling asyncio.run was mandatory.

The asyncio.run function allows to run asynchronous code from a synchronous context by doing the following:

starts an event loop,
runs the async function passed as argument in this (new) event loop,
stops the event loop once the function returned

In more technical terms (notice how the function is called a coroutine):

This function runs the passed coroutine, taking care of managing the asyncio event loop, finalizing asynchronous generators, and closing the threadpool.

What happen when using `await` in synchronous context?

If you happen to use await in a synchronous context you would get the one of the following errors:

SyntaxError: 'await' outside function
SyntaxError: 'await' outside async function

In that case that means you need to use asyncio.run(main()) instead of await main().