Skip to content

Commit 8f42915

Browse files
committed
Document a pattern for slow consumers.
Fix #1656.
1 parent 1c7893a commit 8f42915

1 file changed

Lines changed: 33 additions & 0 deletions

File tree

docs/howto/patterns.rst

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,39 @@ terminates, after canceling the other task::
8787
for task in pending:
8888
task.cancel()
8989

90+
Multiple consumers
91+
------------------
92+
93+
You can run multiple consumer tasks to process incoming messages concurrently.
94+
This helps when a single consumer can't keep up with incoming messages.
95+
96+
Use :class:`~asyncio.Queue` to distribute messages to multiple consumers in a
97+
:class:`~asyncio.TaskGroup`::
98+
99+
import asyncio
100+
101+
async def worker(queue):
102+
while True:
103+
message = await queue.get()
104+
await consume_slowly(message)
105+
queue.task_done()
106+
107+
108+
async def run_workers(queue, num_workers=16):
109+
async with asyncio.TaskGroup() as task_group:
110+
for _ in range(num_workers):
111+
task_group.create_task(worker(queue))
112+
113+
async def handler(websocket):
114+
queue = asyncio.Queue()
115+
workers_task = asyncio.create_task(run_workers(queue))
116+
try:
117+
async for message in websocket:
118+
await queue.put(message)
119+
finally:
120+
await queue.join()
121+
workers_task.cancel()
122+
90123
Registration
91124
------------
92125

0 commit comments

Comments
 (0)