Choosing and Debugging Concurrency

New words in this topic, in plain English. Come back here whenever one feels fuzzy.

Bottleneck

The slowest part of a program, the one that limits total speed.

SDK

A library that a service provider (for example an AI company) gives you so you can call its API from Python without writing the web requests yourself.

Backoff

Waiting longer after each failed retry, so a struggling server gets time to recover.

Jitter

A small random extra delay so many clients do not retry at the same instant.

Rate limit (HTTP 429)

The response a server sends when you ask too often.

Debug mode

A mode in asyncio that logs slow callbacks and coroutines that were never awaited.

Fire-and-forget

Starting a task and never looking at its result. Errors in it get lost.

Back-pressure

Slowing producers when consumers cannot keep up.

Structured concurrency

Tasks must finish inside the block that started them.

1. 1

   Break the pipeline on purpose

   In the pipeline change asyncio.timeout(0.15) to asyncio.timeout(0.005) and predict which prompts now fail. Even fast takes 0.01 s inside the model, so it times out as well. Timeouts must be set from real response times, with room to spare.

2. 2

   Shrink the semaphore

   Change Semaphore(3) to Semaphore(1). The three prompts now run one at a time, so the printed lines come in a different order. Notice that results are still returned in prompt order: gather keeps that, whatever the execution order.

3. 3

   Lose a task

   In the second example replace the safe pattern with a bare asyncio.create_task(boom()) and a await asyncio.sleep(0.05). On your computer you will see Task exception was never retrieved; in the browser the message is worded differently. Either way, the error was never handled by your code.

Errors are the best teachers. Make each change, read the error, guess what went wrong, then reveal the answer.

import asyncio


async def boom():
    await asyncio.sleep(0.01)
    raise RuntimeError("lost in the background")


async def main():
    asyncio.create_task(boom())            # no reference kept, never awaited
    await asyncio.sleep(0.05)
    print("main finished")


asyncio.run(main())

1. 1

   Start with the bottleneck, not the tool: is the program waiting or calculating? Measure first (time.perf_counter, a profiler, Topic 13.7). The question "would a faster network help, or a faster CPU?" settles most cases (Topic 12.1).

2. 2

   Decision guide. Many network or API calls (an LLM batch, a crawler, a service that calls tools): asyncio with a Semaphore (a limit on how many run at once, Topic 12.7). A blocking library you cannot change, a handful of calls: a thread pool or to_thread. CPU-bound pure Python on many items: a process pool. Numeric work on arrays: NumPy (it uses optimised C code, and often many cores). GPU work: PyTorch. One slow step that nobody else waits on: usually none of these.

3. 3

   AI workloads. Calling a hosted model 1,000 times: asyncio, a semaphore sized to the rate limit, a timeout on each call, retries with backoff for transient errors (429 and 5xx). Embedding documents: batch the texts into fewer calls first (fewer round trips beats more concurrency). Streaming to a user: an async generator. Running a model on your own machine: the framework handles the cores and the GPU; your Python only feeds it.

4. 4

   Common bugs. (1) Blocking the event loop (time.sleep, requests, a heavy loop) inside a coroutine. (2) A missing await. (3) Tasks whose exception nobody reads, or whose reference was dropped. (4) Swallowing CancelledError. (5) No timeout, so one stalled call hangs everything. (6) Unbounded concurrency that triggers rate limits or exhausts memory. (7) Shared mutable state between threads without a lock. (8) Starting processes without the main guard or with unpicklable arguments.

5. 5

   Debugging tools. Asyncio debug mode (PYTHONASYNCIODEBUG=1, or python -X dev, or asyncio.run(main(), debug=True)) logs coroutines that were never awaited and callbacks that block the loop for more than 0.1 seconds. Naming your tasks (create_task(coro, name="fetch-1")) makes tracebacks and asyncio.all_tasks() readable. Log with timestamps, task names and thread names. Keep test timings out of assertions: assert on order and results, as the examples in this phase do.

6. 6

   Structured concurrency is the habit that prevents most of these bugs: start tasks inside a TaskGroup so their lifetime is the lifetime of a block, give every call a timeout, and let errors travel up instead of being swallowed. You will use these skills again in Phase 15 (web frameworks that run on the event loop) and Phase 16 (LLM clients, agents that call several tools at once).

• ↔

  Retries: raise the success rate for transient failures but multiply load and cost on a struggling server; always cap tries and back off.

• ↔

  One shared limit vs per-service limits: a single semaphore is simple; separate ones stop one slow service from starving the others.

• ↔

  Debug mode: catches blocking and unawaited coroutines early, but slows the program, so keep it for development.

• Done when you can choose asyncio, threads, processes or NumPy for a described workload and give the reason.

• Done when you can build an async pipeline with a semaphore, a timeout, retries and gathered results.

• Done when you can list the common async bugs and how to spot each.

• Done when you can keep references to background tasks and explain why.

• Done when you can turn on asyncio debug mode and read its warnings.