Stop Asyncio Subprocess Pipe Deadlocks Before They Stall Your Tooling

Choose safe asyncio subprocess patterns for captured output, streaming logs, pipe deadlocks, and shell-command security.

Developer Tools•~6 min read•November 10, 2026•By QZ-L Editorial
#Python#asyncio#subprocesses
Looking for related guides? Start with the qz-l Learning Center and explore more tutorials in this topic cluster.

An async command runner can look correct in a quick test and still freeze when a compiler, test runner, or helper starts producing lots of output. One common cause is a child process writing to a pipe that the parent is not reading.

The fix is not to add more await expressions. Choose how output should flow before choosing how to wait for the process: inherit it, capture a known-small amount with communicate(), or drain each piped stream as it arrives.

How waiting can deadlock

A pipe has finite capacity. If a child writes to a piped stdout or stderr faster than the parent reads, the pipe fills and the child blocks on its next write. If the parent is meanwhile awaiting proc.wait(), it is waiting for the child to exit; the child cannot exit because it is waiting for the parent to make room in the pipe.

This is a wait cycle, not necessarily a slow command. Python's asyncio documentation calls out this exact risk for Process.wait() when stdout or stderr is set to PIPE. It also warns that reading one stream while ignoring the other can block the child on the unread stream.

The first decision is whether you need to capture output at all:

  • If the caller does not need to inspect it, do not create pipes just to wait for an exit status. Let the process use the normal output streams for your application.
  • If output is small and has a known upper bound, use communicate().
  • If output may be large or unbounded, read it incrementally and process it without accumulating the whole stream in memory.

Use communicate() for bounded output

communicate() is the simplest safe choice when a command's output is reasonably small. It reads the configured stdout and stderr streams through EOF and waits for the process. The returned values are bytes; if a stream was not piped, its result is None.

import asyncio
import sys


async def run_small_command():
    proc = await asyncio.create_subprocess_exec(
        sys.executable,
        "-c",
        "print('child output')",
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )

    stdout, stderr = await proc.communicate()

    if proc.returncode != 0:
        message = (stderr or b"").decode(errors="replace")
        raise RuntimeError(f"Child failed: {message}")

    print((stdout or b"").decode(), end="")


asyncio.run(run_small_command())

The important part is that the parent drains both pipes while the child runs. Avoid replacing communicate() with await proc.wait() when either output stream is piped.

There is a trade-off: communicate() retains captured output in memory. It is not a streaming API, so it is a poor fit for verbose builds, long-running workers, or commands whose output size is not predictable. A child can emit enough data to exhaust the parent's memory even though the pipe deadlock is gone.

Stream large output in bounded chunks

For large output, start a reader for every piped stream and keep both readers running while the process executes. StreamReader.read(n) returns up to n bytes, so the application can work a chunk at a time instead of collecting the complete output.

This small example counts and discards the data after each chunk. It demonstrates concurrent draining without retaining the output; a real command runner would send chunks to a bounded log sink, parser, or rotating files.

import asyncio
import sys


async def count_bytes(reader):
    total = 0
    while chunk := await reader.read(64 * 1024):
        total += len(chunk)
    return total


async def run_chatty_command():
    child_code = (
        "import sys; "
        "sys.stdout.write('o' * 1000000); "
        "sys.stderr.write('e' * 1000000)"
    )
    proc = await asyncio.create_subprocess_exec(
        sys.executable,
        "-c",
        child_code,
        stdout=asyncio.subprocess.PIPE,
        stderr=asyncio.subprocess.PIPE,
    )

    if proc.stdout is None or proc.stderr is None:
        raise RuntimeError("Expected both output pipes")

    stdout_task = asyncio.create_task(count_bytes(proc.stdout))
    stderr_task = asyncio.create_task(count_bytes(proc.stderr))
    stdout_size, stderr_size = await asyncio.gather(
        stdout_task, stderr_task
    )
    returncode = await proc.wait()

    print(
        f"exit={returncode}, "
        f"stdout={stdout_size} bytes, stderr={stderr_size} bytes"
    )


asyncio.run(run_chatty_command())

Creating the tasks matters: simply defining two reader coroutines does not run them. Keep references to the tasks and await their completion. Reading stdout to EOF first and only then reading stderr is also unsafe: the child may fill stderr and stop before it closes stdout.

The example's 64 KiB read size limits each read operation; it is not a complete memory or storage policy. A production sink can itself become the bottleneck. If a synchronous write blocks the event loop, other async work stalls; if the sink cannot keep up, backpressure slows the child. Decide whether to use an async sink or move blocking writes off the event loop, and apply retention or rotation limits to files. Otherwise, streaming can move the resource problem from RAM to disk.

Backpressure is not automatically a failure: slowing a producer can be safer than buffering without a limit. If the application cannot tolerate that slowdown, choose a deliberate policy such as dropping low-priority output or stopping the child. An unbounded queue between a reader and a slow sink only moves the memory growth elsewhere.

For line-oriented output, readline() can be convenient, but it waits for a newline. A tool that emits one very long line can make a line-based consumer buffer more than expected. Fixed-size reads are easier to bound when output format or line length is not controlled.

Keep command arguments separate from shell syntax

Use asyncio.create_subprocess_exec(program, arg1, arg2, ...) for ordinary commands. It passes an executable and arguments separately, which avoids building a shell command string from values. Do not concatenate user input into a shell command. create_subprocess_shell() is for cases that actually require shell syntax; Python's documentation makes correct quoting the caller's responsibility.

This does not make an invoked program safe by itself. Validate values according to the program's own argument rules, avoid passing untrusted flags or paths without checking them, and do not treat output from the child as trusted data.

Treat timeouts as process-lifecycle decisions

Asyncio's subprocess wait() and communicate() methods do not take a timeout argument. The documentation points to asyncio.wait_for() for a deadline, but a timeout is only one part of the policy: decide whether the child should be terminated or killed, then wait for it to exit and ensure output-reader tasks are cleaned up.

Check the exit status after output has been drained. Decide whether a nonzero exit is returned, raised as an error, or handled by the caller; do not silently treat successful pipe reads as a successful command.

A quick decision checklist

Need Pattern
Only need the exit status; output need not be captured Avoid PIPE unless necessary
Need a small, bounded result Capture both relevant streams and use communicate()
Output can be large or unbounded Drain each piped stream concurrently in chunks and send it to a controlled sink

Async subprocess code is easiest to reason about when process lifetime and output lifetime are handled together. Know whether output is bounded, keep every pipe moving, and make timeout and storage behavior explicit. Those choices prevent a routine command from turning into a hung event loop or an unbounded memory or disk consumer.

References

Related Posts

Catch Secrets Before Merge: GitHub's New Copilot Review Path

Use GitHub's preview secret scanning in Copilot security review to catch credentials in code changes before they reach a pull request.

Docker Engine 29.10: Check Your Local Registry Configuration Before Upgrading

Docker Engine 29.10 changes loopback registry defaults. Learn how to migrate local registries without weakening transport security.

How to Spot a Fraud Email Link Before You Click

A practical guide to identifying phishing emails, suspicious links, and fake payment pages before they can steal your money or account access.