Backpressure in software: what it is and how to apply it

When a producer is faster than its consumer, something has to give. Backpressure is the set of mechanisms by which the consumer makes the producer slow down, instead of the system falling over.

Originally written on 16 March 2020. Migrated from a WordPress blog and reformatted.

In any pipeline, a stage that receives input faster than it can process it has three options: buffer the excess, drop it, or make the sender slow down. The third is backpressure. It is the only option that preserves both data and stability, and it is the one most systems lack until an incident makes the case for it.

What happens without it

A producer writes to an unbounded queue. The consumer processes at a fixed rate. Under normal load the queue is empty. When load exceeds the consumer's rate, the queue grows. Since it is unbounded, it grows until memory is exhausted, at which point the consumer crashes, the queue is lost, and the producer, which never received any signal, continues writing into a process that no longer exists.

The failure happened at the consumer. The cause was at the producer. Nothing connected them.

Bounded buffers

The first step is a bound. A queue with a maximum size forces a decision when it is full, and that decision is where backpressure lives.

queue = asyncio.Queue(maxsize=1000)

async def producer():
    while True:
        item = await source.read()
        await queue.put(item)      # blocks when the queue is full

put blocks when the queue holds 1,000 items. The producer stops reading from its source. If the source is a socket, the kernel's receive buffer fills, TCP advertises a zero window, and the remote sender stops transmitting. Backpressure has propagated from the slow consumer all the way to the origin without any explicit coordination, because every stage between them blocks when its buffer is full.

This is the essential mechanism: blocking on a bounded buffer propagates upstream. Everything else is a variant.

Where blocking is not available

Asynchronous message brokers. Kafka consumers pull, so a slow consumer simply lags and the broker retains messages up to its retention limit. The backpressure signal is consumer lag, and the producer does not see it unless something reads the lag metric and acts on it. RabbitMQ pushes, and applies flow control by pausing publishers when queue memory exceeds a threshold.

HTTP services. A service cannot block its callers. It can reject: return 429 or 503 with a Retry-After header, and have callers back off. A load balancer with a bounded connection pool per backend does the same thing one layer up. Rejecting early, at the edge, is preferable to accepting a request and timing out on it later, because the rejected request costs nothing.

Reactive streams. Libraries following the Reactive Streams specification (Reactor, RxJava, Akka Streams) make the signal explicit: the subscriber calls request(n) to indicate how many items it can accept, and the publisher may not send more. The buffer bound is negotiated rather than fixed.

Choosing what to do when full

Blocking is correct when the source can wait. It is incorrect when the source is a real-time feed that cannot pause, such as sensor data or a market feed. In that case the choice is between dropping and sampling:

• Drop newest keeps the buffer's existing contents; the incoming item is discarded. Appropriate when older items are more valuable (a queue of commands).
• Drop oldest discards the head to make room. Appropriate when only recent state matters (a stream of positions or prices).
• Sample keeps one item per interval. Appropriate for telemetry.

The choice is a product decision, and it should be explicit in the code rather than a side effect of an out-of-memory error.

Detecting the need

Queue depth over time is the diagnostic. A depth that returns to zero between bursts is healthy. A depth that trends upward is a consumer that cannot keep up, and the question is whether to add consumers, slow the producer, or drop. A depth that is always zero with an idle consumer means the bound is not being reached and backpressure has not been tested; it will be tested in production.

Summary

Bound every buffer. Decide, for each one, what happens when it fills: block, reject, or drop, and which items. Make the decision visible in the code. Monitor depth. A system with these properties degrades under overload; a system without them fails.