The queue that never stops growing#
Picture a service with three workers. Each worker handles one request at a time and finishes it in one tick. So the service can finish three requests per tick — that is its service rate, the ceiling on how fast it can drain work. Nothing you do to the queue changes that number; three workers finish three requests per tick and no more.
Requests arrive and wait in a queue until a worker is free. While they arrive at three per tick or fewer, the queue stays empty or nearly so: a worker is always ready. The queue is doing its job as a shock absorber, smoothing the moments when four happen to land at once.
Now the load rises to five requests per tick and stays there. Five arrive, three leave. Every tick, two more requests pile up that never get worked off. After ten ticks the queue is twenty deep; after a hundred ticks it is two hundred deep. The workers are as busy as they can be, yet the backlog grows without limit, because arrivals are simply larger than the service rate and nothing is stopping the difference from accumulating.
The queue depth is not the part that hurts. The wait is. A request that lands when the queue is two hundred deep sits behind two hundred others, so it waits about sixty-six ticks before a worker even looks at it. By then the user has refreshed the page, and their browser has sent the request again — which lands at the back of an even longer queue. The service is now doing enormous amounts of work whose answers arrive long after anyone is still waiting for them. This is how an overloaded service topples: not from the load, but from the unbounded wait the load creates.
Bound the queue, shed the rest#
TL;DRthe 30-second version
- A queue absorbs short bursts, but it cannot raise the service rate. Under sustained overload the queue only grows, and the wait grows with it.
- The fix is a bounded queue: cap how many requests may wait, and reject new arrivals the instant it is full. That rejection is load shedding.
- Shedding keeps the accepted work fast. The excess is turned away in microseconds instead of sitting in a queue for seconds.
- Backpressure is the same idea aimed upstream: signal the sender to slow down instead of silently swallowing more than you can handle.
The first move is to put a limit on the queue. Say the queue may hold at most six requests. Now when the seventh request arrives and the queue is full, the service does not buffer it. It rejects it immediately, before doing any real work, and moves on. Rejecting excess work at the door under overload is load shedding.
Watch what that does to the wait. With the queue capped at six and three workers draining it, a request never sits behind more than six others, so it waits at most about two ticks — no matter how hard the overload pushes. The accepted requests stay fast and predictable. The requests beyond the cap are turned away, but a fast rejection is a far kinder answer than a response that arrives a minute too late. The caller learns immediately and can retry, back off, or try another replica.
Doing the rejection early — at the edge, the moment a request arrives, before it is parsed or authenticated or routed — is admission control. The earlier you shed, the cheaper the rejection, and the more of the server's capacity stays free for the requests you did accept. Shedding a request after you have already spent effort on it wastes the very capacity you are trying to protect.
Backpressure is the same principle pointed in the other direction. Instead of quietly dropping what you cannot handle, you send a signal back up the chain: slow down. A full queue that refuses new work is a form of backpressure — the sender feels the resistance and eases off. TCP does exactly this at the network layer: the receiver advertises how much buffer space is left, and the sender is not allowed to send past it. The pushback flows upstream so the fast producer matches the slow consumer, and nothing has to be thrown away at all.
PredictUnder a steady overload of five arrivals per tick against three workers, you double the queue capacity from six to twelve. What happens to the worst wait an accepted request suffers?
Hint: The workers still finish three per tick no matter how long the line is.
It roughly doubles. A bigger queue does not add workers, so the drain rate is still three per tick — a request can now sit behind up to twelve others instead of six, so its wait grows from about two ticks to about four. The bigger buffer accepts a few more requests but makes every one of them slower. Capacity is not the lever; the service rate is.
Why the wait tracks the queue#
There is a simple law behind all of this, and it is worth carrying in your head. Little's Law says that the average number of items waiting in a system equals the average arrival rate multiplied by the average time each item spends there. Written out: items-in-system equals arrival-rate times time-in-system.
Rearrange it and you get the wait directly: the time in the system equals the number of items in it divided by the rate they leave. If forty requests are in the queue and the workers clear three per tick, the wait is about thirteen ticks. This is why an unbounded queue is an unbounded wait — as the number in the system climbs with no ceiling, the time each one spends climbs with it. And it is why a bounded queue bounds the latency: cap the number waiting and you have capped the wait, because the drain rate is fixed. The same law tells you where to set the cap. Pick the longest wait you are willing to serve, multiply by the service rate, and that is your queue size.
The overload toolkit: deadlines and serve-order#
Bounding the queue handles how much work you hold. Two more tools handle which held work is worth doing, and they matter most exactly when you are overloaded.
The first is a deadline. Every request has a moment past which its answer is useless — the caller has already timed out, or the user has left. If a request has sat in the queue longer than that deadline, there is no point spending a worker on it: the reply will arrive too late to matter. So you drop it, unserved, and free the slot for work that can still finish in time. Dropping work whose answer is already too late is often more valuable than shedding at the door, because it protects the workers from grinding through a backlog of stale requests while fresh ones pile up behind them.
The second tool is the serve order, and it is the surprising one. The obvious order is first-in-first-out: serve the oldest request first, because it has waited longest and that feels fair. Under normal load, FIFO is right. But under a sustained overload, FIFO is a trap. The oldest request is the one most likely to have already blown past its deadline, so FIFO spends its workers on the requests least likely to still be useful, while the freshest requests — the ones that could actually finish in time — wait at the back and grow stale in turn.
Flipping to last-in-first-out under overload fixes this. Serving the newest request first means the workers spend their limited effort on the requests most likely to still beat their deadline. The old requests at the bottom do get abandoned, but they were probably doomed anyway. The result is that more of the requests you finish actually arrive in time to be useful. You still shed the same amount of work — the service rate has not changed — but you shed the work that was already lost instead of the work you could have saved.
The numbers that decide it#
The whole story is captured by three rates and one cap. Let arrivals be the requests coming in per tick and service be what the workers finish per tick.
| Condition | What the queue does | What the wait does |
|---|---|---|
| arrivals ≤ service | stays shallow; absorbs short bursts and drains | flat and small |
| arrivals > service, no cap | grows by (arrivals − service) every tick, forever | grows without limit |
| arrivals > service, capped at C | fills to C and holds; the excess is shed | flat, at most about C ÷ service |
The middle row is the failure mode. The gap between arrivals and service is a constant leak into the queue, and a constant leak with no drain and no cap is an unbounded quantity. The third row is the fix: the cap turns the unbounded backlog into a fixed one, and by Little's Law a fixed backlog draining at a fixed rate is a fixed wait. That is the entire value of shedding stated as arithmetic — it trades a growing latency for a fixed rejection rate.
Rejecting a request is also far cheaper than serving one, and that asymmetry is what makes shedding work under real overload. A rejection is a size check and a returned error — microseconds. Serving is parsing, database calls, and rendering — milliseconds. So a server that is drowning can still say no to thousands of requests per second while it carefully serves the few it accepted, which is precisely the behavior you want when the alternative is serving none of them.
Choosing how to fail#
Overload forces a choice, and there is no option where every request gets a fast, correct answer — the work simply exceeds the capacity. The design question is which failure you prefer.
- Shed: reject the excess fast. Accepted work stays quick; rejected callers get an immediate, honest error they can act on. The cost is that some requests are refused outright.
- Buffer: accept everything into a growing queue. No request is refused at the door, but every request slows down, and past a point the whole service becomes unresponsive. This is the option that looks kindest and fails hardest.
- Degrade: serve a cheaper, worse answer to everyone. Search a cached subset instead of the full index; skip the personalized ranking. Throughput rises because each response costs less, at the price of quality.
- Push back: signal the sender to slow down, so the excess is never generated. The cleanest option when you control the sender — a batch job, an internal client — and impossible when you do not, like open internet traffic.
Real systems layer these. A well-behaved service pushes back on clients it controls, sheds the excess it cannot push back on, degrades to a cheaper response as it approaches its ceiling, and only serves outright errors when even the degraded path is saturated. The ordering matters: you reach for the gentlest tool that keeps you responsive, and escalate only as the load demands.
Backpressure vs its neighbors#
Several patterns sit near backpressure and are easy to confuse with it. They solve related but distinct problems, and a real system usually runs all of them together.
| Pattern | Question it answers | How it differs |
|---|---|---|
| Backpressure / shedding | I am overloaded right now — what do I do with the excess? | Reacts to the live queue: reject or push back on work you cannot finish in time. |
| Rate limiting | How much may this client send, by policy? | A fixed budget set in advance, per caller — a fairness and abuse rule, not a reaction to your current load. |
| Circuit breaker | A dependency I call is failing — should I stop calling it? | Protects you from a broken downstream by failing fast outward, rather than protecting you from too much inbound work. |
| Autoscaling | The load is genuinely higher — should I add capacity? | Raises the service rate over seconds or minutes. Backpressure is what keeps you alive during the gap before the new capacity is ready. |
The clean way to hold them apart: rate limiting is a budget decided ahead of time, backpressure is a reaction to the queue in front of you right now, the circuit breaker guards the calls you make to others, and autoscaling changes the ceiling itself. Backpressure is the only one of the four that assumes the extra load is real and here, and that you must survive it with the capacity you have this instant.
In production
Google's SRE book devotes its Handling Overload chapter to exactly this progression: graceful degradation, then load shedding, then — at the extreme — serving outright errors when even a degraded response cannot be produced. Its companion chapter on cascading failures traces the mechanism this page opened with: a server tips into overload, its queue and latency climb, client retries pile more load onto an already-drowning server, and the failure spreads to healthy machines. Their guidance is to shed early and to make clients back off, so a local overload cannot cascade into a global outage.
Netflix built an adaptive version. Rather than hand-tuning a queue size, their concurrency-limits library borrows from TCP congestion control: it watches request latency and, when latency starts to rise, infers that the queue is building and lowers the concurrency limit automatically — shedding more. The connection to Little's Law is direct; the ideal concurrency limit is roughly the request rate multiplied by the minimum latency, and the library tracks it as conditions change so nobody has to guess the right cap up front.
Facebook's Fail at Scale describes the deadline and serve-order tools in production. They apply a controlled-delay idea borrowed from network queue management: measure how long requests are sitting in the queue, and when that sojourn time stays above a small target — on the order of a few milliseconds — for a sustained interval, start shedding. Paired with it is adaptive LIFO, which serves newest-first only once the queue signals real overload, so fresh requests beat their deadlines while the doomed backlog is let go.
Common mistakes
- An unbounded queue anywhere. Every in-memory queue, thread pool, and channel needs a bound. One unbounded buffer in the chain is the single most common cause of a service that runs out of memory under load instead of shedding cleanly.
- Shedding late. Rejecting a request after you have parsed, authenticated, and routed it burns the very capacity you are protecting. Shed at the edge, before the expensive work.
- Retries without backoff. A client that immediately retries a shed request just re-adds the load you shed. Shedding only helps if callers back off; otherwise you are rejecting the same request over and over.
- Confusing a full queue with a fast one. A queue that is always full is not a healthy buffer — it means you are permanently overloaded and the queue is pure added latency. A healthy queue is usually near empty and only fills during bursts.
- Serving stale work under overload. Finishing a request whose caller already timed out is wasted effort. Without deadlines, a backlog of stale requests can keep your workers fully busy producing answers nobody will read.
In an interview
When a design question turns to overload — a spike, a thundering herd, a hot partition — the interviewer wants to hear that you will bound your queues and shed rather than buffer. Say it plainly: a queue smooths bursts but cannot raise the service rate, so under sustained overload it must be bounded, and the excess shed fast so accepted work stays responsive.
- Lead with the ceiling: name the service rate and point out that no queueing choice raises it — the only real fixes for more load are more or faster workers.
- Bound every queue and shed at the edge, cheaply, before expensive work. Reach for Little's Law to size the cap: the wait you will tolerate times the service rate.
- Add deadlines so workers never grind through stale requests, and mention adaptive LIFO as the trick for serving newest-first under overload so fresh requests beat their deadlines.
- Separate the neighbors cleanly: rate limiting is a preset per-client budget, the circuit breaker guards your outbound calls, autoscaling raises the ceiling — backpressure is the live reaction that keeps you alive until it does.
- Close the loop with retries: shedding only works if clients back off, so pair it with exponential backoff and jitter to avoid a retry storm.
Isn't dropping requests just failing?
A fast rejection under overload is a better outcome than a slow success that arrives after the caller gave up — and far better than the whole service becoming unresponsive. Shedding is choosing to fail a few requests quickly so the rest succeed quickly, instead of failing all of them slowly.
Backpressure or load shedding — which do I use?
Backpressure when you control the sender and can make it slow down: internal clients, batch jobs, a stream you consume. Load shedding when you cannot, like open internet traffic that will keep arriving no matter what you signal. Most systems use both, at different edges.
How big should the queue be?
Small. Size it from the longest wait you are willing to serve: that wait times the service rate, via Little's Law. A large queue does not help throughput — it only adds latency before you finally shed, so bounding it small keeps accepted work fast.
References
- Google SRE Book — Handling Overload — Graceful degradation, load shedding, and client-side throttling as the response to overload.
- Google SRE Book — Addressing Cascading Failures — How overload, growing queues, and retries cascade from one server into a global outage.
- Netflix — Performance Under Load (adaptive concurrency limits) — A TCP-congestion-control approach that infers overload from latency and sheds automatically.
- Netflix/concurrency-limits (library) — The open-source implementation, with the Little's Law relationship between limit, rate, and latency.
- Facebook — Fail at Scale (controlled delay + adaptive LIFO) — Queue-sojourn-based shedding and serving newest-first under sustained overload.
- Little's Law — The relation between items in a system, arrival rate, and time spent — the math behind bounding the wait.
Ready to try it?
The simulator is a real, deterministic implementation — pick a scenario and step through it, scrubbing the timeline forward and backward through every change.