Design a Video Streaming Service

The origin works until an audience shows up. A popular video is being watched by a huge audience at once, spread across cities and countries. Every viewer is pulling a steady multi-megabit stream, continuously. Together they demand terabits per second of outbound bytes. One origin's link to the internet is orders of magnitude too small to push that. It does not slow down gracefully; past the first few thousand viewers, it simply cannot move the bytes. Where do the bytes come from?

Follow the obvious answer first, because it is what most people reach for: give the origin a bigger pipe. Put it on a fatter uplink, or add a few more origin servers behind a load balancer so there is more outbound bandwidth to go around. The attraction is real — it is the design you already have, scaled up. But the arithmetic kills it. Five hundred thousand people watching at once, each pulling about three megabits a second, is about one-and-a-half terabits per second — from one region. A single high-end serving box tops out around ninety to a hundred gigabits per second of outbound video. That regional demand is more than fifteen times what one box can push, and that is one mid-size region on one popular title. A bigger pipe is not fifteen times bigger; it is an order of magnitude short, and the gap only widens with the audience. Adding a handful of origin boxes does not close a fifteen-to-one gap, and it does nothing about the other problem hiding in the number: those viewers are spread across a continent, and serving all of them from one central location means every byte crosses the whole distance to reach them, slow to start and expensive to carry.

A second instinct is worth killing too, because it sounds sophisticated: put a cache in front of the origin. The trouble is that a plain key-value cache is built to answer lookups faster — it remembers the answer to a question so you do not recompute it. But the origin's problem is not that lookups are slow. It is that bytes cannot leave the building fast enough. A cache sitting next to the origin still has to push every one of those bytes out the same regional uplink. It caches the wrong thing. The ceiling here is bandwidth, not lookup speed, and no amount of remembering answers moves more bytes through a fixed-size pipe. So the answer is neither a bigger pipe nor a cache in one place — it is to spread the serving across many appliances, each close to its viewers, so the aggregate outbound rate is spread across the whole fleet and no single uplink is the ceiling.

A CDN edge tier of PoP caches close to the viewer, the origin serving only fills and misses. Stop serving the bytes from one place. Put the video on a tier of caches spread across the world, sitting close to where the viewers actually are — a CDN edge tier of points of presence, or PoPs. Each PoP is a cluster of serving appliances installed deep in the network, often inside the viewer's own internet provider. A viewer's player does not fetch segments from a distant origin; it fetches them from the nearest PoP, a short hop away. And here is the move that beats the arithmetic: the terabits of demand are no longer one number against one box. They are spread across hundreds of PoPs, each serving the viewers near it. That illustrative one-and-a-half terabits for one region becomes fifteen-to-seventeen appliances working in parallel inside that region, each pushing its ninety-to-a-hundred gigabits to the viewers on its doorstep — instead of one impossible origin pipe.

The origin does not disappear. Its job changes completely: it stops being in the viewer's path. Instead it fills the edge — it pushes copies of the videos out to the PoPs ahead of time, during off-peak hours — and it answers the rare miss, when a PoP is asked for a segment it does not happen to have cached yet: it fetches that one segment from the origin once, serves it, and keeps a copy for the next viewer. In steady state the overwhelming majority of bytes never touch the origin at all, and the hottest titles are served straight from each appliance's memory. The aha is that egress does not scale up; it scales out and outward — out across many appliances, and outward toward the viewer until the bytes are being served from inside the viewer's own network. The right question was never how do I push more bytes from here; it was how do I stop pushing them from here at all.

The edge serves bytes close to the viewer. But two questions the last use case set aside now have to be answered together. What exactly are those bytes — what unit does the edge cache and the player fetch? And how does a viewer on a weak, changing connection get a stream that stays smooth, when the single-server version could only offer one stored quality? A connection is strong at home, weak on a phone, and it changes minute to minute on the move. The service has one job: keep the video playing smoothly the whole time, sharp when the connection is good and never frozen when it is bad, without the viewer touching a quality setting.

Start with what breaks if you serve one fixed quality — you lose either way. Pick a high bitrate and the phone viewer stalls constantly, because their connection cannot pull the bytes fast enough. Pick a low bitrate and the fiber viewer gets a blurry stream on a connection that could have carried far better. And a viewer whose connection changes mid-video cannot be served correctly by any single fixed choice, because the right answer changed while they were watching. So the video has to exist at several qualities, and something has to get each viewer the right one — which the client-pull ladder does without any server-side per-viewer decision.

That leaves the choice of the ladder itself: how many rungs, at what bitrates, and cut how. The baseline is a fixed ladder — one set of bitrates applied to every title, the same rungs whether it is a dark, slow drama or a bright, fast sports clip. It is simple and it works, and it was the industry norm for years, with a top rung around 5.8k kilobits for 1080p. But it spends bits badly: a simple-to-compress title does not need the top rung's full bitrate to look identical, and a fixed ladder has no way to know that. The better default is a per-title, content-aware ladder: analyze each title and choose its rungs for its actual content, so an easy title gets a lower top-rung bitrate at the same perceived quality. On a real title that trimmed the top rung by about 20% for equal quality — bytes saved on every rung, on every segment, for every one of the millions of views. At this egress scale, a fifth off the bitrate is enormous.

A per-title, content-aware ABR ladder of renditions, cut into ~6-second segments, with a client-pull player. One quality is not enough; the video has to exist at several. This is the ABR ladder — ABR for adaptive bitrate. The source video is encoded into a set of renditions, each a full copy of the video at a different quality and bitrate: a top rung at high bitrate for strong connections, a few middle rungs, and a low rung at a few hundred kilobits for the worst connections. Each rendition is cut into short segments, a few seconds each, and the current authoring guidance puts a segment at about six seconds. The whole ladder — every rendition, cut into segments — plus a small manifest listing what exists is what actually gets stored and served.

Now the half that kills the most common myth. The instinct is that the server watches your connection and decides what quality to push you. That is not how it works. The player pulls; the server does not push. When playback starts, the player first fetches the manifest — the small file that lists every rung and where each rendition's segments live. Then the player fetches the video one segment at a time, and before each segment it chooses the rung: it looks at how fast the last few segments actually arrived and how much buffer it has, and it requests the next six-second segment at the highest rung its own measured throughput can sustain. Connection drops on that walk outside? The next segment the player asks for is a lower rung, and playback stays smooth at reduced quality instead of freezing. Connection recovers? The player steps back up. The server has no per-viewer quality decision to make — it just serves whatever segment is asked for, which is exactly why those fetches cache so well at the edge.

Here is where the surprise from the brief pays off, and it is the most important number on the board. Now that you know what is stored — the whole ladder of renditions, cut into segments — size the library from the bytes up. One source hour at the old top rung is about two and a half gigabytes. A full ladder runs several more rungs below that, so budget very roughly twice the top rung for the whole set, call it about five gigabytes per source hour. A region's catalog is about eight thousand titles at a blended runtime of maybe an hour and a half each. Multiply it out and it is about sixty terabytes — and even generously multiplying by ten for extra audio languages, subtitle tracks, and a global footprint, the whole thing is still well under a petabyte. The exact same bytes, served to a huge audience, were terabits per second of egress in the last use case — the wall — while the bytes at rest are tens of terabytes. The scary number was always the small one.

The read path is built: the edge serves bytes near the viewer, and the player climbs a ladder of renditions. But that ladder does not appear by magic. Something has to turn one creator's source video into all those renditions, and something has to get the gigabyte source file in safely in the first place. Creators upload constantly — hundreds of hours of new video every minute across the service. Each source video has to be encoded into its whole ABR ladder before anyone can watch it, and encoding is heavy. Two things cannot happen: the creator cannot wait for the encode to finish before their upload is accepted, and a gigabyte upload cannot be lost to a dropped connection.

The arrival rate itself is the argument for the shape. Encoding on upload is the instinct, and it dies on a number: about five hundred source-hours a minute, each fanned into a ladder of maybe six rungs, is roughly three thousand encoded hour-equivalents a minute. No fixed set of machines bolted to the upload path could clear that inline, and every creator would be left watching a progress bar for the full encode. Work that arrives far faster than it can be done immediately, that must not be lost and can be done slightly later, is the textbook case for a durable job queue and a worker pool: the upload enqueues a job and returns, and the fleet drains the queue as fast as its worker count allows.

The point of a durable job queue is that the job lives in the queue, not in the worker. A worker leases a chunk-encode job, does it, and marks it done; if the worker dies mid-encode, the lease expires and the job goes back to the queue for another worker to pick up. The creator already got their success — the source video is safe in the blob store — so a lost worker only delays that video's renditions slightly, it never loses the video. This is the same self-correcting property a partitioned job queue gives any independent-task pipeline, which is exactly the shape here: the encode jobs are independent of each other, so they fan across a partitioned queue and a worker pool cleanly.

Accept resumable chunks, return on durable store, and encode the ABR ladder async through a job queue and a parallel worker fleet. Take the encoding first, because a number settles it. The instinct is to encode on upload — the creator uploads, the server transcodes into the renditions right then, and returns success when they are ready. At one upload an hour, fine. At the real rate, impossible: roughly five hundred hours of source video arrive every minute, and each hour has to be encoded into a ladder of maybe six rungs, which is about three thousand encoded hour-equivalents of work every minute. No synchronous per-upload step can swallow that. Work arriving far faster than it can be done immediately, that must not be lost and can be done slightly later, is exactly what a durable job queue with a worker pool is for. In practice that queue is an SQS- or Kafka-class durable log. So the upload does not trigger an encode. It enqueues an encode job.

The upload path's only synchronous responsibility is to get the source bytes stored durably and hand the creator a success — the video is safe, it will be ready to watch shortly. Behind the queue, a worker fleet pulls jobs and encodes, and it scales horizontally: more encode work simply means more workers draining the same queue. One more move makes it fast instead of merely unblocked — a single source video does not have to be one long encode on one worker. It is split into many short chunks, each chunk encoded in parallel by a different worker, then the encoded chunks are joined back into the finished renditions. Netflix's own pipeline fans one upload into roughly thirty-one parallel chunk-encode jobs this way. The encode itself — the codec work of turning a chunk of pixels into a rung's bitrate — is treated as an opaque job the worker runs; this design sizes and schedules that work, it does not re-derive the codec.

Now the upload transport, the second failure. A source video is gigabytes, and a plain single upload that drops at ninety percent has to start over from zero, which at gigabyte sizes is brutal and common. The fix is resumable chunked upload: the file is sent in chunks over HTTP, the server tracks how far it got, and a dropped connection resumes from the last received byte rather than restarting. This is a solved, standardized thing — the tus protocol is the open standard behind several major services' resumable uploads — so the design adopts it and moves on. The upload service that accepts these chunks is stateless in front of the durable store: it buffers the incoming chunks, and the moment the full source video is durably stored it returns success and drops an encode job on the queue. Success means your bytes are safe, not your video is encoded.

Step back and look at what is built. It is complete for the happy path: the edge serves bytes near every viewer, the player climbs a ladder that fits its connection, the catalog hands out manifests, and uploads are accepted fast and encoded off to the side. Now the premise arrives already broken, the way a real incident does. On one Christmas Eve, one of the biggest streaming services in the world went dark for about seven hours across the United States, Canada, and Latin America. The cause was not a flood of viewers and not a failure of any serving box. An internal maintenance process was run by mistake against production state for a slice of a cloud provider's load-balancing fleet — the control plane that directs traffic to servers — and it deleted that state. The video bytes were fine. The stored renditions were fine. But the layer that routes a viewer's request to a server was gone in that region, so requests could not be steered to where the bytes lived, and televisions across three continents showed an error instead of a movie.

The instinct is to read this as bad luck with one cloud provider, or an argument for more redundancy in that provider's load balancers. Both miss the lesson, and the lesson is the whole reason the edge tier is shaped the way it is. A more redundant control plane is still a control plane playback depends on in real time — make it five-nines and a bad-enough maintenance slip still takes it, and every stream with it. The engineering answer is not to make the central dependency stronger; it is to remove the dependency from the hot path. The bytes a viewer needs are already at the edge, a short hop away, so playback can ride the nearest cache instead of the farthest control plane.

The honest counterpoint belongs in the lesson, because it is the other half of what this incident teaches. The edge decouples playback, but the incident happened because a single region's control plane was a hard dependency in the first place. The buffer only helps if the design actually leans on it — if the edge is allowed to serve without checking in, and if the control plane itself is not a single regional point of failure that a maintenance slip can delete wholesale. During a control-plane failure, a PoP that already holds a title keeps serving it outright; a PoP that gets a miss cannot fill from a dead region, so that one obscure title degrades — but that is a soft edge, not a hard floor. The popular titles the region is actually watching are already cached, often in RAM, so the overwhelming majority of playback continues while the long tail waits for the control plane to come back. Degrading the rare miss while protecting the common hit is exactly the shape graceful should take.

The edge is the availability buffer — decouple playback, degrade to stale, fail the control plane not the stream. Ask why a deleted control plane could stop playback at all. It is because playback in that moment still depended, in real time, on a central regional control plane to connect viewers to bytes. And that is exactly the coupling the edge tier is built to break. The bytes a viewer needs are already sitting in a cache inside their own internet provider, a short hop away. If playback is decoupled from the origin and the central control plane — if the edge can keep serving the segments it already holds without phoning home for permission on every request — then a regional control plane can disappear and viewers already watching keep watching, because their bytes never needed the control plane in the first place.

This is where the signature dive pays off a second time, reframed. The edge tier was introduced to beat the egress wall — to spread the bytes across appliances near the viewer. Its second, quieter job is exactly this: because the bytes live at the edge, close to the viewer and independent of any one central control plane, the edge is what keeps playback alive when the center fails. Graceful degradation here means the edge keeps serving the segments it already has, even if the catalog is briefly unreachable and the manifest a player holds is a little stale, rather than failing every stream the instant the control plane hiccups. A viewer mid-movie should not care that a management layer three hops away just went down. The reframe is bigger than this board: a system that serves bytes to the world should push the thing viewers depend on — the bytes — as close to them and as far from a single central dependency as it can, and then let the far management layers fail without taking the near bytes with them.