Design a Video Streaming Service

The origin works until an audience shows up. A popular video is being watched by a huge audience at once, spread across cities and countries. Every viewer is pulling a steady multi-megabit stream, continuously. Together they demand terabits per second of outbound bytes. One origin's link to the internet is orders of magnitude too small to push that. It does not slow down gracefully; past the first few thousand viewers, it simply cannot move the bytes. Where do the bytes come from?

Follow the obvious answer first, because it is what most people reach for: give the origin a bigger pipe. Put it on a fatter uplink, or add a few more origin servers behind a load balancer. It is the design you already have, scaled up, and the attraction is real. But the arithmetic kills it. ~500k people watching at once, each pulling ~3 Mbps, is ~1.5 Tbps from one region. A single high-end serving box tops out around ~100 Gbps. That regional demand is well over ~16 times what one box can push, on one mid-size region, on one popular title. A bigger pipe is not that much bigger. It is an order of magnitude short, and the gap widens with the audience. It also does nothing about distance: those viewers are spread across a continent, and serving them from one central location means every byte crosses the whole way, slow to start and expensive to carry.

A second instinct sounds sophisticated: put a cache in front of the origin. But a plain key-value cache is built to answer lookups faster. It remembers an answer so you do not recompute it. The origin's problem is not slow lookups. It is that bytes cannot leave the building fast enough, and a cache next to the origin still pushes every one of those bytes out the same regional uplink. It caches the wrong thing. The ceiling here is bandwidth, not lookup speed, and no amount of remembering answers moves more bytes through a fixed pipe. Neither a bigger pipe nor a cache in one place works. The fix is to spread serving across many appliances, each close to its viewers, so the aggregate rate is carried by the whole fleet and no single uplink is the ceiling.

A tier of caches close to the viewer — the CDN edge tier — with the origin serving only fills and misses. Stop serving the bytes from one place. Put the video on a tier of caches spread across the world, each one sitting close to where viewers actually are. A network of caches placed near viewers like this is called a content delivery network, or CDN, and this one is the CDN edge tier — edge because it sits at the outer edge of the network, nearest the people. Each location in it is called a PoP, short for point of presence, which is just a rack of cache machines parked somewhere close to viewers, very often inside the viewer's own internet provider. Those machines are called appliances: ordinary servers built to do one job, hold video and push it out fast. A viewer's player does not fetch segments from a distant origin. It fetches them from the nearest PoP, a short hop away. Here is the move that beats the arithmetic. The terabits of demand are no longer one number against one box. They are spread across hundreds of PoPs, each serving the viewers near it. That ~1.5 Tbps for one region becomes ~16 appliances working in parallel, each pushing its own ~100 Gbps to the viewers on its doorstep. Not one impossible origin pipe. Many small ones, each close to home.

The origin does not disappear, but its job changes completely. It stops sitting in the viewer's path. Instead it fills the edge, pushing copies of the videos out to the PoPs ahead of time during off-peak hours, and it answers the rare miss: fetching a missing segment once, serving it, and keeping a copy for next time. A PoP does not need the whole library, just the popular slice its region actually watches, a few hundred terabytes, with the most-watched tenth to fifth of that held in memory and served straight from RAM. The rarely-watched rest is not kept there at all, and a request for one of those falls through to the origin as a miss. This fleet is real: Netflix alone runs more than 8k of these appliances, part of why Netflix and YouTube together already account for roughly 28% of all the traffic flowing down to homes worldwide. The failure story is gentle too. If an appliance dies, its viewers are sent to a nearby PoP instead, and a miss just costs one fetch; the edge only ever holds copies of bytes the origin still has, so losing an appliance costs a warm cache, never a video. Outbound bandwidth does not scale up. It scales out, across many appliances, and outward, toward the viewer, until the bytes are served from inside the viewer's own network.

The edge serves bytes close to the viewer. But two questions the last dive set aside now have to be answered together. What exactly are those bytes — what unit does the edge cache and the player fetch? And how does a viewer on a weak, changing connection get a stream that stays smooth, when the single-server version could only offer one stored quality? A connection is strong at home, weak on a phone, and it changes minute to minute on the move. The service has one job: keep the video playing smoothly the whole time, sharp when the connection is good and never frozen when it is bad, without the viewer touching a quality setting.

Start with what breaks if you serve one fixed quality. You lose either way. Pick a high bitrate and the phone viewer stalls constantly, because their connection cannot pull the bytes fast enough. Pick a low bitrate and the fiber viewer gets a blurry stream on a connection that could have carried far better. And a viewer whose connection changes mid-video cannot be served correctly by any single fixed choice, because the right answer changed while they were watching. So the video has to exist at several qualities, and something has to get each viewer the right one, which a ladder the player climbs itself does without any server-side per-viewer decision.

That leaves the choice of the ladder itself: how many levels, at what bitrates, and cut how. The baseline is a fixed ladder, one set of bitrates applied to every title, the same levels whether it is a dark, slow drama or a bright, fast sports clip. It is simple and it works, and it was the industry norm for years, with a top level around 5.8k kilobits a second for 1080p and middle levels at 2.4k and 1.8k kilobits below it. But it spends bits badly. Some titles are easy to compress — a still, dark scene barely changes from frame to frame — and an easy title does not need the top level's full bitrate to look identical. A fixed ladder has no way to know that. The better default is to pick the ladder per title: look at what is actually in the video and choose that title's levels for its own content, so an easy title gets a lower top-level bitrate at the same perceived quality. On a real title that trimmed the top level by about 20% for equal quality, and that saves bytes on every level, on every segment, for every one of the millions of views. At this scale of outbound traffic, a fifth off the bitrate is enormous.

A ladder of quality levels chosen per title, cut into ~6-second segments, with the player picking its own level. One quality is not enough. The video has to exist at several. So the source video gets encoded more than once, into several complete copies of itself at different qualities. Call each copy a quality level. The top level is the sharpest picture at the heaviest bitrate, for the strongest connections. A few levels sit in the middle. The bottom level runs at a few hundred kilobits a second, for the worst connections there are. Stack those levels from lowest to highest and the picture you get is a ladder, which is what everyone calls it. Two trade names are worth knowing, because you will hear them and they mean exactly what you just read: the whole idea is called adaptive bitrate streaming, shortened to ABR, so this stack of levels is the ABR ladder, and one individual copy is a rendition. Level and rendition are the same thing said two ways, and this design says level. Every level is then cut into short segments, a few seconds each, and the current authoring guidance puts a segment at about six seconds. What actually gets stored and served is all of that: every level, cut into segments, plus one small file listing which levels exist and where their segments live.

Now the half that kills the most common myth. The instinct is that the server watches your connection and decides what quality to push you. That is not how it works. The player pulls; the server does not push. When playback starts, the player first fetches that small listing file — it is called the manifest — which tells it every level that exists and where each level's segments live. Then the player fetches the video one segment at a time, and before each segment it picks a level. It looks at how fast the last few segments actually arrived, and at how much video it has already buffered ahead, and it asks for the next six-second segment at the highest level that arrival speed can keep up with. Connection drops on that walk outside? The next segment the player asks for is a lower level, and playback stays smooth at reduced quality instead of freezing. Connection recovers? The player steps back up. The server has no per-viewer quality decision to make. It just serves whatever segment is asked for, which is exactly why those fetches cache so well at the edge.

Here is where the surprise from the brief pays off, and it is the most important number on the board. Now that you know what is stored, the whole ladder cut into segments, size the library from the bytes up. One source hour at the old top level is about 2.6 GB. A full ladder runs several more levels below that, so budget very roughly twice the top level for the whole set: call it 5 GB per source hour. A region's catalog is about 8k titles at a blended runtime of maybe 1.5 hours each. Multiply it out and it is about 60 TB, and even generously multiplying by ten for extra audio languages, subtitle tracks, and a global footprint, the whole thing is still well under a petabyte. The exact same bytes, served to a huge audience, were ~1.5 Tbps of bytes going out in the last dive, the wall, while the bytes at rest are tens of terabytes. The scary number was always the small one. All of that, every level, every segment, needs somewhere to be found from: a small metadata catalog holds the map, kilobytes of pointers per title rather than the gigabytes of the segments themselves. It is read on every single playback start and almost never written, so it is answered from read replicas — extra copies of the catalog kept in step with it and used only for reading, so a flood of lookups never lands on the one copy taking the writes. Its failure story is gentle. A replica that has fallen a little behind still hands the player a slightly stale but working list of levels, and if one specific level is momentarily unavailable at the edge, the player just requests a lower one it can get, since stepping down is already normal behavior, not an error path.

The read path is built: the edge serves bytes near the viewer, and the player climbs a ladder of quality levels. But that ladder does not appear by magic. Something has to turn one creator's source video into every level on it, and something has to get the gigabyte source file in safely in the first place. Creators upload constantly — hundreds of hours of new video every minute across the service. Each source video has to be encoded into its whole ladder before anyone can watch it, and encoding is heavy. Two things cannot happen: the creator cannot wait for the encoding to finish before their upload is accepted, and a gigabyte upload cannot be lost to a dropped connection.

The arrival rate itself is the argument for the shape. Encoding on upload is the instinct, and it dies on a number: about ~500 source-hours a minute, each turned into a ladder of ~6 quality levels, is roughly ~3k hours of finished video to produce a minute. No fixed set of machines bolted to the upload path could clear that while the creator waits, and every creator would be left watching a progress bar until the whole encode finished. Work that arrives far faster than it can be done, must not be lost, and can be done slightly later, is the textbook case for a durable job queue and a pool of workers: the upload adds a job and returns, and the fleet works through the queue as fast as its worker count allows.

The point of a durable job queue is that the job lives in the queue, not in the worker. A worker takes a chunk-encode job on loan — it is marked as claimed for a while, rather than removed — does the work, and marks it done. If the worker dies partway through, that claim runs out and the job goes back on the queue for another worker to pick up. The creator already got their success and the source video is safe in the blob store, so a lost worker only delays that video's levels slightly. It never loses the video. This self-correcting shape works for any pipeline whose jobs do not depend on each other, which is exactly the case here: one video's encode does not care what any other video is doing, so the jobs spread across the queue and the worker pool cleanly.

Accept the file in chunks that can resume after a drop, return as soon as it is stored, and encode the ladder afterwards through a job queue and a worker fleet. Take the encoding first, because a number settles it. The instinct is to encode on upload: the creator uploads, the server converts the video into every quality level right then, and returns success when they are all ready. That conversion has a name — transcoding, turning one video file into other formats and qualities — and it is the expensive step here. At one upload an hour, doing it inline is fine. At the real rate it is impossible. Roughly ~500 hours of source video arrive every minute, and each of those hours has to be encoded into a ladder of about ~6 levels. That is around ~3k hours of finished video to produce every minute — the useful way to count the work, because one hour of source encoded at six levels is six hours of output, not one. No step that runs while the creator waits can swallow that. Work that arrives far faster than it can be done, must not be lost, and can be done slightly later, is exactly what a durable job queue with a pool of workers is for: a list of pending jobs that survives a crash, plus a set of machines pulling jobs off it. In practice that queue is an SQS- or Kafka-class durable log. So the upload does not trigger an encode. It puts an encode job on the queue.

The only thing the upload path has to finish while the creator waits is getting the source bytes stored safely and handing back a success: the video is safe, it will be ready to watch shortly. The upload service that does this keeps nothing of its own between requests except the parts of a file still arriving, so you can run as many copies of it as you need behind a load balancer, and none of them holds anything worth losing. The bytes themselves land in the store, not on the upload service. Behind the queue, a fleet of workers pulls jobs and encodes them: more encoding to do just means more workers pulling from the same queue. One more move makes it fast, not merely unblocked. A single source video does not have to be one long encode on one machine. It is split into many short chunks, each encoded at the same time by a different worker, then joined back into the finished levels. Netflix's own pipeline splits one upload into roughly ~31 chunks this way, running the steps in an order that respects which ones depend on which. The encoding itself — turning a chunk of pixels into one level's bitrate — is treated as a sealed job the worker runs. This design sizes and schedules that work; it does not open up the compression scheme inside.

Now the way the file actually arrives, which is the second failure. A source video is gigabytes, and a plain single upload that drops at ninety percent has to start over from zero, which at gigabyte sizes is brutal and common. The fix is to send the file in pieces and let it pick up where it left off: the file goes over HTTP in chunks, the server tracks how far it got, and a dropped connection resumes from the last byte that arrived instead of restarting. This is a solved, standardized thing. The tus protocol is the open standard behind several major services' resumable uploads, so the design adopts it and moves on. The upload service holds the chunks as they come in, and the moment the full source video is safely stored it returns success and puts an encode job on the queue. Success means your bytes are safe, not your video is encoded. Once the pipeline writes the finished levels back, that store — a blob store, meaning storage built to hold very large files cheaply and keep them safe — becomes the master copy of every source video and every level — the thing the edge fills from, and the thing everything else can be rebuilt from. Lose part of it and those videos are simply gone, so it is copied across several machines and backed up. Unlike the edge or the catalog, nothing can rebuild it. The pipeline writes the video levels before it writes the catalog's manifest entry, on purpose, so a viewer's manifest never lists a level that is not actually there yet; a worker dying between those two writes just delays the manifest entry until the retry, and never breaks it.

Step back and look at what is built. It is complete for the happy path. The edge serves bytes near every viewer, the player climbs a ladder that fits its connection, the catalog hands out manifests, and uploads are accepted fast and encoded off to the side. Now the premise arrives already broken, the way a real incident does. On one Christmas Eve, one of the biggest streaming services in the world went dark for about seven hours across the United States, Canada, and Latin America. The cause was not a flood of viewers and not a failure of any serving box. An internal maintenance process ran by mistake against production state for a slice of a cloud provider's load-balancing fleet, the control plane that directs traffic to servers, and it deleted that state. The video bytes were fine, and so were the stored copies of every quality level. But the layer that routes a viewer's request to a server was gone in that region, so requests could not be steered to where the bytes lived, and televisions across three continents showed an error instead of a movie.

The instinct is to read this as bad luck with one cloud provider, or an argument for more redundancy in that provider's load balancers. Both miss the lesson, and the lesson is the whole reason the edge tier is shaped the way it is. A more redundant control plane is still a control plane playback depends on in real time. Make it about as reliable as anything gets, what engineers call five-nines uptime, and a bad-enough maintenance slip still takes it, and every stream with it. The engineering answer is not to make the central dependency stronger; it is to take the dependency off the path every viewer's request takes. The bytes a viewer needs are already at the edge, a short hop away, so playback can ride the nearest cache instead of the farthest control plane.

The honest counterpoint belongs in the lesson, because it is the other half of what this incident teaches. The edge is what lets playback stand on its own, but the incident happened because a single region's control plane was something every stream could not do without. The safety net only helps if the design actually leans on it, if the edge is allowed to serve without checking in, and if the control plane itself is not a single regional point of failure that a maintenance slip can delete wholesale. During a control-plane failure, a PoP that already holds a title keeps serving it outright; a PoP that does not cannot fetch it from a dead region, so that one obscure title stops working, and that is where the failure lands — at the rare edges, not across the whole service. The popular titles the region is actually watching are already cached, often in RAM, so the overwhelming majority of playback continues while the rarely-watched rest waits for the control plane to come back. Breaking the rare miss while protecting the common hit is exactly the shape graceful should take.

The edge is the safety net — playback should not depend on the control plane, and a stale answer beats no answer. Ask why a deleted control plane could stop playback at all. It is because playback in that moment still depended, right then and there, on a central regional control plane to connect viewers to bytes. That dependency is exactly what the edge tier exists to cut. The bytes a viewer needs are already sitting in a cache inside their own internet provider, a short hop away. If playback does not have to reach the origin or the central control plane at all — if the edge can keep serving the segments it already holds without asking permission on every request — then a regional control plane can disappear and viewers already watching keep watching, because their bytes never needed the control plane in the first place.

This is where the signature dive pays off a second time, reframed. The edge tier was introduced to beat the outbound-bandwidth wall, to spread the bytes across appliances near the viewer. Its second, quieter job is exactly this. Because the bytes live at the edge, close to the viewer and independent of any one central control plane, the edge is what keeps playback alive when the center fails. Degrading gracefully here means the edge keeps serving the segments it already has, even if the catalog is briefly unreachable and the manifest a player holds is stale — out of date by a few minutes, which for a list of quality levels changes almost nothing — rather than failing every stream the instant the control plane stumbles. A viewer mid-movie should not care that a management layer three hops away just went down. The lesson is bigger than this board. A system that serves bytes to the world should push the thing viewers depend on, the bytes, as close to them and as far from a single central dependency as it can, then let the far management layers fail without taking the near bytes with them.