Design a Video Streaming Platform

It is 8 pm on the night before the Class 12 chemistry board exam. A teacher in Kota finishes a one-hour revision lecture and uploads it over patchy hostel Wi-Fi. By 11 pm two lakh students have pressed play: some on home broadband, many on shared Wi-Fi or 4G in crowded rooms. If the site kept one big 1080p file and sent it from one data centre, the upload would restart at every dropped connection, students on weak signals would stare at a spinning wheel, and the data centre's links would choke. This chapter builds the platform that gets that lecture to every one of them.

The product: a video platform for India. Anyone can upload a video, and anyone can watch it, on a phone in a moving train or on a TV at home. Assume about 200 million daily viewers and 100,000 hours of new video uploaded every day. We follow the chemistry lecture through every stage of the design: upload, transcode, package, deliver and play.

The chemistry lecture shows two very different workloads hiding in the brief. Uploading happens once, is slow and is heavy on CPU: the teacher's file must be converted into many versions before any student can watch it. Watching happens two lakh times, and every time it is heavy on bandwidth. Most of this design comes from keeping those two paths apart.

Here is what is in scope today, by who uses it:

Live streaming is the biggest exclusion: it shrinks the pipeline from minutes to seconds, so it appears only as a variant at the end.

The non-functional requirements decide most of the cost, so pin each one to a number:

• Fast start: playback begins within about 2 seconds of a tap, even on a slow connection.
• Few stalls: viewers forgive a softer picture far more than a frozen one, so avoiding buffering beats maximum sharpness.
• Durable uploads: once the teacher sees "processing", the original file is safely stored.
• Available for watching: playback survives the loss of a server, a cache location or a whole data centre.
• Eventually consistent view counts, search and suggestions: a delay of a minute or two is fine.

Start with the watch path, because that is where the bytes are. Think of every viewer as a pipe that must stay full for as long as they watch: the question is how many pipes are open at once, and how wide each one is.

Assume 200 million daily viewers who watch 30 minutes each, at an average of about 2.5 Mbps. That average is a mix: phones on 480p at 1.2 Mbps, laptops on 720p, TVs on 1080p at 5 Mbps. The working, row by row:

No single data centre can send 20 terabits per second across a country with good latency, so a CDN (content delivery network) is not an optimisation here; it is the design.

The lecture makes the same point for one video. If two lakh students each watch the full hour at 2.5 Mbps, each pulls about 1.1 GB: about 225 TB in one night, from a lecture stored as 4.4 GB. If half of them watch at the same moment, that one video needs 1 lakh × 2.5 Mbps = 250 Gbps.

Every number here scales with the average bitrate. If the mix shifts towards TVs, the average rises towards 5 Mbps and the peak towards 40 Tbps; if most viewing moves to phones on 480p, it falls by half. The 4.4 GB came from the upload side, which is next.

Now the upload path. Creators upload 100,000 hours of video a day, and each video is stored as several renditions: versions of the same video at different resolutions and bitrates. So the real question is how many bytes one hour of video becomes.

The table carries that forward to a year, with the two ways a blob store keeps data safe:

Storage grows by hundreds of petabytes a year, so it lives in a blob store that grows by adding machines.

Two choices follow from that. Erasure coding (from the blob store chapter) halves the bill compared with three copies, at the cost of slower rebuilds after a disk fails, which suits files that are written once. And rarely watched videos move to cheaper, slower tiers: last year's lectures on a topic nobody studies any more do not need fast disks.

The original upload is not in that estimate. A phone's 1080p recording is often as big as all five renditions together, so it usually goes to cold storage, kept so the video can be encoded again later with better settings. With the numbers set, the next page designs the calls that carry an upload in.

Clients need only a handful of calls. Notice what is missing: no endpoint streams video. Watching happens through files served by the CDN: a manifest that lists the versions of a video, and the short segments of video it points to. The API accepts uploads and hands out metadata.

POST /v1/uploads                        { title, sizeBytes }       -> 201 { uploadId, chunkSize }
PUT  /v1/uploads/{uploadId}/chunks/{n}  <bytes of chunk n>         -> 200
GET  /v1/uploads/{uploadId}                                        -> 200 { chunksReceived[] }
POST /v1/uploads/{uploadId}/complete                               -> 202 { videoId, status: "processing" }
GET  /v1/videos/{videoId}                                          -> 200 { title, status, manifestUrl }
GET  /v1/search?q=organic+chemistry&cursor=c2                      -> 200 { results[], nextCursor }
GET  /v1/home?cursor=c2                                            -> 200 { videos[], nextCursor }
POST /v1/videos/{videoId}/views         { sessionId, watchedSec }  -> 202

The player gets manifestUrl from the metadata call, then talks only to the CDN.

The upload is resumable. The teacher's 2 GB lecture goes up as about 250 chunks of 8 MB, each acknowledged on its own. When the hostel Wi-Fi drops at 70%, the app asks which chunks arrived and sends only chunks 176 to 250 instead of starting over. Smaller chunks waste less on a drop but cost more requests; 8 MB is a common middle on mobile networks.

Each chunk carries a checksum, so the service rejects one that was corrupted on the way and asks for it again. A checksum of the whole file also allows de-duplication: when a byte-for-byte identical file is uploaded again, as happens with a viral clip, the platform reuses the renditions it already made. An edited copy has a different checksum, so this catches only exact copies.

complete returns 202 Accepted, not 200, because the video cannot be watched yet. Processing takes minutes, so the app polls the status or receives a push notification when it becomes ready. Views are a 202 too: the player sends the event and never waits for the counting system.

The upload has created three kinds of data, and they belong in different places: small structured metadata, huge video files, and a stream of events such as views.

Each kind goes to the store that suits how it is written and read:

The rule behind the table: put each kind of data where its access pattern is cheap, not all of it in one database.

Segments are not rows. With 4-second segments, the one-hour lecture has 900 per rendition, 4,500 in all; a two-hour film has 9,000. Manifests list them and sit next to them in the blob store; the database keeps only the top-level manifest's key.

Metadata is read far more often than it changes. If 200 M viewers open about 10 videos a day, that is 2 billion reads a day, about 23,000 per second. Keep each video's metadata in the distributed cache for a few minutes and delete the entry when the creator edits the video; the database then sees a small share of those reads.

With the calls and the stores in place, here is how they connect. The upload path turns one file into many renditions in the background; the watch path serves them from the edge. The two paths meet only at stored data: the blob store and the video's status. The scene follows the lecture through all five stages, and the teacher's Wi-Fi drops on the way.

Each stage hands over to the next through stored data, so a failure stays where it happens: a dropped connection costs only the missing chunks, and a slow transcoder delays one video, not the CDN. A burst of uploads after results day breaks nothing either; videos wait longer in the queue while more workers start to drain it.

The scene hid the hardest stage inside one box. What happens between "360 jobs" and "ready"? The deep dives start there.

A phone records one file at one size, often 1080p at a high bitrate. That file suits almost nobody: it is too big for a student on weak 4G, and an old TV may not decode its format. Transcoding decodes the original and encodes it again into a set of renditions called a bitrate ladder: the same picture at several sizes, so every network gets one it can carry. Our ladder has five rungs:

Each rung is roughly half the bitrate of the one above, so a viewer whose speed halves drops one or two rungs, not to the bottom.

Encoding is slow: five rungs from one long job can take longer than the video itself, so the lecture would sit in "processing" for over an hour. The fix is to split the original at keyframes (complete pictures that need no earlier frame to decode) into pieces of about 10 seconds. The lecture becomes 360 independent jobs, and a few hundred workers finish them in minutes.

Across the platform, 100,000 hours a day is 36 M pieces, about 420 a second. If a worker needs about a minute per piece for all five rungs (an assumption; it depends on the codec and the machine), about 25,000 workers are busy on average, more after a festival. Shorter pieces finish sooner, but every cut adds a keyframe and costs a little quality per bit, so 10 seconds is a common middle.

Splitting creates a new problem: with 360 jobs per video, some will run twice. Why is that dangerous?

The job queue hands each piece to a worker with a lease, the visibility timeout from the message queue chapter. If the worker has not finished when the lease runs out, the job reappears for another worker. That is how a crashed worker's piece still gets done. But a lease cannot tell a dead worker from a slow one, so sometimes two workers encode the same piece, and both report success.

Two things could then go wrong. Storage could hold two copies of the piece. Worse, if the pipeline counts finished pieces with a running total, the duplicate pushes the count to 360 while another piece is still missing, and packaging builds a lecture with a hole in it. Watch both being prevented:

Two rules make a repeat harmless. Every output goes to a key fixed by the job, such as v/42/720p/piece_017, so a second run overwrites the first. And the tracker stores piece numbers, not a count, so a second "done" changes nothing; packaging starts only when all 360 numbers are in.

The lease length is a trade-off. Set it a little above the slowest normal piece, say 3 minutes for a 1-minute job: too short and healthy jobs run twice, wasting CPU; too long and a crashed worker's piece waits, delaying the whole lecture. A piece that fails on every worker is a poison piece: after 3 attempts it goes to a dead-letter queue and the video is marked failed with a reason, instead of retrying forever. With all 360 pieces encoded, the renditions still cannot be streamed. Packaging is next.

An encoded rendition is one long file. A player could download it from the start, but it could not change quality halfway or jump to minute 40 cheaply. Packaging fixes that by cutting each rendition into segments of 2 to 6 seconds and writing a manifest, a small text file that lists what exists. Think of a textbook: the segments are its pages, the manifest is its table of contents.

Two details make switching possible. Every rendition is cut at the same moments, and each segment starts with a keyframe, so segment 23 at 480p covers exactly the same seconds as segment 23 at 1080p and decodes on its own. A player can change rendition at any segment boundary without losing its place.

The top-level manifest names each rendition and its bitrate; each rendition has its own playlist of segments. A trimmed example in HLS, Apple's format:

#EXTM3U
#EXT-X-STREAM-INF:BANDWIDTH=5000000,RESOLUTION=1920x1080
1080p/index.m3u8
#EXT-X-STREAM-INF:BANDWIDTH=1200000,RESOLUTION=854x480
480p/index.m3u8

Each index.m3u8 lists that rendition's segments in order.

DASH, an MPEG standard, describes the same thing in one XML manifest, and both formats can share the same fragmented MP4 segments, so one copy of the files serves both. They are plain files over HTTP, which is why any CDN can cache them.

With 4-second segments the lecture has 900 per rendition. A 1080p segment is 5 Mbps × 4 s ÷ 8 = 2.5 MB; a 480p one is 0.6 MB. Segment length is the trade-off: 2-second segments start faster and react sooner to a changing network, but double the requests and the keyframes; 6-second segments do the reverse. 4 seconds is a common choice for recorded video. The files are ready. Next: getting them to two lakh students without melting the origin.

Segments are ideal CDN content: they never change once written, their URLs are stable, and many viewers ask for the same ones at once. Popularity is very uneven, too. A small share of videos, such as the lecture everyone watches the night before the exam, gets most of the views, while a long tail of old videos is watched a few times a day. Edges hold the head; the tail mostly comes from regional caches or the blob store.

The hit ratio, the share of requests an edge answers from its own cache, decides how hard the origin works. At 20 Tbps of peak traffic, a 95% hit ratio leaves 1 Tbps for shields and origin; 99% leaves 0.2 Tbps. For the lecture alone, even if 200 edge locations each pulled every rendition once, the origin would send 200 × 4.4 GB, under 1 TB, against about 225 TB delivered. Popular video is cheap to serve from the edge; the long tail is what costs.

Three ways to fill edges, and when each fits:

Pull is the default; push and shields exist to stop the bursts that pull alone would send to the origin.

At 9 pm thousands of students may ask an edge for segment 1 of the lecture in the same second. The edge sends one request upstream and makes the rest wait for it (request collapsing). The largest platforms also put cache servers inside internet providers' networks: the provider saves on its paid links, and the platform pays far less per GB.

The segments now sit at an edge near every student. The last stage of the spine runs on the student's phone: which rendition should it fetch next? The player does not choose once and stick with it: it decides again before every segment. This is adaptive bitrate streaming.

It uses two signals: how fast recent segments downloaded, and how many seconds of video wait in the buffer (downloaded but not yet played). A full buffer allows a step up. A draining buffer forces a step down before the picture freezes. Most players also keep a safety margin, choosing the highest rung below about 80% of the measured speed.

The player steps down before the buffer runs dry, and it may skip a rung: 720p would still drain 1 s per segment at 2 Mbps. It climbs back one rung at a time, because a step up that fails costs a stall, while a slow climb costs only a few seconds of softer picture.

Three habits follow from cutting every rendition at the same moments:

• Start low, then climb. Starting at 480p puts the first frame on screen sooner.
• Switch only at segment boundaries. A switch changes the sharpness, never the position.
• Seeking is cheap. Jumping to minute 40 fetches the segment covering minute 40 and nothing before it.

The spine is complete. Now walk it again one stage at a time and ask what fails, what a viewer would notice, and how the system recovers:

The pattern: failures on the upload path are handled by making work safe to retry; failures on the watch path by having another copy somewhere else.

What dominates the bill: delivery. About 110 PB leaves the edges each day. Even at a tenth of a list egress price of $0.05 per GB, that is about $16 million a month, more than three times the $4.8 million a month for storing a full year of renditions (240 PB at $0.02 per GB-month); check current prices.

Every stage of the spine had a dial. Each choice below buys smoother playback or a lower bill with something else, and the table names both sides:

None of these has a right answer in general; each depends on how often a video is watched.

Codecs show that best. A common middle path is to encode every video in a widely supported codec such as H.264, so almost every device can play it, and add a newer, more efficient codec only for popular videos, where the bandwidth saved pays for the extra encoding. For a lecture watched two lakh times, an encode that saves 30% of the bits saves about 70 TB of delivery in one night; for a clip watched ten times, it saves almost nothing and still costs the CPU.

The same logic sizes the ladder. A slide-based lecture with a still background needs far fewer bits than a cricket match for the same sharpness, so platforms with enough volume tune the ladder per video instead of using one fixed table.

Optional: view counts, search and suggestions, the systems built around the videos once the spine works.

View counts. The lecture can collect thousands of views per second at 9 pm. Writing each view to one database row would make that row the hottest spot in the system. Instead, players send view events to a queue; workers drop bots and repeated refreshes, sum the rest per video every few seconds, and add the totals to sharded counters (see the sharded counters chapter). The number on the page lags by a minute or so, which nobody minds.

Search. When a video becomes ready, an event sends its title, description, tags and auto-generated captions to the search indexer. Ranking mixes text relevance with signals such as views, recency and watch time. An index that is a minute or two behind is fine here.

Suggestions. At a high level, recommending is a funnel that narrows millions of videos down to about twenty:

Candidates are mostly prepared offline by batch jobs; ranking runs at request time with fresh signals, such as the video the viewer just finished. If ranking is slow or down, the home page falls back to a cached list of popular videos so it still loads.

A 45-minute interview leaves about 40 minutes of design once introductions and your own questions are done. Here is how to spend them on a video platform.

Pick the transcoding pipeline if they ask how an upload becomes watchable, how long that takes or what happens when a worker crashes. Pick the edge and the player if they ask about slow networks, start time or serving 20 Tbps. Offer view counting only if time remains.

Interviewers change one thing to see whether your design bends or breaks. Try each question before opening the answer.