Blob Stores

It is past midnight after a wedding in Jaipur, and two hundred guests open the same photo backup app to save what they shot. A cousin's 2 GB video of the baraat reaches 93% when her phone switches towers, and the upload starts again from zero. In the app's data centre, one of the two file servers holding every family's photos reports a dead disk array, and the team spends the night restoring a week-old backup. A restart from zero, one machine holding everything, a lost week: each is a problem a blob store is built to solve.

A blob (binary large object) is a file stored as an opaque run of bytes: a photo, a video, a backup. A blob store, also called an object store, keeps billions of them, from a few KB to many GB each, and returns any one by name. This chapter follows the backup app as it grows from those two file servers to 10 billion files.

Keeping files in the database is the obvious first idea, and a poor one. Databases are built for small rows that are queried and updated in place; a 4 MB photo in a row slows backups and replication and pushes useful rows out of memory, for bytes nobody queries. So the usual split is simple: the file goes in the blob store, and a small row in the database points to it.

Here is how the three options compare.

The blob store is the only column that survives a dead machine and still grows to petabytes, which is exactly what the backup app lacked that night.

From outside, the data model is deliberately small. A bucket is a named container, often one per app and environment, with its own access rules. Inside it, each object is identified by a key, a string such as u/42/2026/10/IMG_2041.jpg for user 42's photo.

The slashes look like folders, but the namespace is flat: the key is just a string, and listing all keys that start with u/42/2026/10/ gives a folder-like view of one month. Each object also carries metadata: system fields such as size, content type, creation time and a checksum, plus your own fields such as the phone it came from.

PUT /backup-prod/u/42/2026/10/IMG_2041.jpg
Content-Type: image/jpeg
Content-Length: 3412880

GET /backup-prod/u/42/2026/10/baraat.mp4
Range: bytes=0-1048575

HEAD /backup-prod/u/42/2026/10/IMG_2041.jpg
GET /backup-prod?prefix=u/42/2026/10/&limit=100
DELETE /backup-prod/u/42/2026/10/IMG_2041.jpg

Put, get (whole or a byte range), metadata only, list by prefix, delete. That is nearly the whole API.

Objects are usually immutable: you replace a whole object rather than editing bytes in the middle, which makes caching and replication much simpler, because a copy can never be half old and half new. A byte-range read still lets a video player fetch only the minute it needs.

Behind that API, a blob store has two very different jobs, so it uses two kinds of servers. The metadata service knows, for every object, which chunks it is made of and which data nodes hold each chunk. The data nodes store chunks on their local disks and do little else.

Metadata is small but read on every request, so it lives in a replicated, partitioned database (the databases chapter) with strong consistency. Each partition's leader is chosen through consensus (the coordination chapter), so two servers never both decide where a key's chunks go. The bytes are huge but simple, so they sit on cheap disks.

The backup app shows the difference in size. Say it reaches 1 crore users with about 1,000 files each: 10 billion objects. Assume an average of 1 MB, since every photo also has small thumbnail objects beside it.

Metadata is about 2,000 times smaller than the data it describes, small enough for a database cluster, while the bytes need thousands of disks. Data nodes send a heartbeat to the metadata service every few seconds with their health and free space. That is how the metadata service knows where new chunks can go, and how it notices when a node stops answering.

The cousin's 2 GB baraat video is where the design meets a real phone network. Sent as one piece, it is fragile: a tower switch at 93% means starting again, and one data node must hold the whole file. So large files are split into chunks of a fixed size, from a few MB to around 128 MB. In chunks, a file can be uploaded in parallel, retried a piece at a time and spread over many nodes.

The client side of this is called multipart upload, and it runs in five steps.

In numbers: the 2 GB video in 16 MB parts is 128 parts. Over home Wi-Fi uploading about 5 MB/s, it takes about 7 minutes, and a dropped connection costs at most one 16 MB part, about 3 seconds of work. The app saves the upload id and the finished part numbers on the phone, so it can resume even after being closed.

Part size is a trade-off. Small parts waste less on a retry but mean more requests and more metadata per object. Managed stores also cap the number of parts, commonly at 10,000 per upload, so a client picks bigger parts for bigger files. Once the parts arrive, each chunk must be stored safely, which raises the next question: what happens to the video when a data node dies?

Here is the whole life of a smaller file, a 48 MB clip from the same wedding: three chunks, three copies of each on four data nodes, then a disk failure the day after.

The bytes never passed through the metadata service, only decisions did, and the video appeared only at "complete", so readers never see half a file. Losing a node cost no data because every chunk had copies on other machines. Repair ran from two survivors at once instead of waiting for one replacement disk, and that difference decides durability at scale.

The scene's writes met healthy nodes. Real writes meet dead or slow ones, and must decide quickly while the phone waits.

• A node is down. The metadata service has seen its heartbeats stop and does not pick it. If a node dies mid-write, the API asks for a replacement and writes the chunk there, so the chunk still has three copies before the client hears OK.
• A node is slow. The write waits for every copy, so one disk taking 2 seconds slows every upload it touches. Give each copy a timeout of a few times the normal p99, then write to a replacement.
• Confirming at two copies. Some stores reply after two copies and let repair make the third. Uploads get faster at the tail, but for a few minutes that chunk survives only one more loss, so do it only with fast repair.
• Erasure-coded writes follow the same rules per piece: the 9 pieces of a 6 + 3 chunk (explained on the next page) go to 9 different nodes, and a piece whose node fails is written elsewhere.

A half-finished write is never visible: a chunk that reached two nodes before the API crashed was never added to the metadata, so no reader finds it, and garbage collection removes the strays later. Three full copies are the simple answer to durability; the next page asks whether they are the cheapest one.

With 10,000 disks failing at roughly 1 to 2% a year, you lose one every two to four days. Durability, the chance that an object survives a year, comes from redundant data on different machines.

Replication keeps full copies, usually 3, on different disks and racks. It is simple, reads are fast, and repairing a lost copy means copying one chunk. The price is 3× the storage.

Erasure coding works like a parity sum: from k data pieces it computes m extra parity pieces, so that any k of the k + m pieces can rebuild the chunk. Reed-Solomon coding is the classic method. With 6 + 3, a chunk becomes 9 pieces on 9 nodes, uses 9 ÷ 6 = 1.5× the storage and survives any 3 lost pieces. The price is work: encoding costs CPU, reading a whole chunk means fetching pieces from 6 nodes, and rebuilding one lost piece means reading 6 others over the network.

Side by side, for 1 PB of data:

Erasure coding halves the disk and survives more losses; it pays in network and CPU on every repair and degraded read. Many stores use both: the backup app's new uploads stay replicated while they are viewed and shared, and files untouched for 30 days are re-encoded to 6 + 3.

Copies only help if they do not die together, and only while they are rebuilt faster than disks fail. A failure domain is a set of things that fail together: a disk, a machine, a rack sharing one power feed and one switch, a building. Three copies in one rack disappear together when that rack loses power, so placement puts each copy in a different rack.

Erasure coding needs the same care per piece. If a 6 + 3 chunk keeps 3 pieces in each of 3 racks, losing one rack uses up all its spare pieces, and one more disk failure during that hour loses the chunk. With one piece in each of 9 racks, a whole rack plus any two disks can fail. So a coded cluster needs at least 9 racks to get the full benefit.

Repair is a race. An object is lost only if every copy, or 4 of its 9 pieces, fails before repair finishes, so durability depends on repair speed as much as on the number of copies. Take a 20 TB disk that is 80% full: 16 TB of chunks to rebuild.

• Onto one replacement disk, writing about 200 MB/s: 80,000 seconds, most of a day exposed.
• Spread over 80 nodes, each re-copying 200 GB at 100 MB/s: about half an hour.
• With 6 + 3 coding, each lost piece is rebuilt by reading 6 others: about 96 TB of network reads. Over 40 nodes at 100 MB/s each, that is about 7 hours, unless repair gets more bandwidth.

Two rules keep repair from causing its own outage. Wait 10 to 15 minutes before declaring a node dead, because most silent nodes are rebooting, and rebuilding 16 TB for a node that returns wastes the network. And repair the chunks with the fewest copies left first, throttled so user reads keep their bandwidth. A chunk down to its last copy skips the wait entirely.

The redundancy scheme now turns into hardware. Take the backup app's 10 billion objects, about 10 PB of data, and assume nodes of 24 disks of 20 TB each. The scheme changes the answer by a factor of two.

Halving the disk halves the machines, but the smaller cluster has less spare bandwidth for its heavier repairs. Fill disks only to about 80%, because repair needs free space to rebuild into. The 40-node answer also has a placement catch from the previous page: those nodes must spread over at least 9 racks, about 4 or 5 nodes each.

Money follows the same shape. Managed object storage costs roughly $0.02 per GB-month (check current prices), so 10 PB is about $200,000 a month. Either way, for files many people download, internet egress at about $0.05–0.10 per GB often costs more than storing them, which matters once the app lets families share albums.

A delete has two halves. Removing the key from the metadata is quick, and from that moment the object is gone from reads and listings. Freeing the chunks on the data nodes happens later, in the background, because deleting across many nodes is slow and can fail halfway.

Deleted objects are not the only source of orphan chunks, chunks that nothing refers to. Abandoned multipart uploads leave parts behind, and a write that failed after reaching some nodes leaves stray copies. All are collected the same way. The backup app adds its own layer on top: deleted photos sit in a trash for 30 days, and only then does the app send DELETE to the store.

Lifecycle rules automate the rest: move objects older than 30 days to a colder tier, delete old versions after 90 days, abort incomplete uploads after 7.

Next, how bytes reach phones. Do not stream files through app servers: a 2 GB download ties up bandwidth and a connection for minutes, and the app adds nothing. Let clients talk to the blob store directly, but only for files they are allowed to see.

A signed URL (often called a pre-signed URL) makes that safe. After checking that the user may access a file, your app server builds a URL containing the path, an expiry time and a signature: a keyed hash of those fields, made with a secret only your servers and the store know. The store recomputes it and serves the file only if it matches and the time has not passed, so nobody can change the path or expiry. Anyone holding the link can use it, so keep expiries to minutes.

POST /api/uploads                     (the app server checks login, quota and file type)
200 {"upload_url": "https://blobs.example.in/backup-prod/u/42/9f1c.jpg?expires=1792134000&sig=4b9e..."}

PUT https://blobs.example.in/backup-prod/u/42/9f1c.jpg?expires=1792134000&sig=4b9e...
Content-Type: image/jpeg
200 OK                                (the bytes went straight to the blob store)

The app server only signs; the bytes go straight between the phone and the store, for uploads and downloads alike.

When a family shares the wedding album with 500 relatives, the same photos are read thousands of times. Put a CDN in front of the bucket (the CDN chapter): the bucket becomes the CDN's origin, and edges near users serve repeat reads. Give files versioned names, such as a content hash in the key, so they can be cached for a year and never need invalidating; an edited photo simply gets a new name.

Stores can also publish an event when an object is created. On a queue (the message queue chapter), they let workers make thumbnails without the uploader waiting, and they answer a question this page skipped: how the app learns that an upload really happened.

A signed upload URL leaves your database in the dark. The phone may never upload, may send a different file, or may crash halfway. Treat each upload as a small state machine, with the database row as its record.

Sign the limits into the URL itself: the exact key, the content type, a maximum size such as 50 MB, and the expiry. Where the store checks them, a 2 GB file or an executable is turned away at the door, because changing any of them breaks the signature. The worker still checks the real bytes.

Events arrive at least once and sometimes late, so the worker's last step is conditional: set ready only where the status is still pending. A duplicate event then changes nothing, and thumbnails are written to fixed keys, so redoing them is harmless. The album shows only ready rows. The app may call POST /uploads/9f1c/complete to mark it ready sooner, but the server must first confirm with a HEAD request that the object exists with the right size: never take the client's word that an upload finished.

Durability is not the only thing that breaks as the backup app grows.

Almost every fix moves load away from one hot spot, whether a node, a partition or the network. The table leaves out one property you must decide and state: consistency, what a reader sees right after a write.

With a single metadata record per key, read-after-write for a new object is straightforward: once the PUT succeeds, every GET sees it, because the commit and the read go to the same partition leader. Two uploads to the same key resolve the same way: each commits a complete chunk list, the later commit wins whole, and the loser's chunks become orphans for the collector. Listings are often served from a separate index and can lag by a moment; document that.

Copies across racks and buildings survive a dead disk, a dead machine and a dead rack. They do not survive losing a whole region to a flood, a fire or a long power cut. For data you cannot lose, keep a cross-region copy in a second region hundreds of kilometres away.

The copy is made asynchronously: each new object is written in its home region, then queued and copied over, usually within minutes. Copying before replying would add a cross-region round trip, 30 ms or more, to every part of every upload. The price is a window of recent uploads that a regional disaster could take with it; that window is your recovery point objective (RPO), the most recent data you accept losing.

Four rules shape the second region:

• Each region is a complete store, with its own metadata, so the copy can serve reads alone if the home region goes dark.
• Copy what matters. Original photos and videos, yes; thumbnails, no, since workers can rebuild them. Keep the copy in a colder, cheaper tier.
• Respect data rules. Data that must stay in India, as payment data must, needs a second region that is also in India.
• A copy is not a backup. Deletes copy across too, so turn on versioning, and lock backups against deletion for a set period, such as 30 days.

For "design a blob store", spend the 40 minutes of design time like this.

Pick durability if they ask how you never lose a file: placement, writes with a dead node, repair speed. Pick metadata if they stress billions of objects or fast listings: partitions by bucket and key, hash prefixes, strong consistency per key.

A senior interviewer then probes what is easy to draw and hard to run:

The same core also appears under other names: