Handbooks / System Design / Chapter 7
Storage: Blobs, Files and Search
7 pages · ~5 min✓ Reviewed
Builds on Databases, Replication and Sharding. Next up: Queues and Async Processing.
Storage Types: Block, File, Object
1. The Three Types
| Block (EBS) | File (EFS) | Object (S3) | |
|---|---|---|---|
| Unit | raw fixed-size blocks | directories + POSIX locking | key + blob + metadata |
| Access | one server, as a disk | NFS / SMB, shared | HTTP, flat namespace |
| Trait | lowest latency | shared tree | billions of objects, cheap, high latency |
| Use | databases, boot volumes | home dirs, legacy apps | media, backups, logs |
2. Objects Are Immutable
- Replaced whole, never edited in place
- Change one byte = re-upload the object
- No append, no rename
bash
aws s3 cp s3://b/old s3://b/new aws s3 rm s3://b/old
rename = copy + delete
3. Pick by Access Pattern
yesnoyesno
S3 Core: Buckets, Keys, Consistency
1. Buckets and Keys
- Bucket names are globally unique
- Object = bucket + key, such as
photos/2026/a.jpg - Folders are only a key-prefix convention
- List with prefix + delimiter
/to fake directories
2. Limits and Consistency
| Fact | Value |
|---|---|
| Single object max |
Presigned URLs and Multipart Upload
1. Presigned URL
- Presigned URL = URL signed with your credentials
- Grants one operation (GET or PUT) on one key until it expires
- Holder needs no AWS credentials
- Browser-direct: file bytes never pass through your server
- Client asksbackend
- Backend signsPUT URL
- Client PUTsstraight to S3
python
url = s3.generate_presigned_url(
'put_object',
Params={'Bucket': b, 'Key': k},
ExpiresIn=900)valid for 900 s
Storage Tiers, Lifecycle, Lakes and Warehouses
1. Tiers by Access
| Tier | Use | Note |
|---|---|---|
| Standard | hot | none |
| Standard-IA / One Zone-IA | infrequent | retrieval fee, 30-day min, 128 KB min billed |
| Glacier Instant | archive | instant restore |
| Glacier Flexible | archive |
Inverted Indexes and Elasticsearch Basics
1. Inverted index and analysis
- Inverted index: term → [doc ids], so search looks up terms, not every doc
LIKE '%word%'cannot use a normal B-tree index: full scan- Analyzer mismatch at index vs query time → missing hits
- Char filtersstrip HTML
- Tokenizersplit words
- Token filterslowercase, stopwords, 'running' → 'run'
2. Shards and queries
Relevance, Search Sync and Cheat Sheet
1. Relevance scoring
- BM25: rare terms (IDF), more matches (TF, saturates), shorter fields
- Boost with
title^3;multi_matchcombines fields filterclauses skip scoring and are cached
json
{"bool": {
"must": {"multi_match": {"query": "redis",
"fields": ["title^3", "body"]}},
"filter": {"term": {"status": "live"}}}}2. Syncing DB to index
| Pattern |
|---|
Check yourself
A text field uses the english analyzer. You index 'Running', then run a term query for 'running'. Zero hits. Why?
- The index stored the stemmed, lowercased token
run. termskips analysis, so it looks for 'running'. Usematchor term 'run'.
Your backend signs a 24-hour presigned PUT URL using an IAM role. Uploads fail after about an hour. Why?
- The role's temporary session token expires first, and the URL dies with it.
- Sign with longer-lived credentials, or issue shorter URLs.