System 12.4 — File Storage (HLD)
In one line
Blob storage + metadata DB + edge serving — S3-style, the LLD from Phase 4 at object scale.
Think of it like this
Google Drive's engine room: billions of files, spread across huge cheap storage, with a lightweight 'phone book' service that just remembers which file lives where.
Key ideas
- 01
Two planes: metadata plane (DB storing file table) and blob plane (object store: S3/GCS/minio).
- 02
Upload path: multipart/chunked upload with checksums; download path: presigned URL direct-from-blob (shift load off your API).
- 03
Scale: 5TB new/user/year? No — realistic: 10k uploads/day × 5MB → 50GB/day → ~55TB over 3 years; cache hot files at CDN.
- 04
Consistency: s3 is strong-managed today; your list/rename/version ops live in the metadata DB.
- 05
Access control: every request checks owner + share links against the metadata; blobs are opaque objects with random keys.
- 06
Failures: blob store retries + multipart resume; metadata primary + replica; CDN TTL invalidation on version bump.
- 07
Extras that always appear: versioning, trash/soft-delete, checksum verification, virus scan pipeline, dedupe by hash.
Code & diagrams
Two planes that never touch each other directly: metadata (small, relational) and blobs (huge, dumb).
Explain without notes
Why presigned URLs are the right answer for 'download the file without proxying through my API' — 3 reasons.
Practice
Sketch upload with multipart, checksum, and the metadata-blob consistency story.
Trade-offs
- ↔
Managed blob (S3) vs self-hosted (minio): price, ops, durability guarantees — say the decision driver.
Run it in production
You've designed it. Now build, operate, and break the same idea hands-on in the DevOps courses:
Completion checklist
I can separate metadata plane from blob plane and use edge/presigned for the hot read path.