How to use this book
This is not a reference manual. It is a study notebook built around one interview question, chosen because that single question forces you through resumable uploads, object storage, queues, workers, streaming, retries, idempotency, checkpointing, backpressure and asynchronous job tracking. Learn this one case deeply and you can answer a large fraction of backend system design questions by reassembling its parts.
Every page has a dotted rail on the right. Use it. The people who do well in system design interviews are the ones who can draw the architecture from memory under pressure, and you only get there by drawing it badly a few times first.
The five block types
| Block | What it means |
|---|---|
| Interviewer asks | A real follow-up you should expect. Answer it out loud before reading on. |
| Senior signal | The specific sentence that separates a senior answer from a mid-level one. |
| Trap | A common answer that sounds right and is wrong. |
| Do this | The concrete practice, with the parameter or command. |
| Key idea | The one thing to carry out of that section. |
A suggested route
- Read Part I and draw the flow once, badly, from memory.
- Read Parts II to V in order. These are the mechanics, and they are where follow-up questions live.
- Skim Parts VI and VII, then come back after you can draw the flow.
- Drill Part VIII the night before. It is the delivery, not the knowledge.
The whole system on one page
┌───────────────┐
│ FRONTEND │
└───────┬───────┘
│ 1. POST /jobs (metadata only)
▼
┌───────────────┐ 2. create job row (status=CREATED)
│ API SERVICE │──────────────────────────────┐
└───────┬───────┘ 3. presigned upload URLs │
│ ▼
│ ┌──────────────┐
4. PUT part 1..N │ │ METADATA │
(direct, parallel, │ │ DB │
resumable, retried) │ └──────┬───────┘
▼ │
┌───────────────┐ │
│ OBJECT STORE │ 5. CompleteMultipartUpload │
│ (S3) │ │
└───────┬───────┘ │
│ 6. notify API / S3 event │
▼ │
┌───────────────┐ 7. outbox row + publish │
│ JOB QUEUE │◀────────────────────────────┘
│ SQS / Rabbit │
└───────┬───────┘
│ 8. lease a message
┌──────────────┼──────────────┐
▼ ▼ ▼
┌──────────┐ ┌──────────┐ ┌──────────┐
│ WORKER 1 │ │ WORKER 2 │ │ WORKER N │ 9. stream, transform,
└────┬─────┘ └────┬─────┘ └────┬─────┘ checkpoint, batch
└──────────────┼──────────────┘
▼
┌─────────────┴─────────────┐
▼ ▼
┌──────────┐ ┌──────────┐
│ RESULT │ │ RESULTS │
│ DB │ │ IN S3 │
└────┬─────┘ └──────────┘
│ 10. progress + terminal status
▼
┌──────────┐ 11. GET /jobs/:id (poll) or SSE stream
│ FRONTEND │
└──────────┘The rest of this book is the justification for every arrow above, and what happens when each one fails.
Part I · Framing the problem
1The problem, stated properly
A weak framing of this question is:
How do I process ten gigabytes of data?
That framing produces a bad answer, because it invites you to talk about parsing speed. The real framing is:
Write that at the top of your notes in the interview. It reframes the question from a throughput problem into a reliability and state-management problem, which is what the interviewer is actually testing.
The requirements, restated as constraints
| Requirement | The constraint it creates |
|---|---|
| Files are gigabytes to terabytes | Cannot fit in memory, cannot fit in one HTTP request |
| Processing takes minutes to hours | Cannot happen inside a request/response cycle |
| Networks fail mid-upload | Transfer must be resumable at sub-file granularity |
| Workers crash mid-job | Processing must be resumable and repeatable safely |
| Many users submit at once | Load must be buffered, not applied directly to workers |
| Users want progress | Job state must be externally observable, not in worker memory |
Every architectural decision in this book falls out of the right-hand column.