System Design Notebook / 01
Large-Scale Data Ingestion
How to accept, store, process and track multi-gigabyte jobs without losing them
Accepting a multi-gigabyte upload is not a bigger version of accepting a form. This one walks the whole path: presigned URLs, multipart and resume, the outbox pattern, the queue, and how a caller finds out the job is still alive.
- pages
- 68
- sections
- 35
- editions
- Reading, Print, Tablet
Sign in to read it
The whole notebook is free for 7 days. No card, no payment. We ask for an account so the download link belongs to someone.
Sign in with GitHub or GoogleWhat is inside
The full table of contents. Nothing here is hidden: if a section you need is not in this list, the notebook is not the right one and you should not spend a week on it.
- The whole system on one page
- 1. The problem, stated properly
- 2. Clarifying questions that buy you the design
- 3. Back-of-the-envelope, before the boxes
- 4. The first principle: separate ingestion from processing
- 5. Why object storage, not the database
- 6. Presigned URLs, keeping the bytes out of your backend
- 7. Multipart and resumable upload
- 8. Choosing the part size
- 9. When the network fails mid-upload
- 10. Completing the upload and verifying it
- 11. The dual-write problem and the outbox
- 12. Why a queue at all
- 13. Kafka or RabbitMQ or SQS
- 14. The worker loop
- 15. Streaming, never loading
- 16. Chunking and checkpointing
- 17. Idempotency
- 18. The lease problem: when the queue thinks you died
- 19. Parallelising one big file
- 20. Retries, poison pills and the DLQ
- 21. The job state machine
- 22. Progress without hammering the database
- 23. Telling the frontend
- 24. Scaling: workers, backpressure and queue depth
- 25. Multi-tenancy and fairness
- 26. Partial failure: what "failed" means
- 27. Cost, the bill nobody mentions
- 28. Observability and security
- 29. The seven-step delivery script
- 30. Whiteboard drawing order
- 31. Follow-up question bank
- Appendix A. Numbers to memorise
- Appendix B. Glossary
- Appendix C. Self-test
How to read it
Read with a pen. Every notebook opens with a question to answer before you start and asks you to redo the answer at the end, and the margin in the Print edition exists so you have somewhere to be wrong first. The Tablet edition is 16:9 with vector text, so note apps draw on it rather than treating it as a photograph.