Large-Scale Data Ingestion

What is inside

Every section, and every one of them opens in the browser. Nothing here is hidden: if a section you need is not in this list, the notebook is not the right one and you should not spend a week on it.

1. How to use this book
2. The whole system on one page
3. 1The problem, stated properly
4. 2Clarifying questions that buy you the design
5. 3Back-of-the-envelope, before the boxes
6. 4The first principle: separate ingestion from processing
7. 5Why object storage, not the database
8. 6Presigned URLs, keeping the bytes out of your backend
9. 7Multipart and resumable upload
10. 8Choosing the part size
11. 9When the network fails mid-upload
12. 10Completing the upload and verifying it
13. 11The dual-write problem and the outbox
14. 12Why a queue at all
15. 13Kafka or RabbitMQ or SQS
16. 14The worker loop
17. 15Streaming, never loading
18. 16Chunking and checkpointing
19. 17Idempotency
20. 18The lease problem: when the queue thinks you died
21. 19Parallelising one big file
22. 20Retries, poison pills and the DLQ
23. 21The job state machine
24. 22Progress without hammering the database
25. 23Telling the frontend
26. 24Scaling: workers, backpressure and queue depth
27. 25Multi-tenancy and fairness
28. 26Partial failure: what "failed" means
29. 27Cost, the bill nobody mentions
30. 28Observability and security
31. 29The seven-step delivery script
32. 30Whiteboard drawing order
33. 31Follow-up question bank
34. Appendix A. Numbers to memorise
35. Appendix B. Glossary
36. Appendix C. Self-test

How to read it

Read with a pen. Every notebook opens with a question to answer before you start and asks you to redo the answer at the end, and the margin in the Print edition exists so you have somewhere to be wrong first. The Tablet edition is 16:9 with vector text, so note apps draw on it rather than treating it as a photograph.