Binary Semaphorelearn
sign in

System Design Notebook / 02

Object Storage

Why a file is not your application's data, and what to do about it

Seven ways the obvious design dies, one at a time, then block, file and object storage compared on what they actually promise. Ends on the trade you are making when you pick one.

pages
86
sections
54
editions
Reading, Print, Tablet

Sign in to read it

The whole notebook is free for 7 days. No card, no payment. We ask for an account so the download link belongs to someone.

Sign in with GitHub or Google

What is inside

The full table of contents. Nothing here is hidden: if a section you need is not in this list, the notebook is not the right one and you should not spend a week on it.

  1. The whole picture on one page
  2. 1. The requirement, and the intuitive answer
  3. 2. Failure 1: the file does not fit
  4. 3. Failure 2: the disk is ephemeral
  5. 4. Failure 3: horizontal scaling, and the 1-in-N 404
  6. 5. Failure 4: the disk grows the wrong way
  7. 6. Failure 5: durability is not availability
  8. 7. Failure 6: your server is an expensive CDN
  9. 8. Failure 7: no transaction across two systems
  10. 9. Block storage
  11. 10. File storage, and what POSIX costs
  12. 11. Object storage: the minimum interface
  13. 12. The trade you are actually making
  14. 13. Key, value, metadata, bucket
  15. 14. There are no folders
  16. 15. Designing the key
  17. 16. Why LIST is not a lookup
  18. 17. What happens when you PUT
  19. 18. The data plane: replication and erasure coding
  20. 19. Eleven nines, read carefully
  21. 20. The metadata plane
  22. 21. Consistency: eventual, then strong
  23. 22. Conditional writes, the primitive that changed things
  24. 23. Architecture A: through your server
  25. 24. Buffering versus streaming
  26. 25. The limits streaming does not remove
  27. 26. Architecture B: presigned URLs
  28. 27. The hole in a presigned PUT
  29. 28. Presigned POST policies
  30. 29. The two-phase upload
  31. 30. Reconciliation: pending, expired, orphaned
  32. 31. CORS, the last mile
  33. 32. Why a single PUT stops working
  34. 33. The multipart protocol
  35. 34. Choosing the part size
  36. 35. The ETag trap
  37. 36. Abandoned uploads: the invisible bill
  38. 37. The full large-file architecture
  39. 38. Resuming
  40. 39. Never relay a download
  41. 40. Three delivery patterns
  42. 41. Why a presigned GET destroys your cache
  43. 42. Range requests
  44. 43. Segmented streaming, and where object storage stops
  45. 44. The bill
  46. 45. The security surface
  47. 46. Lifecycle, storage classes and versioning
  48. 47. Untrusted content, and what to watch
  49. 48. The delivery script
  50. 49. Whiteboard drawing order
  51. 50. Follow-up question bank
  52. Appendix A. Numbers to memorise
  53. Appendix B. Glossary
  54. Appendix C. Self-test

How to read it

Read with a pen. Every notebook opens with a question to answer before you start and asks you to redo the answer at the end, and the margin in the Print edition exists so you have somewhere to be wrong first. The Tablet edition is 16:9 with vector text, so note apps draw on it rather than treating it as a photograph.

more notebooks