Binary Semaphorelearn
sign in

System Design Notebook / 02

Object Storage

Why a file is not your application's data, and what to do about it

Seven ways the obvious design dies, one at a time, then block, file and object storage compared on what they actually promise. Ends on the trade you are making when you pick one.

pages
86
sections
54
editions
Reading, Print, Tablet

Sign in to read it

Sign in and the whole notebook opens, all three editions with it. An account is what makes a download belong to someone.

Sign in with GitHub or Google

Start reading

How to use this book

Start reading

What is inside

Every section, and every one of them opens in the browser. Nothing here is hidden: if a section you need is not in this list, the notebook is not the right one and you should not spend a week on it.

  1. How to use this book
  2. The whole picture on one page
  3. 1The requirement, and the intuitive answer
  4. 2Failure 1: the file does not fit
  5. 3Failure 2: the disk is ephemeral
  6. 4Failure 3: horizontal scaling, and the 1-in-N 404
  7. 5Failure 4: the disk grows the wrong way
  8. 6Failure 5: durability is not availability
  9. 7Failure 6: your server is an expensive CDN
  10. 8Failure 7: no transaction across two systems
  11. 9Block storage
  12. 10File storage, and what POSIX costs
  13. 11Object storage: the minimum interface
  14. 12The trade you are actually making
  15. 13Key, value, metadata, bucket
  16. 14There are no folders
  17. 15Designing the key
  18. 16Why LIST is not a lookup
  19. 17What happens when you PUT
  20. 18The data plane: replication and erasure coding
  21. 19Eleven nines, read carefully
  22. 20The metadata plane
  23. 21Consistency: eventual, then strong
  24. 22Conditional writes, the primitive that changed things
  25. 23Architecture A: through your server
  26. 24Buffering versus streaming
  27. 25The limits streaming does not remove
  28. 26Architecture B: presigned URLs
  29. 27The hole in a presigned PUT
  30. 28Presigned POST policies
  31. 29The two-phase upload
  32. 30Reconciliation: pending, expired, orphaned
  33. 31CORS, the last mile
  34. 32Why a single PUT stops working
  35. 33The multipart protocol
  36. 34Choosing the part size
  37. 35The ETag trap
  38. 36Abandoned uploads: the invisible bill
  39. 37The full large-file architecture
  40. 38Resuming
  41. 39Never relay a download
  42. 40Three delivery patterns
  43. 41Why a presigned GET destroys your cache
  44. 42Range requests
  45. 43Segmented streaming, and where object storage stops
  46. 44The bill
  47. 45The security surface
  48. 46Lifecycle, storage classes and versioning
  49. 47Untrusted content, and what to watch
  50. 48The delivery script
  51. 49Whiteboard drawing order
  52. 50Follow-up question bank
  53. Appendix A. Numbers to memorise
  54. Appendix B. Glossary
  55. Appendix C. Self-test

How to read it

Read with a pen. Every notebook opens with a question to answer before you start and asks you to redo the answer at the end, and the margin in the Print edition exists so you have somewhere to be wrong first. The Tablet edition is 16:9 with vector text, so note apps draw on it rather than treating it as a photograph.

more notebooks