Object Storage

This book follows one lecture end to end and then keeps going where the lecture stops. The lecture is Object Storage, Everything You Need to Know, parts 1 and 2 (youtube.com/watch?v=Ie0TjKI9cDI and ?v=iWrVCxexUWY), and its structure is worth copying: it does not start with S3. It starts with the most reasonable thing a competent developer would write on their first day, then kills it seven times, and only then introduces the thing that survives.

That is also how the topic is examined. Nobody asks "what is S3". They ask "a user uploads a 4 GB video, walk me through it", and the answer is judged by whether you know which failure you are avoiding at each step.

The five block types

The four highlighters

Colour in this book is never decorative. Each one means one thing, so a marked phrase is already half-read before you get to the words.

Figures follow the same discipline. Box drawing recedes to a pale wire so the shape reads first, NODE LABELS take full ink, and anything after a left arrow is an annotation in pencil rather than part of the machine.

What is in the lecture and what is not

Parts I to VII are the two lectures, expanded and reordered. Parts VIII onwards are the production concerns that turn up in interviews immediately after the upload path is agreed, and are not in either lecture: the access-control mechanisms, encryption, lifecycle and untrusted content. Sections overlapping Notebook 01 (Large-Scale Data Ingestion) say so and point at the section there that goes deeper, rather than repeating it.

A suggested route

1. Read Part I in one sitting. Do not skip to the answer. The seven failures are the actual content of an interview answer; object storage is just the noun at the end of them.
2. Read Parts II to IV. These are the "do you understand the system or have you only used the SDK" sections.
3. Work Part V with an editor open. Build the two-phase flow once.
4. Skim VI and VII, then drill VIII.

                    ┌──────────────────┐
                    │     BROWSER      │
                    └────┬─────────┬───┘
       1. POST /uploads  │         │  3. POST file + policy fields
          (intent only)  │         │     (bytes go direct, never to you)
                         ▼         │
                  ┌─────────────┐  │
                  │ YOUR SERVER │  │      ┌────────────────────────┐
                  │             │  │      │     OBJECT STORAGE     │
                  │ authorise   │  │      │                        │
                  │ pick key    │  └─────▶│  bucket + key = object │
                  │ sign policy │         │                        │
                  └──────┬──────┘         │  metadata plane        │
       2. row: status    │                │    sorted index        │
          = pending      │   5. HEAD key  │  data plane            │
                         │  ◀───────────▶ │    14+6 shards         │
                         ▼                └───────────┬────────────┘
                  ┌─────────────┐                     │
                  │  DATABASE   │  6. status=ready    │ 7. GET via
                  │             │                     │    signed URL
                  │ what exists │                     ▼
                  │ who owns it │              ┌─────────────┐
                  │ orig. name  │              │     CDN     │──▶ users
                  └─────────────┘              └─────────────┘
 
        4. client calls POST /uploads/:id/complete after the PUT

Two rules are encoded in that picture, and they are the whole architecture:

• The bytes never touch your server. Your server authorises, names, signs and records. It does not carry payload.
• The database is the index; the bucket is the storage. The bucket knows how to hold bytes. It does not know which user owns them, what the file was called, or whether the upload finished.

A user needs to upload a profile picture. You have a backend with handlers, services, repositories, a database and auth. Nothing exotic is being asked for.

The intuitive implementation writes itself:

  browser                 your server                     disk + db
 
  <input type=file>  ──▶  parse multipart/form-data
                          write bytes to /var/app/uploads/avatar.png
                          INSERT INTO users(avatar_path) VALUES (...)
                     ◀──  200 OK
 
  <img src=/avatar>  ──▶  SELECT avatar_path
                          read the file off disk
                     ◀──  image bytes

The browser sends the file as multipart/form-data, you pull it out of the request, you save it to the local file system, because it is a file and files live in file systems, and you record the path in a row so you can find it again.

It works. It works on your laptop, it works in staging, and it works in production on the day you ship it. That is exactly what makes it dangerous: the design does not fail at the moment it is wrong, it fails later, under conditions your development machine never reproduces.

The wrong assumption underneath it is small and worth naming precisely, because the interviewer is listening for it:

The next seven sections are the seven ways this comes apart. Read them as a sequence of forced moves, not a list of complaints.

The pattern survives profile pictures. A phone photo is 300 KB to 10 MB, a mirrorless camera file might be 20 MB, and all of that fits comfortably in a server sized for JSON.

Then a feature lands that needs video, and someone uploads 4 GB.

  container memory limit: 4 GB
 
  request arrives ──▶ framework helpfully parses the whole body
                      into memory before handing it to your handler
 
     0 GB  ▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏  ▏
     1 GB  ████████▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏  ▏
     2 GB  ████████████████▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏▏  ▏
     3 GB  ████████████████████████▏▏▏▏▏▏▏▏▏▏▏▏▏  ▏
     4 GB  ████████████████████████████████ OOMKilled

ParseMultipartForm in Go, the default body parser in an Express app, the same helper in every framework: the convenient call is the one that buffers. The container hits its memory limit and the kernel kills it.

Note the second-order effect, which is the part people miss. The instance restarts, and every file ever uploaded to that instance is gone, because it was on the container's file system. One user's oversized video destroyed other users' data. The blast radius of an OOM is not one request.

A container's file system lives exactly as long as the container.

  deploy v1        ┌────────────┐
                   │ container  │  /app/uploads/  a.png  b.png  c.mp4
                   └────────────┘
                          │
  push new code, CI/CD builds a new image
                          │
                          ▼
  deploy v2        ┌────────────┐
                   │ container  │  /app/uploads/  (empty)
                   └────────────┘

The image contains your code. It does not contain your users' files. So every deploy, every crash, every OOM kill, every node drain, every scale-in event wipes them. This is not a bug in Docker; it is the property that makes containers interchangeable, and it is the same property your deployment strategy depends on.

If you deploy on Kubernetes, ECS, Railway, Render, Fly, or any serverless platform, this applies to you. On serverless it is worse: the writable area is usually a small /tmp that disappears between invocations.

The partial solution: a persistent volume

If you know Kubernetes you already have an answer in mind: attach a PersistentVolume so the storage outlives the container.

That does work, for this failure. It is worth being precise about what it buys and what it does not:

This is the failure that most often gets asked about directly, because it is about statelessness rather than storage.

Three instances behind a load balancer, round robin:

                        ┌───────────────┐
      upload  ─────────▶│ LOAD BALANCER │
                        └───┬───┬───┬───┘
                            │   │   │
                  ┌─────────┘   │   └─────────┐
                  ▼             ▼             ▼
            ┌──────────┐  ┌──────────┐  ┌──────────┐
            │  node 1  │  │  node 2  │  │  node 3  │
            │ avatar ✓ │  │  (none)  │  │  (none)  │
            └──────────┘  └──────────┘  └──────────┘
 
      two minutes later, the browser asks for the image.
      round robin sends it to node 3.        ──▶  404

The upload landed on node 1. The read is routed by an algorithm that knows nothing about where bytes are, so it lands wherever it lands. Your hit rate is about 1/N, and it gets worse every time you scale out, which is the exact opposite of what scaling is supposed to do.

Servers scale horizontally. Disks do not.

  more traffic     ──▶  add another instance        (horizontal, online)
  more files       ──▶  make the one disk bigger    (vertical, disruptive)

Growing a disk means a resize: detach, expand, reattach, sometimes copy, usually with downtime or at least a maintenance window. And every provider has a ceiling, so "just make it bigger" has a last iteration.

Meanwhile user-uploaded content is typically the fastest growing data in the entire system, and it is close to monotonic, because people delete far less than they upload.

  1,000 users × 100 MB per month  =  100 GB / month, forever
 
  month  1     6     12    24    36
  size   0.1   0.6   1.2   2.4   3.6  TB   and it never shrinks