Scaling and Performance

This book follows Backend Scaling and Performance Engineering, parts one and two (youtube.com/watch?v=z7kt_p44rjs and watch?v=sOhAopEwjH4, 4h05 across the two, the longest subject in the Backend from First Principles playlist).

Most scaling material is a catalogue: here is caching, here is sharding, here is a load balancer. A catalogue is useless in an interview, because the interviewer is not asking whether you know what a read replica is. They are asking whether you would reach for one, and why, and what it would cost you.

So this book is organised around one question instead.

The interview version of the same question is shorter and more brutal: which number told you to do that?

Scope

Performance means something different in a browser, in a compiler, in a game engine. This book stays where the lecture stays, in the backend you write and the infrastructure directly around it: your handlers, your queries, your cache, the machines your code runs on, and the network between them.

What this book adds

The lecture is strong on intuition and light on numbers. Where a piece of folk wisdom has an actual formula behind it, this book gives you the formula, because a number is what turns an opinion into an answer. It also fills four gaps the lecture leaves open: backpressure, retry storms, cache stampede, and tail latency amplification. Each is marked where it appears.

The five block types

Recall rules and full-page redraw plates appear at the end of each part. Use them. Every prompt in this book also appears in Notebook 00, the Question Bank, where you can work them cold.

Related notebooks

This book deliberately does not repeat what other notebooks cover in depth.

• Indexes, EXPLAIN ANALYZE, isolation levels: Notebook 04, Postgres.
• Queues, workers, leases, idempotency, dead letters: Notebook 01, Large-Scale Data Ingestion, Parts IV and V.
• Object storage, presigned URLs, CDN origins: Notebook 02, Object Storage.
• Rate limiting and the abuse side of load: Notebook 05, Backend Security.

A user says the app is slow. A product manager repeats it. Somebody opens a ticket. And every one of those sentences contains exactly zero actionable information.

Here is what "the app is slow" can mean, all of which have different fixes:

  "the app is slow"
        │
        ├── every request takes 800ms          ← a systemic problem
        ├── one endpoint takes 8s              ← one bad query or call
        ├── it is fine until 6pm               ← a capacity problem
        ├── it is fine for me, slow for them   ← a geography problem
        ├── the first click is slow            ← a cold start or cold cache
        └── it hangs, then works               ← queueing, or a retry storm

Six different systems. Six different fixes. One sentence.

The whole of Part I exists to give you those five words. Latency, percentile, throughput, utilization, headroom. They are not jargon for its own sake. Each one is a different question you can ask about the same slow endpoint, and together they narrow "slow" down to something with a fix.

Latency is the time from the user's action to the user's result. Not the time your handler took. The whole round trip: the click, the request across the network, your routing, your handler, your database, whatever external calls you made, the response back, and the render.

Two requests to the same endpoint do not take the same time.

  request A ──────  50 ms      hit the cache, server was idle
  request B ──────────────────────────── 200 ms
                               missed the cache, server was
                               already handling 50 other requests
 
  average    ─────────────── 125 ms      describes neither request

That average of 125ms is arithmetically correct and practically worthless. It describes a request that never happened.

Now scale the same arithmetic up, because this is where it becomes dangerous. You measure a thousand requests and the average comes out at 100ms. Excellent number. Put it on a dashboard. But underneath it:

  990 requests    completed under  50 ms
   10 requests    took            5000 ms
 
  average                          100 ms   ← looks healthy

At a million requests a day, that 1% is ten thousand real people staring at a loading spinner for five seconds. The average hid all of them, and it will keep hiding them no matter how bad they get, because an average is a summary that deletes exactly the information you need.

Percentiles

The fix is to stop summarising and start reporting the shape of the distribution. A percentile names a latency and tells you what fraction of requests came in under it.

The useful mental move is to subtract from 100. P99 means one in a hundred. P95 means one in twenty. Say the fraction out loud and the number stops being abstract: "one request in twenty takes over two seconds" is a sentence somebody will act on. "P95 is 2000" is not.

There is an obvious reason to care about the slow tail: those are real users having a bad time, and fairness alone says they count.

There is a second reason that is less obvious and more useful in an interview.

The requests in your tail are not random. They are slow for reasons, and the reasons correlate with value. The request that fans out to three services, hits a large join, calls the payment provider and waits on a webhook is slow because it is doing something important. The request that returns a cached marketing page is fast because it is doing nothing.

  a fast request                     a slow request
  ──────────────                     ──────────────
  GET /health                        POST /checkout
  GET /static/logo.svg               POST /orders/:id/refund
  GET /products (cached)             GET /reports/annual
  browsing                           paying

So the population sitting at P99 skews towards users doing the things you charge for. Optimising the median makes your marketing pages faster. Optimising the tail makes your checkout faster.

The gap the lecture leaves: tail latency amplification

Here is something the lecture does not cover, and it is the single most important consequence of caring about tails.

Suppose one request to your API fans out to ten internal services. Each of them is well behaved: P99 of 100ms, meaning each is slow one time in a hundred. Your overall request is only fast if every single one of the ten is fast.

  P(all ten fast)  =  0.99 ^ 10  =  0.904
 
  so roughly 1 request in 10 hits at least one slow call.
  a 1-in-100 event at the service level
  becomes a 1-in-10 event at the request level.

Your P90 is now determined by your dependencies' P99. Fan out to a hundred services and it is worse than a coin flip.

Throughput is how many requests your system completes per unit of time, usually requests per second. Latency is about one request. Throughput is about all of them.

You need both, because a latency number without a throughput number is meaningless. "Our P99 is 150ms" invites the question "at what load?" A system can be beautifully fast at 10 requests per second and fall apart at 1,000.

   10 rps  →  P99  150 ms
  100 rps  →  P99  180 ms
  500 rps  →  P99  400 ms
 1000 rps  →  P99  2000 ms    ← same code, same query, same machine

Nothing in your application changed between the first row and the last. What changed is that requests started waiting for each other.

This is the relationship to internalise, and it is not linear:

  latency
    │                                          ╱
    │                                        ╱
    │                                      ╱
    │                              ______╱
    │        ______________──────
    │───────
    └───────────────────────────────────────── throughput

Flat, flat, flat, then a wall. Throughput does not degrade latency gradually. It degrades it slowly and then all at once.