Measuring systems
FoundationsLatency and Throughput
How long one request takes, and how many you can serve per second — two different axes that people mix up constantly. Little's law is the one equation that ties them together.
Requests in flight = arrival rate × time each one takes. Make anything slower and you need more of everything to hold the same rate.
Try it
Move the dials — the sentence under the picture changes.In plain words
Latency is how long one request takes — the time between asking and getting an answer. Throughput is how many requests you get through per second. They sound related and people use them interchangeably, but they are different axes: a motorway has high throughput (thousands of cars an hour) and, at 3 a.m., low latency (twenty minutes end to end). At 5 p.m. the throughput is the same and the latency is an hour.
The one equation
- arrival rate (λ)
- 500 req/s
- time inside (W)
- 25 ms
- in flight (L = λ × W)
- 12.5
- in flight once W is 250 ms
- 125
That is what the widget shows. Turn the database call up and watch the service fill with requests that are not doing anything — just waiting. Every slot they hold is one a new request cannot have.
Three numbers, not one
| What it measures | Unit | Made worse by | |
|---|---|---|---|
| Latency | Time for one request | ms | Slow code, slow dependencies, queueing |
| Throughput | Requests completed per second | req/s | Too few copies, a bottleneck anywhere on the path |
| Concurrency | Requests inside the system right now | count | High latency × high throughput — Little's law |
A part has two ceilings, and either can be the one that binds. It can be out of throughput — its CPU cannot process more per second. Or it can be out of slots — it has capacity to spare but every connection is held open waiting on something else. The playground reports whichever binds and calls the second one "held at once".
CPU at 95%. Each request takes real work. Fix: more copies, or less work per request.
CPU at 15%, every connection busy. Each request is waiting. More copies help; making the thing it waits on faster helps more.
Where latency comes from
A request's time is a sum, and most of it is usually not your code.
Useful reference points, so a number means something when you see it:
L1 cache reference 1 ns
Main memory reference 100 ns
Read 1 MB from memory 10 µs
SSD random read 100 µs
Read 1 MB from SSD 1 ms
Round trip inside a data centre 0.5 ms
Database query, indexed 1–10 ms
Round trip, same continent 20–40 ms
Round trip, across an ocean 100–150 ms
Disk seek (spinning) 10 msWhy "average" latency lies
If 99 requests take 10 ms and one takes 2 seconds, the average is 30 ms and one user in a hundred is furious. Real systems report percentiles: p50 (the median), p95, p99 — "99% of requests finished within this time". The playground's headline latency is a p99 for exactly this reason; see tail latency for why the tail matters more the bigger the system gets.
Where it goes wrong
- Optimising the wrong axis. Making your code 5 ms faster when the request spends 80 ms on the network and 20 ms in the database changes almost nothing the user can feel.
- Adding copies to fix latency. More servers raise throughput. A request that takes 200 ms on one server takes 200 ms on ten. Only queueing delay goes away with copies.
- Ignoring concurrency. Connection pools, thread pools, file descriptors and socket limits are all concurrency ceilings. Little's law says exactly how big they need to be: rate × time. Size them from that, not from a guess.
- Measuring at the server. The user's latency includes DNS, TLS and the network. Measure from where the user is.
Take this with you
- The one idea: latency and throughput are different axes; concurrency is their product, and it is what actually runs out.
- In an interview, state Little's law and apply it to a connection pool or thread pool — it turns a hand-wave into a number.
- At work, find every pool and limit on your request path and check it against rate × time at your slowest dependency latency, not your average.