Metrics: Counters, Gauges & Histograms

The average is lying to you#

Start with the paging story, because it's the whole point. Your service handled a hundred requests in the last second. Ninety-nine of them came back in 10 milliseconds. One of them took 2 full seconds — maybe it hit a cold cache, maybe it queued behind a slow query. The user on that one request is angry enough to tweet about it. Now: what does the average latency say?

Work it out. The total time is 99 × 10ms plus 1 × 2000ms, which is 990 + 2000, about 2990 milliseconds across 100 requests. Divide and the mean is roughly 30 milliseconds. Your dashboard shows 30ms and glows green. But a real user just waited 2 seconds, and if you get a thousand requests a second, that's ten furious users every second the graph calls healthy. The average didn't lie about the math — it answered a question nobody asked. You don't care about the typical request. You care about the worst ones users actually hit.

The four kinds of metric#

A service exposes its metrics as named numbers that a collector scrapes every few seconds. There are only four shapes, and picking the right one is most of the skill. Start with the two simple ones.

A counter is a number that only ever goes up. It counts events since the process started: total requests served, total errors, total bytes sent. It never decreases — the only reset is when the process restarts and it drops back to zero. You almost never read a counter's raw value; what you want is how fast it's climbing. So you apply a rate: 'requests_total went from 8000 to 8500 over the last 10 seconds' means 50 requests per second. The counter stores the running total; the rate turns it into the throughput you actually graph.

A gauge is a number that goes up and down. It's a snapshot of something right now: memory in use, number of active connections, queue depth, CPU temperature. Unlike a counter, its current value is the point — you read it directly, and you take its average, max, or min over a window. The test for which one you have: if it can decrease, it's a gauge; if it only accumulates, it's a counter. 'Requests served' is a counter. 'Requests in flight' is a gauge, because a request can finish and bring it back down.

The third shape is the histogram, and it exists to answer the percentile question the average couldn't. Instead of keeping one number, a histogram keeps a row of buckets. Each bucket is itself a counter: 'how many requests finished in under 10ms', 'under 50ms', 'under 100ms', 'under 1s', and so on. You pre-declare the bucket boundaries when you instrument the code. Every observation falls into a bucket and bumps its count. Because the buckets together describe the shape of the whole distribution, the collector can estimate any percentile from them — p50, p95, p99 — without ever storing a single raw measurement.

The fourth shape is the summary. It answers the same percentile question but computes the answer inside your application, over a sliding time window, and exposes the finished numbers — 'my p99 over the last 10 minutes is 340ms'. It hands you the quantile directly instead of buckets you have to interpret. That sounds strictly better, and for a single instance it often is. But it hides a trap that only shows up when you have more than one server, which is the whole next section.

Here are the four side by side. Read the last column first — it's the one that trips people up.

The aggregation trap: never average a percentile#

You run the same service on twenty machines. Each one emits its own metrics. The dashboard has to fold twenty streams into one number per graph — and whether that's correct depends entirely on the metric type. This is where the histogram and the summary, which looked interchangeable, split apart.

Counters are easy: throughput is additive, so the fleet's request rate is the sum of each server's rate. Twenty servers at 50 requests a second is 1000 a second. Nothing subtle there.

Histograms are almost as easy, and this is their superpower. A histogram is just a set of bucket counts, and counts add. To get the fleet-wide latency distribution you add bucket 'under 10ms' across all twenty servers, add bucket 'under 50ms' across all twenty, and so on. That gives you one merged histogram describing every request the fleet served — and now you compute the true p99 over all of it. Merge first, then compute the percentile. It's exactly right.

That's the real distinction between the two percentile types. A histogram ships the raw material (bucket counts) and lets the server merge and then compute, so it aggregates correctly across a fleet. A summary ships the finished answer per instance, which is cheaper to store and accurate for one machine, but mathematically un-mergeable. This is why Prometheus and most large deployments default to histograms: at fleet scale, mergeability beats a slightly nicer single-instance number. The mergeable-sketch idea has its own topic — the t-digest simulator (/tdigest/sim) shows how you can estimate a p99 over a huge stream from a small structure that you can still merge, which is the property a raw percentile lacks.

Cardinality: the label that blows up the bill#

Metrics carry labels — key-value tags that let you slice one metric many ways. A request counter isn't just requests_total; it's requests_total{method="GET", status="200", endpoint="/cart"}. Labels are what make metrics useful: you can graph errors by endpoint, or latency by region. But they hide a cost that scales in a way that surprises people, and it's the single most expensive mistake in monitoring.

Here's the rule that decides your bill: every unique combination of label values is a separate time series, stored and indexed on its own. One metric with 4 HTTP methods, 5 status codes, and 20 endpoints isn't one series — it's 4 × 5 × 20 = 400 series, each with its own samples over time. That's fine. The trouble starts when a label has many possible values.

Add a user_id label so you can break requests down per user. You have a million users. Now multiply: 400 × 1,000,000 = 400 million time series for a single metric. Each series costs memory (the monitoring system typically holds every active series in RAM), disk, and index space. A number that was 400 became 400 million because one label was unbounded. Monitoring systems fall over from exactly this — it's called a cardinality explosion, and it usually arrives as an out-of-memory crash on the monitoring host, not on the service you were watching.

There's a lighter-weight way to get the 'how many distinct users' number without paying per-user cardinality: estimate it. Counting unique values cheaply is its own problem, and the HyperLogLog simulator (/hll/sim) shows how you can count millions of distinct items in a couple of kilobytes instead of storing one series each. When you truly need a distinct-count, estimate it as a single gauge — don't reach for a per-value label.

What to measure: RED and USE#

Knowing the metric types tells you how to measure. It doesn't tell you what. A service has thousands of possible numbers; instrument all of them and you drown, instrument the wrong ones and you're blind. Two small frameworks answer 'what', and they split by what you're looking at: a request-serving service, or a physical resource.

RED is for a service — anything that handles requests, like an API, a gateway, or a microservice. Tom Wilkie named it at Weaveworks around 2016. For every service, measure three things:

• Rate — requests per second the service is handling (a counter you rate()).
• Errors — how many of those requests are failing (a counter, usually read as a fraction of the rate).
• Duration — how long requests take, as a distribution so you get the p99 (a histogram).

Those three answer 'is my service serving users well?' from the caller's side, and because every service reports the same three, one dashboard shape works for your whole architecture — you can put someone on call for a service they've never seen and they still know which three graphs to read.

USE is for a resource — a CPU, a disk, a memory pool, a network link. Brendan Gregg named it. For every resource, measure three different things:

• Utilization — what fraction of the time the resource is busy (a disk at 90% busy).
• Saturation — how much work is queued waiting because the resource is full (the run queue length).
• Errors — error events from the resource (dropped packets, disk errors).

The pairing is the point: RED watches the work (requests flowing through a service), USE watches the machinery (resources the work runs on). When a RED dashboard shows latency climbing, a USE dashboard on the underlying resources usually shows which one saturated. Use both — RED to notice users hurting, USE to find the bottleneck causing it.