Observability Is Not Three Pillars.

At 02:47 you get paged: checkout is failing for one customer in twenty. You have all the tools. You open the metrics dashboard, and yes, the error rate is climbing. You tail the logs, and there are thousands of lines, some errors, but nothing that tells you who is failing or why. You have distributed tracing, so you go looking for a broken trace, and the sampler kept one request in a hundred and none of them are the broken ones.

Three tools, all lit up, and forty minutes in you still can't answer the one question that matters: why is checkout failing for this group of users, on this build, calling this one service, and nobody else?

Here's the trap, and it's the same in all three tools. Each one already threw away the detail you need, before you knew you needed it. The metric averaged it away. The log kept the words but not the structure to search on. The trace kept the structure, but the sampler deleted it. You weren't missing a fourth tool. You were missing the thing observability actually is, and the famous "three pillars" picture talked you out of it.

Here's the one idea to hold onto: an incident is always a question you didn't see coming, about a slice of traffic you didn't know to track. You can't ask a question at 02:47 if your data already threw the answer away.

Metrics, logs, and traces aren't three kinds of truth. They're three lossy copies of one thing: a single, detailed record of one unit of work. Keep that record whole and you can slice it however the incident needs. Split it into three stores that don't talk to each other, and you get three tools that all fail the same way.

So the whole job comes down to keeping one thing, and keeping it three ways at once:

• One record per unit of work. One request, one job, one message: a single structured record holding everything you knew about it.
• Wide, with lots of detail. Not five labels but fifty fields: user, build, region, downstream service, feature flags, all of it. The detail the next incident needs is one you haven't thought of yet.
• Linked into a chain. Each record carries the id of the work that caused it, so a request spread across twenty services stitches back into one story.

The three pillars are just this one record, damaged three different ways to make it cheap. The rest of this post is what each one throws away, which loss you can live with, and where it all falls apart at 02:47.

Metrics throw away the detail

Start with the tool you trust most, because it fails you the quietest. A metric is a record with almost every field deleted and the rest added up as they arrive. http_requests_total isn't a record of your requests, it's a running count, with the actual requests thrown away. That's why it's cheap: a counter is a few bytes no matter how many requests it counts. And it's why it can't answer the cohort question. You can't break down a total by a field the total never kept.

The obvious fix is to keep the field, add it as a label. This is the mistake almost everyone makes, because it looks free and isn't.

In a metrics store like Prometheus, every unique combination of label values becomes its own time series, each one stored and held in memory. Five regions is 5 series. Add 40 endpoints and you have 200. They multiply, they don't add. Now add user_id "just for this one incident," and with two million users you don't add two million series, you multiply everything you already had by two million.

Labels multiply, they never add:

    region(5) × endpoint(40) × status(3)                =           600 series
    region(5) × endpoint(40) × status(3) × user_id(2M)  = 1,200,000,000 series

This is a cardinality explosion, and it doesn't fail gently. A time-series database's memory grows with the number of series, so those new series don't slow it down, they crash it. And it happens at the worst time, because the high-detail label almost always gets added during an incident, when someone wants to "see the individual failures." The monitoring falls over on top of the outage it was meant to help you fix. On a good day the metric just gets dropped and your dashboard goes blank. Either way, the tool you reached for is the tool the incident just broke.

The first time I did this, I added a customer_id label to a latency histogram to chase one customer's complaint. A histogram is already a dozen series per label set (one per bucket, plus a sum and a count), so multiplying by our customer count added several million series in about a minute. Prometheus went from four gigabytes of memory to killed, and I took down the dashboards for every on-call engineer in the company to answer a question about one account. The label was the incident.

This is why the standard metric conventions, RED (rate, errors, duration) per endpoint and USE (utilization, saturation, errors) per resource, are deliberately kept low-detail. They label by things with tens of values, never millions. They answer a fixed, predictable set of questions cheaply and forever.

# RED: cheap, because every label here has only a handful of values.
sum(rate(http_requests_total{status=~"5.."}[5m])) by (endpoint)
  /
sum(rate(http_requests_total[5m])) by (endpoint)

# Want this broken down by user_id? Then you don't want a metric.
# You want the wide record, searched after the fact.

So metrics aren't wrong, they're finished. They're the answers to questions you already knew to ask, worked out in advance. Keep them for exactly that. But the cohort question at 02:47 was never going to come from here, and adding labels doesn't get you there, it just buys you a second outage.

Traces are the chain, if the ids survive every hop

So you keep the records whole, one per unit of work. Now a request that touches twenty services makes twenty records, and twenty loose records are useless. What ties them back into one story is a trace. Every record (called a span) carries the same trace_id, and each one names the span that caused it: its parent. Follow those parent links and twenty scattered spans line up into one timeline, with a duration on every hop. Now "checkout is slow" becomes "checkout is slow because the payments call waited 1.9 seconds on a downstream that's fine in the US and dying in Europe."

GET /checkout                        2.1s
├─ auth.verify                       0.1s
├─ inventory.reserve                 0.1s
└─ payments.capture                  1.9s
   └─ POST bank-api  (eu-west)       1.8s

That stitching only works if the ids travel with the request. A span in the payments service can only call itself a child of the checkout span if checkout told it the trace id and its own span id, in a header, on every single call. This is context propagation, and it's the whole game. Get it right and one trace covers your whole system. Miss it on one hop and the trace doesn't shrink, it splits in two, ending exactly at the hop you forgot.

The header is a web standard, W3C Trace Context: a traceparent header holding the trace id, the parent span id, and a sampling flag. OpenTelemetry adds it on the way out and reads it on the way in, as long as the call goes through something it has hooked into.

val propagator = openTelemetry.propagators.textMapPropagator

// On the way out: add the trace context to the outgoing headers.
// Every service must do this on every call, or the trace splits here.
val headers = mutableMapOf<String, String>()
propagator.inject(Context.current(), headers) { carrier, key, value ->
    carrier?.put(key, value)          // writes "traceparent: 00-<trace_id>-<span_id>-01"
}
httpClient.post(url, headers, body)

// On the way in: read it, so this service's spans join as children, not new roots.
val parentCtx = propagator.extract(Context.current(), request.headers,
    object : TextMapGetter<Headers> {
        override fun keys(carrier: Headers) = carrier.names()
        override fun get(carrier: Headers?, key: String) = carrier?.get(key)
    })
val span = tracer.spanBuilder("payments.capture")
    .setParent(parentCtx)             // without this, a whole new trace starts here
    .startSpan()

It almost never breaks on plain HTTP calls, because your framework handles those for you. It breaks at the seams the framework can't see. A message goes onto a Kafka queue and the traceparent isn't copied into it, so the consumer starts a fresh trace and the story ends at the queue. Work gets handed to a background thread and the context (which lives on the original thread) doesn't follow it. A webhook, a batch job, a curl in a cron: anything that comes in through a door OpenTelemetry didn't wrap starts a new trace.

And here's the cruel part: the trace dead-ends at exactly the service you forgot about, which, during an incident, is usually the service with the bug. The corner nobody instrumented and the corner with the bug tend to be the same corner.

So a trace is just wide records with a parent pointer, and that pointer is only as strong as the weakest hop it has to cross. Which leads to the problem that makes traces expensive: you can't afford to keep them all.

Sampling: you can't keep every trace, so which do you drop?

A detailed record per request, with a span per hop, sounds affordable until you multiply by real traffic. A busy service produces far more trace data than it does actual responses, and storing all of it costs more than the requests earn. So you keep some and drop the rest. The only real question is when you decide, because when you decide controls what you get to keep.

Head sampling decides at the very start of the request: hash the trace id, keep 1%, drop the rest, and pass that decision down so every service agrees. It's simple and nearly free. It also has one fatal flaw: it decides before anything has happened. Before you know if the request errored. Before you know if it was slow. So it keeps a random 1%, 1% of the boring successes and 1% of the errors. During an incident you need the errors and the slow ones, and head sampling just threw away 99% of them. You're paying for tracing, and the broken trace isn't in the slice you kept.

Tail sampling fixes that by deciding at the end, once the trace is done and you can see what happened. Keep every trace with an error. Keep every trace slower than a second. Keep a small sample of the normal ones for comparison. Now the 1% you store is the 1% worth storing, and the broken trace is right there at 02:47. The catch is the cost. To decide at the end, something has to hold all of a trace's spans until it finishes, and those spans arrive from many services, so every span of one trace has to land on the same collector. That means a load-balancing tier in front, routing by trace id, feeding collectors big enough to buffer every in-flight trace. You pay in memory and moving parts for the right 1% instead of a random one.

                head sampling                   tail sampling
  decides       at the start                    after the trace finishes
  keeps         a random 1%                     every error + slow one + a sample
  cost          almost nothing                  buffers spans, needs a routing tier
  the incident  broken trace probably dropped   broken trace kept on purpose
  use when      huge volume, tight budget       you refuse to lose the rare ones

# OpenTelemetry Collector: keep what an incident actually needs.
# (Route by trace id to this collector first, so it sees a whole trace.)
processors:
  tail_sampling:
    decision_wait: 10s          # hold a trace's spans up to 10s for it to finish
    policies:
      - name: keep-errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: keep-slow
        type: latency
        latency: { threshold_ms: 1000 }
      - name: baseline-sample
        type: probabilistic
        probabilistic: { sampling_percentage: 1 }   # a little of the normal stuff

Most real systems do both: a bit of head sampling to shed obvious load, then tail sampling so nothing interesting slips through. One rule though: never let the sampler judge a trace before the trace has had a chance to get interesting.

SLOs: the one number allowed to wake you

Now you have good data: wide records, stitched into traces, sampled so the interesting ones survive. None of it should page you. Raw data is for debugging an incident once you know there is one. What tells you there is one is a single number, and getting that number right is its own skill, because the usual mistake is to alert on everything at once.

The trap is alerting per component: CPU over 80%, this queue over 1000, that pod restarted. Each one fires on things users never feel, and the pile of them trains you to ignore the pager, so you miss the alert that mattered under the forty that didn't. You don't want to know a component is unhappy. You want to know users are, and you want that to be the only thing loud enough to wake you.

That's what a Service Level Objective is. You pick a Service Level Indicator, a ratio of good requests to total requests, like "served under 300ms without a 5xx" over "all requests," measured straight off the records you already keep. The SLO is your target: 99.9% over a rolling 30 days. This is the most predictable question you have, "are users okay?", so it's exactly the kind a cheap, low-detail metric should answer. It's the RED metric from earlier, finally earning its keep as the thing that pages you.

# SLI: good requests / all requests.
# "good" = under 300ms and not a 5xx. Low-detail on purpose.
# Record this as a rule; compute the 30-day SLO off the recorded ratio,
# don't rate() a raw 30-day window live.
sum(rate(http_request_duration_seconds_bucket{le="0.3", status!~"5.."}[5m]))
  /
sum(rate(http_request_duration_seconds_count[5m]))

The idea that makes an SLO useful day to day is the error budget. 99.9% good over 30 days means 0.1% is allowed to be bad, and that 0.1% is a budget you get to spend. Now reliability isn't "never break," it's "don't overspend," which you can actually alert on. You don't alert on "errors above X." You alert on burn rate: how fast you're spending the budget. Spending slowly enough to last the month is fine and silent. Spending a month's budget in an hour is a page right now, because you're minutes from broke. In practice you make a long window and a short window agree before firing, so a brief blip stays quiet and a real burn still pages fast: a fast burn wakes a human, a slow burn opens a ticket.

# Burn rate: how many times faster than sustainable you're spending the budget.
# Above ~14.4 you'd burn a 30-day budget in ~2 days, so page on the fast window.
(
  1 - (
    sum(rate(http_request_duration_seconds_bucket{le="0.3", status!~"5.."}[1h]))
    / sum(rate(http_request_duration_seconds_count[1h]))
  )
) / 0.001   # 0.001 = the 0.1% budget for a 99.9% SLO

One number, measured off your records, sized by how fast users are actually losing. Everything else is a dashboard you open after it tells you to.

Where observability breaks

Every failure here has a name, and knowing the names is how you catch them in review instead of at 02:47.

Context dropped at a boundary. HTTP calls trace themselves; queues, background threads, and cron jobs don't. The traceparent doesn't get copied, the trace splits, and it dead-ends on the service you forgot, usually the one with the bug. Pass the context across every boundary, not just HTTP.

Head sampling threw away the errors. You pay for tracing, and the broken trace isn't in the random 1% you kept. Sample at the tail and keep every error and slow trace on purpose, or accept that tracing goes blind exactly when you need it.

The cardinality bomb. Someone adds user_id or request_id as a metric label mid-incident and multiplies your series into the millions, crashing the metrics store on top of the outage. High-detail fields go on records, never on metric labels. If a value can be more than a few dozen things, it's not a label.

A label filled from user input. The nastier cousin of the last one. Any label fed from outside, a URL with ids in it, a User-Agent, a header, can be pushed to unlimited values by a bug or an attacker, who now has a cheap way to take down your monitoring one request at a time. Limit every label to a known set first, and strip ids out of paths (/orders/:id, not /orders/48213).

Alerting on components, not symptoms. Forty pages for CPU and queue depth that no user felt, burying the one page that mattered. Alert on the SLO burn rate, the symptom, and let component metrics be things you look at after, not things that wake you.

Clock skew between services. Span times come from each machine's own clock, so if two clocks disagree you get timelines where a child starts before its parent. Trust the parent/child links for order, not the raw timestamps, and keep the machines on NTP.

Logs with no structure. A million lines of log.info("processing order " + id) isn't observability, it's a haystack: you can't group by the failing cohort because it's buried inside a string. Log structured fields, not sentences, so the thing you need at 02:47 is a field you can filter on.

Three stores that don't join. The root problem this whole post is about. Metrics here, logs there, traces somewhere else, no shared id to jump between them, so you see a spike and can't get to the traces behind it. Put the trace id on your logs and attach it to your metric samples as exemplars, so "this spiked" is one click from "here's why."

When to reach for this, and when not

None of this is free: tail-sampling collectors, high-detail storage, context passed across every hop. So be honest about when it's worth it. The whole case rests on one thing: you can't predict the questions. That's true when you have many services, many kinds of users, and lots of variety in who's hitting you, a real distributed system with a real spread of customers. There, the next incident is a slice you haven't imagined, and only detailed, linked records can answer it.

Skip it when that's not you. One service with one database doesn't need distributed tracing, there's nothing distributed to trace, and the stack trace already tells you where it broke. A low-traffic internal tool doesn't need tail sampling, keep everything, or keep nothing and read a log. A cron job that emails a report doesn't need an error budget; it needs someone to notice when the email doesn't show up. And even in a big system, most questions are predictable, and cheap metrics answer them forever. The wide-record machinery is for the one question a month that metrics can't touch, rare, and expensive to get wrong. Which is exactly the kind of thing worth paying for, and worth not overusing.

Cheat-sheet

Screenshot this next time someone wants to add user_id as a metric label.

Before you call a system "observable":
[ ] One wide, structured record per unit of work, not just metrics + logs?
[ ] Does the trace id ride across EVERY hop, queues and threads included?
[ ] Does sampling keep errors and slow traces, not a random 1%?
[ ] Are high-detail fields on records, never on metric labels?
[ ] Is every user-supplied label limited to a known set?
[ ] Do you alert on an SLO burn rate, not per-component thresholds?
[ ] Can you jump metric → trace → log by a shared id?

tool          throws away               good for
-----------   -----------------------   ------------------------------------
metric        the individual requests   cheap, permanent answers to
                                         questions you already knew to ask
log           the structure, unless     a searchable record, only if you
              you log fields            log fields, not sentences
trace         most of itself, to the    the chain across services: where it
              sampler                   broke and how long each hop took
wide record   nothing, at write time    the question you didn't see coming

metric LABEL (few values)        record FIELD (many values)
-----------------------------    ------------------------------
endpoint, status, region         user_id, request_id, trace_id
instance                         build, session, url-with-ids
  low-detail, fixed set            high-detail, one per request

The three pillars are one detailed record, damaged three ways to make it cheap. A metric averages the details away, so it only answers questions you saw coming, and blows up the moment you try to keep one more. A log keeps the words but not the structure, so you can't group by the users who are failing. A trace keeps the structure, but the sampler deletes it, unless you sample at the tail and keep the errors on purpose. Every incident is a question you didn't see coming, about a slice you didn't know to track. So keep it all: one record per unit of work, packed with detail, its trace id riding every hop across every service, sampled by what turned out to matter, with a single burn rate the only thing allowed to wake you. You can't ask a question at 02:47 if your data already threw the answer away. Stop paying for three stores that don't talk. Keep the record whole.