You Sampled 1% of Traces, and the Only Request That Mattered Wasn't in It

The alert fires at 2:14am. Checkout p99 for one tenant is sitting at 11 seconds, up from 400ms. You have the number, you have the window, you know the route.

You open the tracing UI, filter to that tenant and that endpoint, and get four traces back. All four are healthy and sub-second.

The slow requests happened. They were just never written down. Nothing in your stack is broken either, which is the part that stings: the sampler did exactly what you configured it to do.

Your sampler decides before anything interesting has happened

Most OpenTelemetry SDKs ship with parentbased_traceidratio as the default. The service that starts the trace hashes the trace id, compares it against your ratio, and writes one bit into the traceparent header going downstream. That bit is the entire decision. The W3C trace context spec calls it the sampled flag.

00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
                                                     ^^ sampled

Downstream services read that bit and obey it, and they should. A trace where three services opted in and two opted out is worse than no trace at all, because it looks complete. So by the time the payment call times out and the ledger write burns three retries, the keep-or-drop call is long settled, made by a process whose only input was a random number.

Run the arithmetic on a 1% ratio. If 0.3% of requests fail, you keep roughly three failed traces per 100,000 requests. Your p99.9 is rarer than that. You're sampling uniformly across requests when the events you actually need are distributed nothing like uniformly.

I watched a team respond to this by cranking the ratio to 25%. The trace bill went up accordingly and they still missed the next incident, because a quarter of a rare event is still a rare event.

Keep every span, decide at the end

Tail sampling flips the order. Services emit 100% of spans to a collector, the collector buffers them keyed by trace id, waits until the trace looks finished, and then decides based on what the finished trace contains. The decision finally gets to read the thing it's deciding about.

The config is not exotic:

processors:
  tail_sampling:
    decision_wait: 30s
    num_traces: 100000
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: slow
        type: latency
        latency: { threshold_ms: 2000 }
      - name: paying-tenants
        type: string_attribute
        string_attribute: { key: tenant.tier, values: [enterprise] }
      - name: baseline
        type: probabilistic
        probabilistic: { sampling_percentage: 2 }

Policies are OR'd, so a trace matching any one of them survives. That last probabilistic policy matters more than it looks. Without a healthy baseline you end up with a trace store containing only broken requests and nothing to compare them against, which is its own kind of blind.

There's one hard constraint. Every span in a trace has to land on the same collector instance, otherwise each instance renders a verdict on its own fragment. That forces two tiers: a front tier that does no sampling and uses the loadbalancing exporter with routing_key: traceID, and a back tier that runs tail_sampling.

What the sampler tier costs you

Storage gets cheaper. Egress and CPU do not. You are now moving every span across the network in order to throw 98% of them away. On a chatty service emitting 40 spans per request that's real traffic, and most vendors bill ingest, not just retention.

The back tier holds state, so it stops being a pod you can kill without thinking. Budget memory as spans per second times decision_wait times span size, and accept that a rollout drops whatever is buffered mid-flight.

Then the failure mode nobody warns you about: decision_wait is a deadline, not a promise. Stretch a trace across a queue hop or a retry with backoff, and the span that shows up at 40 seconds arrives after the verdict already shipped. You get a trace that looks whole and isn't, which is a worse lie than a gap. Long async work wants to be its own trace with a span link pointing home, not one trace spanning a workflow.

Autoscaling is awkward for the same reason. You can't park the sampler tier behind a plain round robin balancer, and scaling it reshuffles trace id routing, which splits traces for the duration of the scale event.

The version that fits in an afternoon

If owning a stateful collector tier isn't where you want to be this quarter, two cheaper moves get most of the value.

Set head sampling per route. /healthz at 0%, /checkout at 100%, the long tail at 1%. It's crude and it works, because you generally already know which endpoints you'd be squinting at during an incident.

Then wire up exemplars, so your latency histogram buckets carry trace ids and the p99 bucket links to a request that actually lived in it. This one is my favourite for the effort involved; it turns a dashboard from a dead end into a door. It only pays off if that trace survived sampling, so it needs the always-keep-errors rule next to it.

Metrics tell you something happened. Traces tell you why. You can't rebuild the second from the first, which makes your sampling policy the component that quietly decides which incidents you're permitted to debug after the fact.