Distributed Tracing Lifecycle Visualization
Walk through the Distributed Tracing lifecycle with guided steps for trace id, span id, context propagation, waterfall, sampling.
Continue with the guided workspace
Use the paired workspace to connect this guide's mental model with live component state, visible signals, and the outcome each step or scenario is meant to prove.
Distributed tracing follows a single request as it fans out across many microservices, message queues, and databases, stitching the pieces back into one end-to-end timeline. Each hop becomes a span, all spans share a trace ID, and the resulting waterfall shows where time is actually spent — which is hard to see from any single service’s logs.
For parameter experimentation and stress-testing, the Distributed Tracing Simulator is also available.
Mental Model
Distributed tracing tracks the lifecycle of a request as it flows through multiple physical services, message queues, and databases.
- Trace: Represents the entire end-to-end journey of a request.
- Trace ID: A globally unique identifier shared by every microservice participating in a single request transaction.
- Span: Represents a single unit of work within a service (e.g., an HTTP handler call or a database query).
- Span ID: A unique identifier for a specific span.
- Parent ID: Identifies the span that triggered the current span, allowing tracing UI engines to construct a tree-like hierarchy (the waterfall view).
When a service makes an external call, it injects the current trace context into request headers so that downstream services can read the parent span ID and trace ID, stitching the end-to-end flow together.
Components and Why They Exist
Client
This component exists to initiate operations and supply user request payloads to our microservices system. It dispatches the initial HTTP request or RPC trigger to the edge API Gateway. In the demo, observe the client block sending a network request to the API Gateway.
API Gateway
This component exists to act as the single entry point, generate the unique trace context, and route requests downstream. It intercepts raw client requests, generates a globally unique Trace ID and the Root Span, and routes traffic. In the demo, observe the initial root span creation and trace context header injection.
Trace Context
This component exists to flow across network boundaries, allowing downstream services to link their spans. It serializes trace identifiers, span identifiers, and sampling flags into outgoing HTTP header strings. In the demo, observe the trace context value (traceparent) moving along the network request lines.
Catalog Service
This component exists to process catalog listings and fetch stock data from downstream inventories. It receives catalog requests, extracts incoming header contexts, starts child spans, and queries downstream catalog databases. In the demo, watch span_id: sp-catalog appearing as a child of the gateway span.
Payment Service
This component exists to isolate financial operations and verify card transaction authorizations. It processes payments, coordinates with database systems, and creates child spans for local transaction threads. In the demo, observe active database connection requests and latency tracking.
Inventory Service
This component exists to track product stock levels and verify availability before confirming purchases. It runs queries against inventory storage in parallel with other billing services. In the demo, watch spans showing parallel operation alongside payment service tasks.
Database
This component exists to persist microservice states and run queries with measured execution times. It processes database commands and queries, returning results that are wrapped as database client spans. In the demo, watch SQL execution blocks and query-level latency spans.
Collector
This component exists to aggregate raw span batches from all services before indexing them. It ingests span streams, manages memory buffers, and forwards batches to the filtering pipeline. In the demo, observe memory buffer queue size and span ingestion throughput.
Sampling Gate
This component exists to filter out high-volume, healthy traces, reducing downstream storage costs. It evaluates sampling rules (head- or tail-based) and drops redundant traces while preserving anomalies. In the demo, observe trace status indicators switching between sampled and dropped.
Trace Store
This component exists to persist span records durably and build searchable trace indexes. It saves span logs and indexes traces by service name, trace ID, timestamps, and custom attributes. In the demo, observe trace logs accumulating in database indexes.
Waterfall View
This component exists to visualize nested service spans and identify call-path latency bottlenecks. It builds trace hierarchies from parent span IDs, displaying execution bars on a timeline grid. In the demo, observe the cascading gantt-chart lines highlighting bottleneck latencies.
Alerting Node
This component exists to monitor latency telemetry and notify teams when service performance breaks SLAs. It evaluates latency thresholds and raises alarms if response times degrade. In the demo, watch status lights switching to warning when dependencies slow down.
Edge Cases and Misconceptions
- Broken context propagation: If any service in a deep microservices chain is not configured to propagate context (e.g., a library update omits the outbound HTTP header middleware), the trace is broken. The downstream spans appear as separate root traces, creating fragmented graphs that make end-to-end debugging impossible.
- Clock skew in waterfalls: Span start/end timestamps are recorded locally by each service. If backend servers have drifted clocks (even by 10-20 milliseconds), waterfall UI builders can display negative latency spans, showing child spans starting before their parents did. Network time sync (NTP) is crucial.
- Asynchronous tasks as children: Developers often make async worker threads children of the triggering HTTP route. However, if the HTTP call returns early and the worker runs in the background, a standard child span will falsely inflate the parent HTTP span’s duration. Using Span Links instead of direct parent-child references correctly models async handoffs.
- Sampling bias: Relying purely on head-based sampling means a slow request or database error occurring late in the call path might be dropped if the gateway decided not to sample that trace ID at the request’s start. Tail-based sampling resolves this but introduces memory overhead on collectors.
Frequently Asked Questions
What makes spans part of one trace?
Every participating span carries the same trace ID. Parent span IDs record the causal relationship, while the trace ID keeps the complete request searchable as one timeline.
Why are payment and inventory shown in parallel?
The catalog service can start both child operations without waiting for either result. Their spans share a parent, but neither is the other’s parent, so the waterfall can show concurrent work rather than a false serial dependency.
Common Questions
How is context propagated across network boundaries?
Context propagation transfers tracing metadata (such as Trace ID, Span ID, and flags) across network boundaries using standardized protocols like the W3C Trace Context specification. The metadata is injected into outgoing HTTP request headers (e.g., as a `traceparent` header) or message metadata, and extracted by downstream services to establish parent-child relationships.
What is the difference between head-based and tail-based sampling?
Head-based sampling makes the sampling decision at the very beginning of a request's lifecycle (e.g., at the API Gateway), dropping or keeping the entire trace before knowing if it will succeed or encounter latency. Tail-based sampling buffers all spans for a trace at the collector layer and evaluates the complete trace before saving it, ensuring that rare errors and slow requests are always retained while normal traces are sampled down.
How do asynchronous message queues affect parent-child trace spans?
Asynchronous messaging breaks the synchronous request-response flow. Making a consumer process span a direct child of a producer span would incorrectly suggest the producer blocked waiting for the consumer. Instead, standard practices like OpenTelemetry use Span Links, referencing the producer's trace/span context to show correlation without implying a synchronous parent-child execution block.
Why can a trace look healthy while users still experience slowness?
A trace only explains the sampled request path that was captured. If sampling drops slow traces, client-side time is missing, or queue wait time is not instrumented, the waterfall can understate real user latency. Good tracing pairs spans with metrics, logs, and sampling rules that preserve errors and high-latency requests.