跪拜 Guibai
← All articles
Operations

Service Mesh Observability: The Unified Sidecar Playbook for Traces, Metrics, and Logs

By vipxieliang ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams running Istio, Linkerd, or any Envoy-based mesh can replace scattered SDK instrumentation with one unified observability layer, but only if they propagate Trace Context correctly across proxies and survive the sampling and cardinality pitfalls that silently break traces and blow up Prometheus memory.

Summary

Traditional microservice observability forces each service to embed SDKs, log manually, and report metrics, which turns into a mess of inconsistent instrumentation, broken traces across old or third-party services, and painful upgrade cycles. Service Mesh flips that by running an Envoy sidecar in front of every service, so all traffic passes through a proxy that naturally sits at the perfect observation point.

The article builds a complete observability stack on that foundation. Distributed tracing works through W3C Trace Context headers: Envoy reads `traceparent` on inbound requests, creates its own span, and injects context on outbound calls, so inter-service call trees appear even when business code has no instrumentation. Metrics follow the RED methodology, with Envoy exposing Prometheus-format stats on `:15090/stats/prometheus`, and the article provides PromQL for rate, error rate, and P99 latency. Access logs get the same Trace ID injected, letting teams drill from a Grafana alert into Tempo to find a slow trace, then into Loki to see the exact request logs.

The practical details matter: header allow-lists in Istio, consistent sampling at the ingress, avoiding high-cardinality labels like `path`, and separating trace and log storage. The article also acknowledges the mesh layer's blind spot; it cannot see inside a service, so the best setup combines sidecar observability with application-level instrumentation, unified through the same Trace Context.

Takeaways
— Envoy sidecars capture all inbound and outbound traffic, so they can collect traces, metrics, and access logs for every service without modifying application code.
— Distributed tracing relies on W3C Trace Context headers, primarily `traceparent` (format `00-<trace-id>-<parent-id>-<flags>`) and optionally `tracestate`.
— Envoy generates its own span on inbound requests and injects new context on outbound requests, so service-to-service call trees are visible even with zero business-code instrumentation.
— Istio may strip unknown headers by default, so `traceparent` and `tracestate` must be explicitly allow-listed for cross-proxy propagation.
— Sampling decisions should be made once at the entry point and passed downstream via the `traceparent` flags bit, because independent proxy sampling can fragment traces.
— Envoy exposes metrics in Prometheus format at `:15090/stats/prometheus`, with downstream, upstream, and server metrics.
— The RED methodology uses Rate (QPS), Errors (error rate), and Duration (response time distribution), implemented via PromQL like `sum(rate(envoy_http_downstream_rq_total[1m])) by (destination_service)` and `histogram_quantile(0.99, ...)`.
— Envoy's `response_flags` label (e.g., `UF` for upstream connection failure, `UO` for upstream overflow, `NR` for no route) breaks generic 5xx errors into specific root causes.
— Access logs can include Trace ID via `%REQ(X-B3-TRACEID)%` or by parsing `traceparent`, enabling drill-down from Grafana dashboards to Tempo traces to Loki logs on the same `trace_id`.
— Avoid high-cardinality labels like `path` or `user_id` in Prometheus; they cause memory to grow rapidly.
— Keep trace storage (Tempo) and log storage (Loki) separate and correlate via `trace_id`; do not try to use one storage for everything.
— Build stable Grafana dashboards before setting alert thresholds, otherwise false positives will overwhelm you.
Conclusions

The article positions mesh observability as a complement, not a replacement, for application-layer instrumentation; the sidecar sees inter-service traffic but is blind to database calls and internal logic, so a truly complete picture requires both layers sharing the same Trace Context.

The biggest operational risks in this design are not the Envoy config but the subtle propagation details: header allow-lists, sampling consistency, and matching propagation formats between the SDK and proxy, where small mistakes silently create disconnected traces.

The recommendation to control metric cardinality and separate trace/log storage reflects real cost constraints; at scale, observability data volume becomes a financial problem, and the article's sampling and storage guidance is a practical answer to that.

Grafana's Exemplar feature is presented as the cleanest bridge between metrics and traces, but it only works if trace IDs are embedded in metric labels, which itself requires careful design of the Prometheus scrape configuration.

Concepts & terms
Trace Context
A set of identifiers passed in HTTP headers (notably `traceparent` in the W3C format) that link all spans of a single request across services, letting distributed tracing tools reconstruct the full call tree.
RED methodology
A service-level monitoring framework built on Rate (requests per second), Errors (error rate), and Duration (response time distribution), often used instead of resource-oriented USE when the goal is understanding service health.
Envoy response_flags
A label on Envoy metrics and access logs that classifies proxy-level outcomes, such as `UF` (upstream connection failure), `UO` (upstream overflow), and `NR` (no route), helping distinguish why a 5xx occurred.
Grafana Exemplar
A feature that attaches a trace ID to a metric data point, allowing you to click a point on a chart and jump directly to the corresponding trace in the tracing backend.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗