跪拜 Guibai
← Back to the summary

Service Mesh Observability: The Unified Sidecar Playbook for Traces, Metrics, and Logs

1. Introduction: Why the Mesh Layer Is the Best Entry Point for Observability

Microservice architecture splits a monolith into dozens or hundreds of services, and with it the call chain grows longer and fault localization becomes harder. The traditional approach is to instrument each business service, log, and report metrics, but this has three unavoidable problems:

Service Mesh strips inter-service communication out of the business process and hands it to a sidecar proxy (e.g., Envoy) for unified handling. This means all traffic entering and leaving a service passes through the proxy, and the proxy naturally has positional advantage for collecting observability data. This article revolves around the three main threads of distributed tracing, metrics, and logs, and explains how to build a unified observability solution at the mesh layer.

The diagram below summarizes how the three main threads converge into one unified observability solution:

flowchart LR
  subgraph MESH["Mesh data plane (Envoy Sidecar)"]
    T[&#34;Distributed tracing<br/>Trace Context propagation&#34;]
    M[&#34;Metrics<br/>RED methodology&#34;]
    L[&#34;Access logs<br/>Trace ID injection&#34;]
  end
  T --> T2[&#34;Tempo / Jaeger&#34;]
  M --> M2[&#34;Prometheus&#34;]
  L --> L2[&#34;Loki / ES&#34;]
  T2 --> G[&#34;Grafana unified view&#34;]
  M2 --> G
  L2 --> G
  G --> A[&#34;Metrics find problems → Tracing locates the chain → Logs restore details&#34;]

2. The Unique Value of Mesh-Level Observability: A Full View of Inter-Service Communication

2.1 From an "Application View" to a "Network View"

Application-layer instrumentation sees "what requests my service handled," while the mesh layer sees "what happened between service A and service B." The latter can answer many questions the former cannot:

2.2 Zero Invasiveness and Language Independence

The sidecar proxy runs as an independent process, fully decoupled from the business language. As long as Java, Go, Python, and Node.js services are connected to the mesh, they automatically gain unified metrics, tracing, and access log capabilities without changing a single line of business code.

2.3 Unified Data Semantics

All service traffic passes through the same kind of proxy, which means metric names, label dimensions, and log formats are naturally unified. This lays the foundation for subsequent aggregation analysis, cross-service comparison, and SLO calculation; no more headaches over "every team names metrics differently."

2.4 The Boundary of the Full View

It must be stated objectively that the mesh layer cannot see logic inside the service. When a request calls a database, cache, or downstream HTTP interface inside a service, only application-layer instrumentation can cover those details. Therefore the best practice is to combine the mesh layer with the application layer: the mesh layer handles the full picture of inter-service communication, the application layer handles intra-service details, and the two are connected through a unified Trace Context.

The following diagram compares the observation scope of the application layer and the mesh layer, and how they are connected via Trace Context:

flowchart TD
  subgraph APP[&#34;Application layer (inside service)&#34;]
    A1[&#34;Business logic&#34;]
    A2[&#34;Database / cache calls&#34;]
    A3[&#34;Downstream HTTP interfaces&#34;]
  end
  subgraph MESH[&#34;Mesh layer (inter-service communication)&#34;]
    B1[&#34;Inbound request span&#34;]
    B2[&#34;Outbound request span&#34;]
    B3[&#34;Retry / timeout / mTLS&#34;]
  end
  A1 --> A2
  A1 --> A3
  B1 --> A1
  A3 --> B2
  B1 -.->|&#34;Trace Context linking&#34;| B2

3. OpenTelemetry and Mesh Integration: Trace Context Propagation Across Proxies

3.1 The Core of Distributed Tracing: Context Propagation

The essence of distributed tracing is linking every step a request goes through into a tree. The link is Trace Context; a set of identifiers passed along with request headers:

As long as every link in the chain can read and pass through these headers correctly, the trace connects.

The propagation of Trace Context in a cross-service call can be understood as follows:

sequenceDiagram
  participant C as &#34;Client&#34;
  participant E1 as &#34;Envoy (frontend)&#34;
  participant S1 as &#34;frontend service&#34;
  participant E2 as &#34;Envoy (order)&#34;
  participant S2 as &#34;order service&#34;
  C->>E1: &#34;Request (no traceparent)&#34;
  E1->>E1: &#34;Generate trace-id, create inbound span&#34;
  E1->>S1: &#34;Inject traceparent then forward&#34;
  S1->>E2: &#34;Call with traceparent&#34;
  E2->>E2: &#34;Read traceparent, create child span&#34;
  E2->>S2: &#34;Pass through traceparent&#34;
  S2-->>E2: &#34;Response&#34;
  E2-->>S1: &#34;Response&#34;
  S1-->>E1: &#34;Response&#34;
  E1-->>C: &#34;Response&#34;

3.2 Envoy's Tracing Capabilities

Envoy has built-in tracing support, configured via the tracing block to specify the reporting cluster and sampling policy:

tracing:
  http:
    name: envoy.tracers.opentelemetry
    typed_config:
      "@type": type.googleapis.com/envoy.config.trace.v3.OpenTelemetryConfig
      grpc_service:
        envoy_grpc:
          cluster_name: otel_collector
      service_name: mesh-proxy

The key point: Envoy reads the upstream traceparent on inbound requests, generates its own span, and injects the new context into the request headers on outbound requests. This way, even if business code has zero instrumentation, the call relationships between services are fully recorded.

3.3 Pitfalls in Cross-Proxy Propagation

Several traps to watch for in practice:

  1. Header allow-list: Some mesh implementations (e.g., Istio) clean unrecognized request headers by default; confirm that traceparent and tracestate are in the pass-through allow-list.
  2. Sampling consistency: If each proxy samples independently, upstream might sample while downstream does not, breaking the chain. It is recommended to make sampling decisions uniformly at the entry and pass them down via the flags bit of traceparent.
  3. Application and proxy layer handoff: If business services also use the OpenTelemetry SDK, ensure the SDK and Envoy use the same propagation format, otherwise you get two disconnected chains.
  4. Async message scenarios: Context propagation for message brokers like Kafka and RabbitMQ requires extra configuration; you cannot rely only on HTTP headers.

3.4 A Typical Trace Shape

flowchart LR
  A[&#34;Client&#34;] --> B[&#34;Envoy (frontend)&#34;]
  B --> C[&#34;frontend service&#34;]
  C --> D[&#34;Envoy (order)&#34;]
  D --> E[&#34;order service&#34;]
  E --> F[&#34;Envoy (payment)&#34;]
  F --> G[&#34;payment service&#34;]

In this chain, each Envoy produces both inbound and outbound spans; if the business service also has instrumentation, it adds internal spans. What you see in Tempo/Jaeger is a complete call tree that clearly distinguishes "network time" from "service processing time."

4. Metric Collection: From Envoy Metrics to the RED Methodology

4.1 What Metrics Envoy Exposes

Envoy's metric system is divided into three layers:

These metrics are exposed by default in Prometheus format on the admin port (usually :15090/stats/prometheus) and can be scraped directly by Prometheus.

4.2 The RED Methodology

RED is a service-oriented metric methodology, with the three letters standing for:

The correspondence between the three RED dimensions and Envoy metrics and observation goals is as follows:

flowchart LR
  R[&#34;Rate<br/>Request rate&#34;] --> R1[&#34;envoy_http_downstream_rq_total&#34;]
  E[&#34;Errors<br/>Error rate&#34;] --> E1[&#34;envoy_http_downstream_rq_xx&#34;]
  D[&#34;Duration<br/>Response time&#34;] --> D1[&#34;envoy_http_downstream_rq_time_bucket&#34;]
  R1 --> Q[&#34;QPS dashboard&#34;]
  E1 --> Q2[&#34;Error rate dashboard&#34;]
  D1 --> Q3[&#34;P99 latency dashboard&#34;]

Compared to the USE (resource-oriented) methodology, RED suits inter-service communication observability better. Based on Envoy metrics, you can construct queries like:

# Rate: requests per second
sum(rate(envoy_http_downstream_rq_total[1m])) by (destination_service)

# Errors: 5xx error rate
sum(rate(envoy_http_downstream_rq_xx{response_code_class="5"}[1m])) by (destination_service)
/
sum(rate(envoy_http_downstream_rq_total[1m])) by (destination_service)

# Duration: P99 response time
histogram_quantile(0.99,
  sum(rate(envoy_http_downstream_rq_time_bucket[1m])) by (le, destination_service)
)

4.3 Planning Metric Labels

The value of metrics depends heavily on label dimensions. At minimum, keep the following labels:

Label Description Purpose
source_workload Caller workload Locate problem source
destination_workload Callee workload Locate problem target
destination_service Target service name Service-level aggregation
response_code HTTP status code Error analysis
response_flags Envoy response flags Distinguish timeout/reset/no healthy upstream

Among these, response_flags is especially useful: UF indicates upstream connection failure, UO indicates upstream overflow, NR indicates no route. These flags quickly break down "5xx" into specific causes.

4.4 Metric Collection Architecture

flowchart TD
  A[&#34;Envoy Sidecar&#34;] -->|&#34;scrape :15090&#34;| B[&#34;Prometheus&#34;]
  B --> C[&#34;Recording Rules&#34;]
  C --> D[&#34;Grafana dashboard&#34;]
  C --> E[&#34;Alert rules Alertmanager&#34;]

For large clusters, use Prometheus federation or solutions like Thanos/VictoriaMetrics for long-term storage and cross-cluster aggregation to avoid excessive load on a single instance.

5. Log Correlation: Linking Trace ID with Access Logs

5.1 The Value of Access Logs

Metrics tell you "there is a problem," tracing tells you "which chain the problem is on," and logs tell you "what actually happened." Envoy's access logs record detailed information about each request, including response code, duration, upstream address, response flags, and more.

5.2 Injecting Trace ID into Access Logs

This is the key step for linking logs and traces. Envoy supports using %REQ(X-REQUEST-ID)% or extracting the trace ID via %DYNAMIC_METADATA% in the access log format:

access_log:
  - name: envoy.access_loggers.file
    typed_config:
      "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
      path: /dev/stdout
      log_format:
        json_format:
          timestamp: "%START_TIME%"
          method: "%REQ(:METHOD)%"
          path: "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
          protocol: "%PROTOCOL%"
          response_code: "%RESPONSE_CODE%"
          response_flags: "%RESPONSE_FLAGS%"
          duration_ms: "%DURATION%"
          upstream_host: "%UPSTREAM_HOST%"
          trace_id: "%REQ(X-B3-TRACEID)%"
          span_id: "%REQ(X-B3-SPANID)%"

Note: different tracing formats use different header names. When using the W3C standard, traceparent is a combined header and requires extra parsing; with B3 format, you can directly extract X-B3-TraceId.

5.3 Log Collection and Correlated Queries

Logs are collected by collectors such as Fluent Bit / Vector and written to Loki or Elasticsearch. The key to correlation analysis is using the same Trace ID as the query condition:

  1. Discover a P99 spike for some service on the Grafana dashboard;
  2. Click to drill down, jump to Tempo to view slow traces in that period;
  3. Select a slow trace and get the trace_id;
  4. Query all related logs for that request in Loki with {app="order"} |= "<trace_id>".

This path connects "metrics find problem → tracing locates chain → logs restore details" into a closed loop.

The complete correlation troubleshooting loop is shown below:

flowchart TD
  A[&#34;Grafana dashboard sees P99 spike&#34;] --> B[&#34;Drill down to Tempo&#34;]
  B --> C[&#34;View slow traces in period&#34;]
  C --> D[&#34;Select trace, get trace_id&#34;]
  D --> E[&#34;Loki query logs by trace_id&#34;]
  E --> F[&#34;Restore request details, locate root cause&#34;]
  F --> G[&#34;Return to dashboard to verify fix&#34;]
  G --> A

5.4 Log Field Standardization Suggestions

For efficient correlation queries, standardize log fields:

At the same time, control log volume: full logging of access logs is costly at high QPS. Record errors and slow requests in full, and sample normal requests proportionally.

6. Practical: A Mesh Observability Stack Based on Prometheus + Grafana + Tempo

6.1 Overall Architecture

flowchart TD
  subgraph MESH[&#34;Service Mesh data plane&#34;]
    A1[&#34;Envoy Sidecar A&#34;]
    A2[&#34;Envoy Sidecar B&#34;]
  end
  A1 -->|&#34;OTLP traces&#34;| C[&#34;OpenTelemetry Collector&#34;]
  A2 -->|&#34;OTLP traces&#34;| C
  A1 -->|&#34;metrics :15090&#34;| D[&#34;Prometheus&#34;]
  A2 -->|&#34;metrics :15090&#34;| D
  A1 -->|&#34;access log&#34;| E[&#34;Fluent Bit&#34;]
  A2 -->|&#34;access log&#34;| E
  C --> F[&#34;Tempo&#34;]
  E --> G[&#34;Loki&#34;]
  D --> H[&#34;Grafana&#34;]
  F --> H
  G --> H

6.2 Deploying OpenTelemetry Collector

The Collector handles receiving, batching, and forwarding trace data:

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlp:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]

6.3 Configuring Prometheus to Scrape Envoy Metrics

scrape_configs:
  - job_name: 'envoy-sidecars'
    metrics_path: /stats/prometheus
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_container_name]
        action: keep
        regex: istio-proxy
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: destination_workload
      - source_labels: [__address__]
        regex: '(.+):\d+'
        target_label: __address__
        replacement: '${1}:15090'

6.4 Grafana Dashboard Design

A practical mesh observability dashboard usually includes four sections:

  1. Global overview: overall QPS, error rate, P99 latency, grouped by service;
  2. Service details: after selecting a service, show its upstream/downstream call relationships and status code distribution;
  3. Exception drill-down: error trends categorized by response_flags, clickable to jump to Tempo;
  4. Log panel: embed a Loki query to show error logs in the current time window.

Grafana's Exemplar feature can attach trace_id directly to metric charts; clicking a data point jumps to the corresponding trace, which is the most elegant way to bridge metrics and traces.

6.5 Alert Rule Examples

groups:
  - name: mesh-slo
    rules:
      - alert: HighErrorRate
        expr: |
          sum(rate(envoy_http_downstream_rq_xx{response_code_class="5"}[5m])) by (destination_service)
          /
          sum(rate(envoy_http_downstream_rq_total[5m])) by (destination_service)
          > 0.05
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Service {{ $labels.destination_service }} 5xx error rate exceeds 5%"

      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum(rate(envoy_http_downstream_rq_time_bucket[5m])) by (le, destination_service)
          ) > 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Service {{ $labels.destination_service }} P99 latency exceeds 1s"

6.6 Lessons from Implementation

7. Summary

Service Mesh provides a unified, zero-intrusion data collection layer for observability. Through OpenTelemetry's Trace Context for cross-proxy propagation, through Envoy metrics implementing the RED methodology, and through Trace ID linking access logs with traces, the Prometheus + Grafana + Tempo stack forms a complete closed loop of "metrics find problems, tracing locates the chain, logs restore details."

It is worth emphasizing that mesh-level observability is not a replacement for application-level observability but a complement. Only by combining the two with consistent semantics can you truly achieve comprehensive control over a distributed system.