Operations

OpenTelemetry in Production: Skyscanner's 24-Cluster Governance vs Mastodon's Minimalist Collector

By vipxieliang · Nov 4, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

OTel standardization stops at the protocol and SDK; the operational choices around Collector deployment and sampling determine cost, reliability, and team burden. These two blueprints give Western teams concrete reference points for matching observability architecture to their actual scale instead of overbuilding on day one.

Summary

Skyscanner's observability platform team manages 24 production clusters with a three-tier collector layout: node-level DaemonSet agents handle lightweight enrichment, cluster-level Gateways do sampling, routing, and rate limiting, and a regional aggregation layer reduces backend write pressure. All configuration lives in a GitOps repository with canary rollouts and automatic rollback, and every collector runs memory_limiter to prevent OOM.

Mastodon takes the opposite extreme: one Collector Deployment per Namespace, replicas: 1, config in the same repo as the app manifests. Isolation comes from the Namespace boundary, failure impact stays local, and a newcomer can understand the setup in half a day. The cost is linear instance growth and no cross-Namespace global sampling.

The article's takeaway is not that one approach is superior, but that deployment architecture determines the ceiling for governance. Teams with a dedicated platform engineer and more than ten clusters should follow Skyscanner; teams with few Namespaces and no specialist should start with Mastodon and head sampling, only adding a Gateway as scale grows.

Takeaways

— Skyscanner runs 24 production clusters with a three-layer Collector architecture: DaemonSet agents on each node, a Gateway Deployment per cluster, and a regional aggregation layer before backend storage.

— All Collector configuration for Skyscanner lives in one GitOps repository, with canary validation on a single cluster before gradual rollout and automatic rollback when metrics degrade.

— Mastodon deploys one OTel Collector Deployment per Namespace with a single replica, versioning the Collector ConfigMap together with application manifests.

— The article recommends memory_limiter at the front of the Collector pipeline and batch at the end; exporters should configure sending_queue and retry_on_failure for backend flapping.

— A combined sampling strategy uses 10%~20% head sampling at the SDK, then tail sampling in the Collector that retains 100% of error traces and slow traces over 500ms, sampling baseline traces at 5%.

— Skyscanner uses a DaemonSet + Gateway combination; Mastodon approximates the Gateway role with a Namespace-level Deployment scoped to one tenant.

Conclusions

The decision path assumes most teams do not have a dedicated observability engineer, so it steers them toward DaemonSet and head sampling before considering Gateway aggregation, which is a pragmatic counterweight to the industry's default bias toward full tail sampling and centralized Gateways.

Mastodon's per-Namespace Collector makes multi-tenancy a non-issue by construction: the Namespace is the tenant boundary, so there is no routing logic to write and no cross-tenant leak to worry about.

The article's insistence on versioned configuration and canary rollouts is a direct acknowledgement that Collector misconfiguration is a common production incident source, and that GitOps discipline scales with cluster count, not team size.

Concepts & terms

Head sampling

Sampling done at the application SDK level before telemetry leaves the process, typically by a fixed probability. Cheap and simple, but cannot make decisions based on the entire trace because the full trace is not yet assembled.

Tail sampling

Sampling done at the Collector after a trace is complete, based on its full context. Can preserve error spans, slow latencies, or other criteria while discarding ordinary spans, at the cost of holding traces in memory for a decision window.

OTel Collector Gateway

A centralized or cluster-level deployment of the OpenTelemetry Collector that aggregates traffic from agents, applies sampling, filtering, routing, and rate limiting, then exports to backends. Requires high availability design because it concentrates load.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗