OpenTelemetry Production Practice: Skyscanner's 24-Cluster Management Experience and Mastodon's Small-Team Approach

1. Introduction: Why Two Reference Implementations Are Needed

OpenTelemetry (OTel) has become the de facto standard for cloud-native observability, but the "standard" only solves the protocol and SDK level. When truly deployed in production, the hardest part is often not instrumentation, but how to deploy and manage the Collector, and how to control cost.

This article breaks down two public reference implementations:

• Skyscanner: a large-scale unified management solution across 24 production clusters and hundreds of Collector instances;
• Mastodon: a minimalist design with one Collector per Namespace, maintained by a small team.

Neither is better or worse; it's a matter of matching scale and team constraints. Understanding their trade-offs is more valuable than copying any single configuration.

2. Comparison of Two Reference Implementations: Large-Scale vs Small-Team OTel Strategy

Here is an overall comparison table; subsequent sections expand on each item.

Dimension	Skyscanner	Mastodon
Cluster scale	24 production clusters	Few clusters, single team
Collector deployment mode	Gateway (centralized) + few DaemonSets	One Deployment per Namespace
Configuration management	GitOps + centralized config distribution	Versioned with application manifests
Multi-tenant isolation	Pipelines divided by cluster/team	Naturally isolated by Namespace
Sampling strategy	Tiered sampling + tail sampling	Primarily head sampling, low ratio
Team investment	Dedicated observability platform team	Part-time maintenance, pursuing "zero ops"
Core goal	Unified governance, cost control	Simple, predictable, sufficient

One-sentence summary: Skyscanner trades architectural complexity for governance, Mastodon trades feature restraint for maintenance cost.

From an overall strategy perspective, the trade-offs of the two approaches can be abstracted into a comparison path:

flowchart LR
  subgraph S["Skyscanner: Large-scale governance"]
    S1["24 production clusters"] --> S2["Tiered Collectors"]
    S2 --> S3["GitOps centralized config"]
    S3 --> S4["Tiered sampling + tail sampling"]
  end
  subgraph M["Mastodon: Minimalist for small teams"]
    M1["Few clusters / single team"] --> M2["Namespace-level Collector"]
    M2 --> M3["Config versioned with app"]
    M3 --> M4["Primarily head sampling"]
  end
  S4 --> R["Unified OTLP protocol and backend"]
  M4 --> R

3. Skyscanner's Approach: Collector Management Architecture Across 24 Production Clusters

3.1 Architecture Layers

Skyscanner's observability platform team faces a core problem: 24 clusters, many business teams, multiple signals (traces/metrics/logs) — how to avoid each team doing their own thing.

Their approach is a layered Collector:

flowchart TD
  A["Application Pods (OTel SDK)"] --> B["Node-level Agent (DaemonSet)"]
  B --> C["Cluster-level Gateway (Deployment)"]
  C --> D["Regional aggregation layer"]
  D --> E["Backend storage (Tempo / Mimir / Loki)"]

• Node-level Agent: runs as a DaemonSet, receives data from applications on the same node via OTLP, does the lightest processing (e.g., resource attribute enrichment, batching).
• Cluster-level Gateway: each cluster has a set of Gateways, handling heavy tasks like sampling, filtering, routing, rate limiting.
• Regional aggregation layer: aggregates across clusters before writing to backend, reducing backend connections and write pressure.

3.2 Configuration Management: Treating the Collector as Code

24 clusters mean 24 (or more) configurations. Skyscanner's key experience:

• Centralized versioned config: all Collector configs live in a unified repository, distributed via GitOps, no manual changes on live configs;
• Layered config reuse: base config (batch, memory_limiter, health_check) as common snippets, each cluster only overrides differences;
• Canary changes: config changes are validated on a single cluster first, then gradually rolled out, avoiding one bad config breaking all clusters.

The flow from commit to full rollout can be summarized as:

flowchart TD
  A["Modify config repository"] --> B["CI validates config validity"]
  B --> C["Single-cluster canary rollout"]
  C --> D{"Are metrics and error rates normal?"}
  D -->|No| E["Auto-rollback and alert"]
  D -->|Yes| F["Gradually roll out to remaining clusters"]
  F --> G["Full rollout and log change"]

3.3 Stability Points at Scale

• Every Collector must configure memory_limiter to prevent OOM from traffic spikes;
• Add load balancing in front of the Gateway to avoid hot spots on single instances;
• Implement backpressure protection for downstream backends: when queues are full, drop low-priority data first rather than blocking applications.

4. Mastodon's Approach: Minimalist Per-Namespace Collector Design

4.1 Design Thinking

Mastodon's scale is far smaller than Skyscanner's, and the team has no dedicated observability engineer. Their choice is straightforward: deploy one Collector Deployment per Namespace.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: mastodon
spec:
  replicas: 1
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
        - name: otel-collector
          image: otel/opentelemetry-collector-contrib:0.100.0
          args: ["--config=/conf/config.yaml"]
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          volumeMounts:
            - name: config
              mountPath: /conf
      volumes:
        - name: config
          configMap:
            name: otel-collector-config

4.2 Why This Is Enough

• Natural isolation: the Namespace itself is a tenant boundary; one Collector only handles data for that Namespace, no complex routing rules needed;
• Config moves with the app: the Collector's ConfigMap and business manifests live in the same repository; changing config and changing app is the same PR;
• Small failure domain: if a Collector goes down, only its Namespace is affected, no global cascade;
• Low cognitive load: no Gateway layer, no cross-cluster aggregation; a newcomer can understand it in half a day.

The data flow is straightforward: each Namespace is an independent pipeline.

flowchart LR
  subgraph NS1["Namespace: mastodon"]
    A1["Application Pod"] --> C1["otel-collector"]
  end
  subgraph NS2["Namespace: mastodon-streaming"]
    A2["Application Pod"] --> C2["otel-collector"]
  end
  C1 --> B["OTLP backend"]
  C2 --> B

4.3 Costs and Boundaries

The costs of this approach are also clear:

• Instance count grows linearly with Namespace count, so resource utilization is not high;
• Cannot do global sampling and aggregation across Namespaces;
• Each Collector must separately configure backend exporters, which is repetitive.

Applicable boundary: when Namespace count is within a few dozen, the team has no dedicated platform engineer, and cost is not extremely sensitive.

5. Choosing Collector Deployment Mode: DaemonSet, Sidecar, or Gateway

The three modes each have their use cases; production often combines them.

Mode	Pros	Cons	Typical scenario
DaemonSet	One per node, resource-controlled; collects host metrics locally	No cross-node aggregation; upgrade affects whole node	Host metrics, log collection
Sidecar	Same lifecycle as app, best isolation	Instance count explodes, high resource overhead	Strong multi-tenant isolation, special protocol conversion
Gateway	Centralized processing, good for sampling/routing/rate limiting	Single point of pressure, needs HA design	Unified egress, tail sampling

5.1 Selection Advice

• Start with DaemonSet by default: covers most node-level collection needs with the least ops cost;
• Add Gateway when centralized governance is needed: "heavy work" like sampling, masking, routing lives in the Gateway; agents stay light;
• Use Sidecar sparingly: only for strong isolation or protocol conversion; otherwise instance count spirals.

Skyscanner uses a DaemonSet + Gateway combination; Mastodon uses a Namespace-level Deployment that approximates the Gateway role but scoped to a single tenant.

The positional relationship of the three deployment modes in the data pipeline:

flowchart TD
  A["Application Pod"] -->|"Inside same pod"| S["Sidecar Collector"]
  A -->|"OTLP"| D["Node-level DaemonSet Agent"]
  S --> G["Gateway Collector"]
  D --> G
  G -->|"Sampling / Masking / Routing"| B["Backend storage"]

6. Sampling Strategy and Data Pipeline Optimization: Controlling Cost While Maintaining Observability

6.1 Tiered Sampling

Sampling is the most effective cost-control measure, but blind global downsampling loses critical traces. Recommended tiering:

• Head Sampling: done at the SDK side at a ratio; lowest cost, but cannot make decisions based on full traces;
• Tail Sampling: done at the Collector side based on complete trace; can retain all error traces and slow traces;
• Combined strategy: first 10%~20% head sampling at SDK side, then tail sampling at Gateway side to keep 100% of errors and slow requests exceeding a threshold.

The decision points in the pipeline:

flowchart LR
  A["Application SDK"] -->|"Head sampling 10%~20%"| B["Collector"]
  B --> C{"Tail sampling decision"}
  C -->|"Error traces"| D["100% retained"]
  C -->|"Slow traces > 500ms"| D
  C -->|"Normal traces"| E["5% probability retained"]
  D --> F["Backend storage"]
  E --> F

6.2 Tail Sampling Configuration Example

processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100000
    policies:
      - name: errors-policy
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces-policy
        type: latency
        latency:
          threshold_ms: 500
      - name: baseline-policy
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

6.3 Data Pipeline Optimization Points

• Batching: put the batch processor at the end of the pipeline to reduce network round trips;
• Memory protection: put memory_limiter at the very front of the pipeline, before all processing logic;
• Attribute trimming: use attributes / transform processors to remove useless tags, lowering storage cost;
• Queue and retry: configure sending_queue and retry_on_failure on exporters to avoid data loss during backend flapping;
• Separate pipelines by signal: use independent pipelines for traces, metrics, logs to avoid cross-interference.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [prometheusremotewrite]

7. Conclusion: How to Choose the Right Approach for Your Team

Abstract the two approaches into a decision path:

flowchart TD
  A["Does the team have a dedicated observability platform engineer?"] -->|Yes| B["Are there more than 10 clusters?"]
  A -->|No| C["Are Namespaces within a few dozen?"]
  B -->|Yes| D["Skyscanner-style: layered Gateway + GitOps centralized governance"]
  B -->|No| E["Single-cluster Gateway + DaemonSet"]
  C -->|Yes| F["Mastodon-style: Namespace-level Collector"]
  C -->|No| G["Start with head sampling, then gradually introduce Gateway"]

Core conclusions:

1. Solve the deployment pattern first, then talk about sampling: the deployment architecture determines the ceiling of subsequent governance;
2. Config must be versioned: regardless of scale, manual changes to live Collector configs are a source of incidents;
3. Sampling must be tiered: head sampling controls baseline cost, tail sampling preserves critical traces;
4. Don't pursue unification too early: small teams that copy large-scale approaches are often dragged down by ops complexity first.

OpenTelemetry's value is that it gives you a unified protocol and components, but architecture decisions still must return to your own scale, team, and cost constraints.