跪拜 Guibai
← All articles
Big Data · Database · SQL

Databend's OpenTelemetry Exporter Turns Logs, Traces, and Metrics into SQL Queryable Tables

By Databend ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Storing OTel signals in a lakehouse removes the friction of hopping between separate log, trace, and metric systems. Engineers can join observability data with order tables in one SQL query, which speeds up incident response and opens the door to long-term retention and business-impact analysis without building custom pipelines.

Summary

When a payment fails, the error log is only the start. You need to follow the trace, check latency, and see which orders were affected. If logs, traces, and metrics live in separate systems, that means cross-checking Elasticsearch, Jaeger, and Prometheus. Databend's open-source OpenTelemetry exporter solves that by writing OTLP data straight into Databend Cloud as three tables: otel_logs, otel_traces, and otel_metrics. The exporter uses the native Streaming Load API, preserves raw OTLP objects in VARIANT columns, and stores timestamps as UInt64 nanoseconds to avoid precision loss.

A walkthrough on Databend Cloud runs a simulated e-commerce payment flow across four services. With about 8% of payments failing and 10% exceeding 400 ms, the demo generates 6,300 logs, 13,628 spans, and 693 metric points from 2,100 checkouts. Using SQL, you can filter error logs, reconstruct a call chain via trace_id, compute P95 latency per service, and join logs with traces to quantify lost revenue. The demo shows 177 failed orders worth $36,058.96. The same tables can be queried with BI tools, Grafana, or AI agents.

The exporter is still in development, not yet in the OpenTelemetry Collector Contrib, and uses at-least-once delivery, so duplicates are possible. It supports all five metric types, retains exemplars and histogram buckets, and allows pre-created schemas for least-privilege production setups. For teams already using Databend for analytics, this path puts observability data next to order tables, enabling incident analysis without exporting data to a separate system.

Takeaways
— The exporter writes OTLP data to three tables: otel_logs, otel_traces, and otel_metrics, using Databend's native Streaming Load API.
— It supports gauge, sum, histogram, exponential histogram, and summary metric types, preserving exemplars, bucket boundaries, min/max, and percentiles.
— Raw OTLP payloads are stored in VARIANT columns; common fields like time, service, and trace_id become separate columns.
— Timestamps are stored as UInt64 nanoseconds to keep full precision, unlike Databend's TIMESTAMP which uses microseconds.
— Delivery is at-least-once: retries can cause duplicate rows, and there is no deduplication mechanism.
— The exporter is in development and not yet in the official OpenTelemetry Collector Contrib; you must build it with ocb.
— To keep tables simple, the exporter does not automatically migrate existing tables and has no built-in TTL.
— In the demo, 2,100 checkouts produced 6,300 logs, 13,628 spans, and 693 metric points with zero Collector errors.
— The demo found 177 failed orders for a total of $36,058.96 lost revenue by joining logs and traces.
— The exporter's CI runs end-to-end tests against a real Databend instance, and unit tests reach 90.4% coverage with race detection.
Conclusions

Keeping complete OTLP objects as VARIANT preserves flexibility, but forces every query to navigate nested paths, which shifts the cost of understanding data into the analyst's SQL.

An at-least-once delivery model with no deduplication might be acceptable for logs and traces but could produce inflated counts or skewed metrics in long-running production use.

The decision to keep the exporter outside Contrib until community feedback suggests the team is cautious about API stability; production users should expect changes to configuration or behavior.

The demo's artificial failure injection (8% payment failures) makes it easy to validate the workflow, but real incidents typically involve noisier data and more ambiguous root causes.

Concepts & terms
OTLP
OpenTelemetry Protocol, the vendor-neutral wire protocol that transmits logs, traces, and metrics from instrumented applications to a Collector.
Streaming Load
Databend's high-throughput, HTTP-based ingestion endpoint that accepts NDJSON batches and performs bulk inserts without requiring SQL generation.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗