Flink · Big Data

Douyin Group Rebuilds Its Real-Time Data Warehouse on Apache Paimon, Slashing Flink State from Terabytes to Gigabytes

By Flink_China · Jul 31, 2025

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams running real-time Flink pipelines at scale routinely hit a wall with state size and external KV store fragility. This production migration shows that a Paimon lakehouse can collapse a multi-component architecture into a simpler one, turning a TB-level state problem into a manageable GB-level one without sacrificing query concurrency.

Summary

A team at Douyin Group rebuilt its real-time data warehouse on Apache Paimon to fix cascading stability and cost problems in its legacy Flink-plus-MQ architecture. The old design forced engineers to manually construct retraction messages in Flink to handle changelogs, causing Keyed State to balloon into the terabytes and making jobs unstable. External KV lookups for dimension joins at 8 million RPS peak traffic also created a heavy dependency on a fragile caching layer.

The Paimon lakehouse converges dimension tables, changelog propagation, and batch queries inside the storage layer. DWD tables are defined as Paimon primary-key tables with `changelog-producer = 'lookup'`, so downstream DWS jobs consume changelogs directly without deduplication or manual retraction logic. Dimension joins happen locally against Paimon's LSM-Tree-based storage, removing the external KV bottleneck and improving data freshness from 50 minutes to under two minutes.

In a long-period aggregation scenario for live-stream reporting metrics, the new design offloaded state from Flink into Paimon, making Flink task state negligible while keeping query concurrency at millions via an external KV store. In a high-throughput short-video metrics pipeline, state size dropped from terabytes to hundreds of gigabytes, and the team eliminated all external dimension-table requests.

Takeaways

— Replacing an MQ layer with Paimon lets downstream Flink jobs consume changelogs directly, eliminating the need to manually build retraction messages and the associated Keyed State bloat.

— Paimon dimension tables stored on HDFS with SSD backing cut resource costs significantly compared to external KV stores, while supporting high-concurrency point lookups via an LSM-Tree structure.

— Setting `changelog-producer = 'lookup'` on a Paimon primary-key table allows DWS aggregation jobs to read changelogs natively, reducing state from terabytes to hundreds of gigabytes in a high-throughput pipeline.

— Paimon's built-in batch read capability removed the dependency on Hive for development and testing, dropping data freshness for tests to minute-level and accelerating the test cycle.

— Dimension-table data freshness improved from 50 minutes (capped by Flink's `lookup.cache.ttl`) to under two minutes by setting the Paimon dimension table's Flink Checkpoint interval to one minute.

— Even after migrating to Paimon, the team still routes final aggregated metrics to an external KV store to serve queries at millions of QPS, keeping the serving layer decoupled from the lakehouse.

Conclusions

The core unlock is that Paimon absorbs the complexity of changelog handling and state management that Flink-plus-MQ architectures force onto application code. This isn't just a storage swap; it removes entire classes of Flink operator logic.

External KV stores become a liability at 8 million RPS not because they can't serve data, but because the cache-penetration tail forces a trade-off between freshness and stability. Paimon sidesteps this by co-locating dimension data with compute, turning a network-call problem into a local-read problem.

The decision to keep an external KV store for final metric serving, even after adopting Paimon, signals that lakehouse query performance still hasn't closed the gap with purpose-built KV stores for high-QPS point reads. The lakehouse handles the heavy lifting upstream; the serving layer stays specialized.

Concepts & terms

Changelog Producer (Lookup mode)

A Paimon table setting that materializes changelog streams for downstream consumers. When set to 'lookup', Paimon generates changelogs by looking up the previous row value on compaction, enabling downstream Flink jobs to consume insert, update, and delete events directly without manual retraction logic.

Retract messages in Flink

In streaming SQL, a retraction is a pair of messages — a delete of the old row followed by an insert of the new row — used to update mutable results when the upstream source is a changelog. When the message queue cannot propagate changelogs natively, developers must construct these retractions manually using functions like `Last_Value`, which forces Flink to hold large amounts of state.

Lookup Join

A join where each incoming record from the main stream triggers a point lookup against an external table (the dimension table) to enrich the record. In Flink, this is typically backed by an external KV store; with Paimon, the lookup hits local LSM-Tree files, removing the network call.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗