跪拜 Guibai
← All articles
Python

Self-Hosted Flight Fare Monitor Pulls Real Data From 6 Platforms and 10 Channels

By 蜗牛875 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

The project is a working template for aggregating many scraped sources behind one schema while being honest about reliability gaps. The same channel-matrix and three-way alignment pattern applies to any monitoring task where sources disagree, sessions expire, and you need forensic evidence of what the scraper actually saw.

Summary

Self-hosted fare monitoring has a new reference implementation: ticket-monitoring aggregates 6 Chinese booking platforms (Ctrip, Qunar, Fliggy, Tongcheng, Tuniu, Zhixing) through 5 browser and 5 App channels, normalizing flight price, number, times, stops, duration, terminal, cabin, aircraft, punctuality, and baggage into a unified schema stored in SQLite. A console page shows side-by-side comparisons of the same flight across sources, a price calendar and K-line view, and alert channels for DingTalk, Windows notifications, ntfy, Bark, and Alibaba Cloud calls or SMS.

The project grew out of a concrete annoyance: comparing a direct flight with a connecting one on a Shanghai-to-Tokyo route showed ¥975 direct vs ¥1414 connecting, but the direct flight had no checked baggage while the connecting one included 23kg, so the headline difference was misleading. Worse, calendar prices, detail prices, and per-source data disagreed. The tool keeps those differences visible and excludes sparse calendar rows from alert logic.

Technically, each browser channel picks its most stable path (H5, SSR, or HTTP), App channels run on a MuMu Android emulator via adb and uiautomator2/OCR for canvas pages, and login states are snapshotted and restored. A 10-by-7 channel capability matrix labels each cell as supported, unsupported, or unverified, and unverified is never treated as supported. Deployment runs from a prebuilt binary, source, or Docker with built-in Chromium and Chinese fonts.

Takeaways
— Supports 6 platforms and 10 channels: 5 browser, 5 App; App channels default off and require MuMu (Android emulator, ~15GB, Windows, VT).
— A live probe on Shanghai→Tokyo pulled 282 flight details: 163 from Ctrip, 119 from Tuniu; 3 sources returned nothing and were left blank.
— Calendar prices are never used for alert thresholds; only full flight rows with flight number and times can trigger alerts.
— The tool does not bypass CAPTCHAs or auto-book; it stops and waits for manual login intervention when sessions expire.
— Docker deployment persists config, price database, browser session archives, and logs in local volumes across container rebuilds.
— The channel capability matrix uses three states: supported, unsupported, unverified; unverified is treated as not supported.
— Multi-leg booking verification currently only covers a 2-segment Ctrip flight; 3-segment and other platforms are not wired up.
Conclusions

Keeping raw and parsed rows with three-way alignment turns the scraper into a forensic tool; when a source page changes, you can see exactly where parsing diverged from the mirror.

The honest three-state capability matrix (supported/unsupported/unverified) is a rare counterweight to the usual claim-everything scraper readme; it sets expectations that unknown endpoints are not confirmed working.

Treating calendar prices and detailed flight rows as separate data classes prevents false alerts from sparse data, a design choice many aggregators skip.

Concepts & terms
MuMu
An Android emulator used for running flight booking apps locally; enabling App channels needs it, roughly 15GB on Windows with VT, and it stays dormant when App channels are off.
Three-way alignment
A validation method that compares parsed data, a rendered mirror of the original page, and the raw response path; any mismatch surfaces immediately for debugging.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗