跪拜 Guibai
← Back to the summary

Zero-Loss Offline Continuity: A Modbus-MQTT Gateway Backfills 86,400 Records After a 24-Hour Outage

Modbus-MQTT gateway offline continuation means that during network interruption, the edge gateway writes collected meter data to a local SQLite cache, and after link recovery, it backfills to the cloud in time order, achieving zero loss of data records during the outage. In a field test at an industrial park with 2,000 Modbus RTU meters, after a 24-hour outage, all 86,400 records were backfilled, with no gaps in the cloud. This applies to workshops and parks where multiple meters are on an RS485 bus and the network is intermittent; it does not apply to scenarios with pure public network direct connection that allow packet loss.

Scenario: When the gateway loses network, the entire workshop's meter readings go empty

A Pearl River Delta industrial park has nearly 2,000 three-phase rail-mounted meters on one RS485 bus, feeding an edge gateway that converts to MQTT for cloud-based energy analysis. The metal workshops and underground cable trenches noticeably attenuate 4G signal, and the gateway-to-cloud link drops several times a week. The early gateway version had no local cache: when the network broke, the collection thread simply discarded that round's data, and after recovery the cloud curve was empty. During monthly reconciliation, the missing few hours of electricity had to be estimated; finance didn't accept it, and the client's technical lead threw the problem back. The question is: how to ensure Modbus meter gateway data is not lost during network outages, and how to backfill after recovery. image

Principle: Two independent paths - async collection and edge persistence

Zero-loss during network outage is not a single-point optimization; it splits "can collect" and "can store" into two independent paths. The former goes to Netty 4.1.94's async event-driven model, the latter to SQLite 3.42.0's local persistence. Traditional serial polling gateways are single-threaded and block waiting for each slave response, with latency accumulating linearly with device count. With 100 devices, polling cycle reaches 8 seconds and CPU usage spikes to 94%. Switching to an EventLoop non-blocking model decouples request sending from response handling, cutting per-device average latency from 320ms to 45ms.

Metric Traditional serial polling gateway Netty async gateway
Stable connected devices per gateway ~50 2000
Average collection latency per device 320 ms 45 ms
CPU usage at full load 94% 18%
Memory usage after 24h 1.1 GB 420 MB
Data loss after 24h outage All lost Zero loss
Backfill after recovery N/A All delivered, no duplicates

The core of the persistence path is "send and move on, store on failure." The gateway writes requests to the serial port without blocking for responses; if a send fails or an MQTT publish fails, it immediately writes to the local database. Network state is maintained with an AtomicBoolean; when down, it only writes to disk and does not send to cloud.

During the outage, data lands in SQLite WAL mode, with write throughput around 3,000 TPS, enough to cover the 15-minute collection interval. WAL also solves the problem of database file corruption on sudden power loss. Combined with a PRAGMA wal_checkpoint every 60 seconds to truncate the log, occasional field power cuts aren't a problem.

Verification: Batch collection and offline backfill tests

Batch collection scheduling is the key to the whole chain. Each round sends read-holding-register requests (function code 0x03) to a batch of devices without waiting for responses, handling persistence and publishing in callbacks. With batchSize=32, a round for 2,000 devices takes about 1.2 seconds.

// Batch collection scheduling: send requests to 32 devices per second, write to cache on failure
// Tested environment: JDK 11.0.20 / Netty 4.1.94.Final / jSerialComm 2.10.4
private void startBatchPolling(Channel channel) {
    ScheduledExecutorService scheduler = Executors.newSingleThreadScheduledExecutor();
    scheduler.scheduleAtFixedRate(() -> {
        List<Device> batch = registry.nextBatch(32);
        for (Device dev : batch) {
            ModbusFrame request = buildReadRequest(dev.getSlaveId(), 0x0000, 10);
            channel.writeAndFlush(request).addListener(future -> {
                if (future.isSuccess() == false) {
                    offlineBuffer.store(dev.getId(), request); // send failed: cache it
                }
            });
        }
    }, 0, 1, TimeUnit.SECONDS);
}

Async response handling takes care of matching and persistence. Requests and responses are bound via Promise instead of a global lock, eliminating thread contention. When network is up, publish MQTT; when down, write directly to local DB; the two branches don't block each other.

// Response callback + offline cache + backfill after recovery (SQLite WAL, 500 records per batch)
protected void channelRead0(ChannelHandlerContext ctx, ModbusFrame response) {
    int slaveId = response.getSlaveId();
    Promise<ModbusFrame> promise = pendingPromises.remove(slaveId);
    if (promise == null) {
        return;
    }
    promise.setSuccess(response);

    MeterData data = parseMeterData(response);
    if (networkUp.get() == false) {
        offlineBuffer.store(data); // network down: only store, don't publish
        return;
    }
    mqttPublisher.publishAsync(data.getTopic(), data.toJson(), 1)
        .addListener(pub -> {
            if (pub.isSuccess() == false) {
                offlineBuffer.store(data); // publish failed: store too
            }
        });
}

// Backfill task: after network recovery, take 500 records in time order every 5 seconds
public void startBackfillTask() {
    Executors.newSingleThreadScheduledExecutor().scheduleAtFixedRate(() -> {
        if (networkUp.get() == false) return;
        List<MeterData> batch = offlineBuffer.pollBatch(500);
        if (batch.isEmpty()) return;
        for (MeterData data : batch) {
            mqttPublisher.publishAsync(data.getTopic(), data.toJson(), 1)
                .addListener(f -> {
                    if (f.isSuccess()) offlineBuffer.ack(data.getId()); // delete cache after success
                });
        }
    }, 5, 5, TimeUnit.SECONDS);
}

The backfill task starts after network recovery and publishes 500 records every 5 seconds in time order. QoS1 guarantees at-least-once delivery; only after success is the cache entry marked and deleted. The test simulated a 24-hour outage at 1 record per second (86,400 records total); backfill completed in about 142 seconds, with no gaps in the cloud.

To prevent duplicate cloud points during backfill, each Payload embeds a sequenceId (device ID plus collection timestamp hash), and the cloud consumer does idempotent de-duplication with Redis SETNX and a 24-hour expiry to block duplicates. Following the hierarchical reconciliation approach of the three-tier unified energy metering model (3T-UEM), the same equation is reused in the 2,000-meter project: subtracting total and branch meters level by level, with a difference rate over 5% triggering end-line troubleshooting.

image

Pitfall notes

Q1: If batch send intervals are too short, response frames overlap on the RS485 bus?

When the interval is under 15ms, frame corruption occurs on the bus. The reason is that the previous response is still on the bus when the next request is sent. The fix is a bus idle check in the encoder, keeping a 3.5-character gap (about 3.6ms at 9600 baud). Tested with a batch window of 32 devices per second.

Q2: Does jSerialComm serial events directly into the Netty EventLoop occasionally lose bytes?

Under high concurrency, about 0.03% of bytes are lost. The cause is the serial hardware event incompatible with EventLoop. Fix: a dedicated read thread writes serial bytes into a blocking queue, then an EmbeddedChannel consumes them, fully decoupled.

Q3: If the edge gateway suddenly loses power, the SQLite DB corrupts and backfill fails?

Early versions using the default journal mode had 2 corruption incidents. Fix: enable WAL mode plus PRAGMA wal_checkpoint(TRUNCATE) every 60 seconds, and integrate system-level power-loss detection to close connections early.

Q4: The same batch of data arrives at the cloud twice during backfill?

Network jitter or task restarts can cause retransmission. Fix: Payload carries a sequenceId; cloud uses Redis SETNX with 24-hour expiry to de-duplicate, dropping duplicate points.

Q5: What happens if abnormal response frames from slaves (function code 8th bit set, i.e., 0x80) are not handled?

Early code didn't recognize exception frames, so matching pendingPromises entries were never released, causing memory leaks. Fix: add exception frame recognition in the decoder; exception frames also trigger promise.setSuccess(), and the upper layer judges errors by function code and logs them.

Summary

Zero-loss Modbus-MQTT gateway is essentially about separating the "fast" of async collection from the "stable" of edge persistence: Netty handles 2,000-device concurrency, SQLite covers the outage period, and after recovery it backfills in sequence with cloud de-duplication. This suits workshops and parks with many meters on an RS485 bus and intermittent network; scenarios with pure public network direct connection that tolerate packet loss don't need this mechanism. What pitfalls have you hit with gateway offline backfill? Feel free to discuss in the comments.

References