跪拜 Guibai
← All articles
Backend · Agent

RAG Sync as Reconciliation: Stable Chunk IDs, Read-Back Verification, Manifest-Last Commits

By 初学AI的小高 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Stale chunks that survive an update get retrieved and cited by the model, so prompt fixes can't rescue retrieval quality. This reconciliation approach gives incremental sync crash recovery and provable deletion, which are the parts of RAG that actually fail in production.

Summary

Tutorials stop at "load, chunk, vectorize, write," which only covers the first build. Ongoing knowledge bases face daily directory syncs, modified files, documents shrinking from 20 chunks to 12, source deletions, and crashes mid-sync. Without answers, old and new slices coexist and the model cites stale clauses, outdated product descriptions, or expired heritage records; prompt tweaks can't fix the root cause.

A working synchronizer defines an expected state per document, then reconciles the vector store to it. Three identifiers matter: `doc_id` carries business identity across content changes, `content_hash` (SHA-256 of normalized text) detects version changes, and `chunk_id` combines the document hash with a zero-padded index so the same slice keeps the same ID. Sync upserts desired records first, deletes the set difference (`old_ids - desired_ids`), re-reads to verify, then commits the SQLite manifest. Deterministic IDs and idempotent upserts make every crash point recoverable; delete follows the same verify-then-commit order.

The scheme is tested with an in-memory collection (4 unit tests) plus one ChromaDB persistence test, all passing in about a second, with caveats that test embeddings and local persistent clients don't validate production performance. Production edges include versioning the splitter and embedding config into an index-wide hash, batch upserts using the client's max batch size, and dual-version `active_version` switching for strict consistency.

Takeaways
— `collection.add` is a one-time write, not a sync; sync must compute the expected chunk set for each document and reconcile the vector store to it.
— `doc_id` carries business identity and survives content changes; `content_hash` (SHA-256 of normalized text) flags version changes; `chunk_id` = SHA-256 of `doc_id` plus a zero-padded chunk index gives stable per-slice IDs.
— Repeated upserts with stable IDs converge to the same state, so idempotency means "same result even if run twice," not "guaranteed to run once."
— Deleting by set difference (`old_ids - desired_ids`) removes exactly the stale tail when a document shrinks from 20 chunks to 12.
— The SQLite manifest commits only after read-back verification (`actual_ids == desired_ids`), so any "synced" status refers to a state that passed check.
— Crash recovery rests on three properties: deterministic IDs, idempotent upsert, and recomputing diffs from actual records on every run.
— Deletion order is delete vector records, read back to confirm none remain, then remove the manifest line; soft delete can use an `inactive` status plus metadata filtering.
— 4 in-memory unit tests plus 1 ChromaDB 1.5.1 persistence integration test pass in about 1 second; deterministic 3-dimensional test embeddings don't represent real model quality.
— Production extras: fold parser, splitter, and embedding config into an `index_version` hash, batch upserts using the client's max batch size, and dual-version `active_version` switching for strict consistency.
— Directory sync finds deletions by scanning sources, reading the manifest, and deleting `doc_id`s present in the manifest but absent from the scan; read failures should keep the old version rather than trigger deletion.
Conclusions

Manifest-last commit ordering converts SQLite from a source of truth into a commit record, which is what makes arbitrary crash points recoverable.

Read-back verification denies the manifest any authority over actual state; the vector store is the only state that matters, so any drift gets repaired on the next run.

The in-memory collection test approach isolates the sync state machine from embedding quality, making invariants testable without model downloads or real vector behavior.

Inserting content at the document head shifts all later chunk positions, so stable per-position IDs trade cheap updates for full re-vectorization on any leading change.

Concepts & terms
Reconciliation protocol
An approach that defines the expected state for a vector store, adjusts actual records to match it, and verifies the result, rather than appending writes.
doc_id
A stable business identity for a document that persists across content changes, as opposed to a random per-upload UUID.
content_hash
SHA-256 of normalized document text used to detect whether content changed between syncs.
chunk_id
A deterministic ID combining the document identity hash and a zero-padded chunk index, enabling idempotent upserts across sync runs.
Manifest
SQLite records of each document's last successfully committed content hash and chunk IDs, used for fast-skip, auditing, and crash recovery.
From the discussion

The accepted design treats the index as reconcilable state: set-difference cleanup replaces fake overwrite updates, and read failure preserves the old version instead of deleting. The deterministic chunk ID scheme (document hash plus sequence number) forces full re-vectorization when content shifts, a cost acknowledged and accepted for now, with content-aware chunking and chunk-level hash reuse postponed. A separate concern flags concurrent sync of the same document as needing protection against stale writes overwriting fresh state.

— Set-difference deletion (stale_ids = old_ids - desired_ids) is cleaner than overwriting the first N chunks, because the latter leaves stale tail chunks in the index that can still be recalled.
— Treating a read failure as a deletion request is dangerously wrong; preserving the old version and alerting is the safer behavior.
— Using document hash plus sequence number as chunk ID makes every front insertion shift all later chunks and forces full re-vectorization, raising the question of whether that cost should be accepted or mitigated with content-aware chunking and chunk-level hash reuse.
— The author’s position is to accept the re-vectorization cost for now to guarantee idempotency, old-chunk cleanup, and interruption recovery, deferring chunk-level hash reuse and vector caching until data volume grows.
— Concurrent sync of the same document must guard against a stale write overwriting a newer write.
Featured comments
组合搭配函数

“stale_ids = old_ids - desired_ids” — I agree with that line. When a document shrinks from 20 chunks to 12, the set difference directly computes the 8 IDs to delete, which is much cleaner than “overwriting the first 12 counts as an update” — the latter is a fake update, the old tail stays in the store waiting to be recalled. I also agree with the choice to keep the old version and alert on read failure; treating “didn’t read” as “user requested deletion” is the most dangerous misjudgment in a sync engine. One practical question: with chunk_id using document hash plus chunk sequence number, inserting a block at the beginning shifts all subsequent chunk positions, so all must be re-vectorized. In projects like HeritageMind, do you just accept that cost, or have you already switched to content-aware chunking with chunk-level hash reuse?

初学AI的小高

Thanks for the follow-up; this issue is real. The current approach prioritizes idempotent updates, clean removal of old chunks, and resumability after interruption, so it accepts the cost of re-vectorization after chunk positions change. HeritageMind hasn’t implemented chunk-level hash reuse yet; at this stage the data volume is small, so consistency comes first. Once the data grows, stable segmentation and a vector cache can be considered later.

卷土的土

Concurrent sync of the same document needs to prevent old writes from overwriting newer ones.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗