RAG Sync as Reconciliation: Stable Chunk IDs, Read-Back Verification, Manifest-Last Commits
Stale chunks that survive an update get retrieved and cited by the model, so prompt fixes can't rescue retrieval quality. This reconciliation approach gives incremental sync crash recovery and provable deletion, which are the parts of RAG that actually fail in production.
Tutorials stop at "load, chunk, vectorize, write," which only covers the first build. Ongoing knowledge bases face daily directory syncs, modified files, documents shrinking from 20 chunks to 12, source deletions, and crashes mid-sync. Without answers, old and new slices coexist and the model cites stale clauses, outdated product descriptions, or expired heritage records; prompt tweaks can't fix the root cause.
A working synchronizer defines an expected state per document, then reconciles the vector store to it. Three identifiers matter: `doc_id` carries business identity across content changes, `content_hash` (SHA-256 of normalized text) detects version changes, and `chunk_id` combines the document hash with a zero-padded index so the same slice keeps the same ID. Sync upserts desired records first, deletes the set difference (`old_ids - desired_ids`), re-reads to verify, then commits the SQLite manifest. Deterministic IDs and idempotent upserts make every crash point recoverable; delete follows the same verify-then-commit order.
The scheme is tested with an in-memory collection (4 unit tests) plus one ChromaDB persistence test, all passing in about a second, with caveats that test embeddings and local persistent clients don't validate production performance. Production edges include versioning the splitter and embedding config into an index-wide hash, batch upserts using the client's max batch size, and dual-version `active_version` switching for strict consistency.
Manifest-last commit ordering converts SQLite from a source of truth into a commit record, which is what makes arbitrary crash points recoverable.
Read-back verification denies the manifest any authority over actual state; the vector store is the only state that matters, so any drift gets repaired on the next run.
The in-memory collection test approach isolates the sync state machine from embedding quality, making invariants testable without model downloads or real vector behavior.
Inserting content at the document head shifts all later chunk positions, so stable per-position IDs trade cheap updates for full re-vectorization on any leading change.
The accepted design treats the index as reconcilable state: set-difference cleanup replaces fake overwrite updates, and read failure preserves the old version instead of deleting. The deterministic chunk ID scheme (document hash plus sequence number) forces full re-vectorization when content shifts, a cost acknowledged and accepted for now, with content-aware chunking and chunk-level hash reuse postponed. A separate concern flags concurrent sync of the same document as needing protection against stale writes overwriting fresh state.
“stale_ids = old_ids - desired_ids” — I agree with that line. When a document shrinks from 20 chunks to 12, the set difference directly computes the 8 IDs to delete, which is much cleaner than “overwriting the first 12 counts as an update” — the latter is a fake update, the old tail stays in the store waiting to be recalled. I also agree with the choice to keep the old version and alert on read failure; treating “didn’t read” as “user requested deletion” is the most dangerous misjudgment in a sync engine. One practical question: with chunk_id using document hash plus chunk sequence number, inserting a block at the beginning shifts all subsequent chunk positions, so all must be re-vectorized. In projects like HeritageMind, do you just accept that cost, or have you already switched to content-aware chunking with chunk-level hash reuse?
Thanks for the follow-up; this issue is real. The current approach prioritizes idempotent updates, clean removal of old chunks, and resumability after interruption, so it accepts the cost of re-vectorization after chunk positions change. HeritageMind hasn’t implemented chunk-level hash reuse yet; at this stage the data volume is small, so consistency comes first. Once the data grows, stable segmentation and a vector cache can be considered later.
Concurrent sync of the same document needs to prevent old writes from overwriting newer ones.