跪拜 Guibai
← Back to the summary

RAG Sync as Reconciliation: Stable Chunk IDs, Read-Back Verification, Manifest-Last Commits

RAG 知识库更新实战:增量索引、版本控制与可验证删除

Many RAG tutorials stop at "load document—chunk—vectorize—write to vector store." That flow works for the first time you build a knowledge base, but it doesn't answer the more common questions after the knowledge base goes live:

If these questions have no clear answers, retrieval results may contain both old and new versions at once. When a model ends up citing old terms, old product descriptions, or expired intangible-heritage materials, tweaking the prompt alone doesn't fix the root cause.

This article implements a small but complete synchronizer: Python computes content fingerprints and the sync plan, SQLite stores the sync manifest, ChromaDB stores chunks. The focus is not a specific vector store API, but a reconciliation protocol that can be ported to other storage.

1. add is not knowledge base sync

The most direct ingestion code is usually:

collection.add(
    ids=chunk_ids,
    documents=chunks,
    metadatas=metadatas,
)

It expresses a one-time write action, not "make the vector store eventually consistent with source files."

Sync must first define the expected state: for document docs/a.md, which chunk IDs should currently exist, which version each chunk belongs to, and which IDs from older versions should no longer exist. Then adjust the vector store's actual state to this expected state.

Think of it as:

Current state of source file
      ↓
Compute expected chunk set
      ↓
Read actual set in vector store
      ↓
upsert expected records + delete surplus records
      ↓
Re-read and verify
      ↓
Finally commit sync manifest

This is fundamentally different from "run the ingestion script again."

2. First distinguish three easily-confused identifiers

A maintainable index needs at least three kinds of identifiers.

2.1 doc_id: which document is this

doc_id represents business identity, not a random UUID generated per upload. For example:

laws/civil-code.md
heritage/suzhou-embroidery.md
course/python/week-03.md

When the same document's content changes, doc_id remains unchanged. Systems where paths may move can use a database primary key or a stable key in object storage instead.

Note: craft_name, display file name, and database primary key are not necessarily the same concept. When a project already has a stable primary key, use it directly. Don't improvise with names as identity.

2.2 content_hash: what version is this document now

Compute SHA-256 over normalized text:

import hashlib


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()

Same hash means it can enter the fast-skip path; different hash means at least reconciliation is needed. It is not a version number in the legal sense, but it works well for detecting content change.

2.3 chunk_id: which record is this in the vector store

A stable ID is generated from "document identity hash + chunk sequence number":

def chunk_id(doc_id: str, index: int) -> str:
    prefix = hashlib.sha256(doc_id.encode("utf-8")).hexdigest()
    return f"{prefix}:{index:06d}"

When the document syncs a second time, the chunk at the same position keeps the same ID, and upsert overwrites its text, vector, and metadata; if the chunk count decreases, the extra old IDs are explicitly deleted.

This strategy is simple and explainable, but it has a cost: after content is inserted at the beginning of a document, subsequent chunk positions may shift entirely and need re-vectorization. Large-scale systems can further do content-aware chunking and chunk-level hash reuse; the first version need not solve every optimization at once.

In Chroma's data model, each record itself contains a unique ID, vector, optional metadata, and document; the specific semantics of document ID are decided by the application. The official docs also note that Chroma cannot query by part of a composite ID, so this article additionally puts doc_id in metadata and uses it to query all records of a document. Chroma data model | Document IDs

3. SQLite manifest is not a second vector store

The sync manifest stores only the last successfully committed state per document:

CREATE TABLE documents (
    doc_id TEXT PRIMARY KEY,
    content_hash TEXT NOT NULL,
    chunk_ids_json TEXT NOT NULL,
    synced_at TEXT NOT NULL
);

It has three roles:

  1. Fast-skip when content is unchanged;
  2. Record which chunks a successful sync included;
  3. Provide time and version info for debugging and auditing.

But you can't trust only the manifest. The process may crash after writing Chroma but before committing SQLite, or someone may modify the vector store bypassing the synchronizer. So the final judgment for every update and delete must re-read the actual ID set in Chroma for that doc_id.

In other words:

Manifest = acceleration and audit
Actual Chroma records = external state that must be reconciled
Source files = source of expected state

4. Connecting Chroma to the synchronizer

Install and create a local persistent collection:

python -m pip install chromadb
import chromadb


client = chromadb.PersistentClient(path="./chroma_data")
collection = client.get_or_create_collection("knowledge_base")

Chroma's official docs position PersistentClient as a local development and testing method; production environments are better served by server-side deployment with HttpClient or AsyncHttpClient. Don't treat the same in-process architecture as a production solution just because the example persists locally. Chroma Python Client

Next, define the minimal interface the synchronizer needs:

from typing import Any, Protocol


class CollectionLike(Protocol):
    def upsert(
        self,
        *,
        ids: list[str],
        documents: list[str],
        metadatas: list[dict[str, Any]],
    ) -> Any: ...

    def get(
        self,
        *,
        where: dict[str, Any],
        include: list[str],
    ) -> dict[str, Any]: ...

    def delete(self, *, ids: list[str]) -> Any: ...

Using a protocol instead of calling Chroma directly everywhere in business logic has two benefits: the sync algorithm can be unit-tested, and swapping to a different vector store later is easier by replacing the adapter layer.

5. Core sync algorithm: write new state first, then clean old state

To highlight the sync flow, the example uses fixed-character-count chunking:

def split_text(text: str, chunk_size: int = 500) -> list[str]:
    if chunk_size <= 0:
        raise ValueError("chunk_size must be positive")

    normalized = text.replace("\r\n", "\n").strip()
    if not normalized:
        return []

    return [
        normalized[i:i + chunk_size]
        for i in range(0, len(normalized), chunk_size)
    ]

Production projects should replace this with an existing heading-, paragraph-, or semantic-based splitter. The chunking algorithm itself should also be versioned, e.g. splitter_version=v2; when the algorithm changes, re-indexing should be triggered even if the source text is unchanged.

The key sync code:

from dataclasses import dataclass


@dataclass(frozen=True)
class SourceDocument:
    doc_id: str
    text: str


class DocumentSynchronizer:
    def __init__(self, collection, manifest):
        self.collection = collection
        self.manifest = manifest

    def _ids_in_store(self, doc_id: str) -> set[str]:
        result = self.collection.get(
            where={"doc_id": doc_id},
            include=[],
        )
        return set(result.get("ids", []))

    def sync(self, document: SourceDocument, chunk_size: int = 500):
        content_hash = sha256_text(document.text)
        previous = self.manifest.get(document.doc_id)

        # Fast path also checks actual IDs once, to avoid drift between manifest and vector store
        if previous and previous["content_hash"] == content_hash:
            expected = set(previous["chunk_ids"])
            if self._ids_in_store(document.doc_id) == expected:
                return {"action": "unchanged"}

        chunks = split_text(document.text, chunk_size)
        desired_ids = [
            chunk_id(document.doc_id, i)
            for i in range(len(chunks))
        ]
        desired_set = set(desired_ids)
        old_ids = self._ids_in_store(document.doc_id)

        if desired_ids:
            self.collection.upsert(
                ids=desired_ids,
                documents=chunks,
                metadatas=[
                    {
                        "doc_id": document.doc_id,
                        "content_hash": content_hash,
                        "chunk_index": i,
                    }
                    for i in range(len(chunks))
                ],
            )

        stale_ids = sorted(old_ids - desired_set)
        if stale_ids:
            self.collection.delete(ids=stale_ids)

        actual_ids = self._ids_in_store(document.doc_id)
        if actual_ids != desired_set:
            raise RuntimeError(
                "vector-store verification failed: "
                f"expected={desired_set}, actual={actual_ids}"
            )

        # Only commit manifest after external state passes verification
        self.manifest.commit_document(
            document.doc_id,
            content_hash,
            desired_ids,
        )
        return {
            "action": "updated" if previous else "created",
            "upserted": len(desired_ids),
            "deleted": len(stale_ids),
        }

Chroma's official API exposes upsert, get, delete as collection-level operations; where filters records by metadata. Chroma Core API | Metadata Filters

This algorithm has four important details.

5.1 Use upsert so repeated runs converge to the same state

Repeatedly writing the same batch of stable IDs doesn't create a second set of records; it overwrites the corresponding IDs. Idempotency is not "the script guarantees it only runs once," but "even if it runs repeatedly, the result stays identical."

5.2 Write expected records first, then delete surplus records

When updating a document, writing the current version first and then cleaning the old tail shortens the window where the document is entirely unretrievable. But there is no cross-system transaction between Chroma and SQLite, so this still is not a strictly atomic update.

5.3 Delete by set difference, not by guessing the old chunk count

stale_ids = old_ids - desired_ids

When a document shrinks from 20 chunks to 12, the set difference precisely yields the 8 IDs to delete. Don't use "overwriting the first 12 counts as an update."

5.4 Commit the manifest last

Only when actual_ids == desired_ids does SQLite record the new version. The "synced" status the next run sees corresponds to at least a state that passed set verification the previous time.

6. Why retry works after a mid-way crash

Suppose the update process is interrupted at different positions:

Interruption point Possible state How next sync handles it
Before upsert Still old version Re-run all steps
After upsert, before delete New records written, old tail still present Upsert same IDs again, recompute set difference
After delete, before manifest commit Vector store already in new state, manifest still in old state Re-reconcile, overwrite same IDs, then commit manifest
After manifest commit All three consistent Fast-verify and skip

Recovery capability comes from three conditions:

This is not a full distributed transaction. If two sync processes modify the same doc_id concurrently, they can still overwrite each other. Production environments should add a mutex, task lease, or single-writer queue per document or knowledge base.

7. Deletion must also be a verifiable operation

When a source file disappears, you shouldn't just delete a row from the SQLite manifest:

def delete_document(self, doc_id: str):
    old_ids = self._ids_in_store(doc_id)

    if old_ids:
        self.collection.delete(ids=sorted(old_ids))

    remaining = self._ids_in_store(doc_id)
    if remaining:
        raise RuntimeError(f"delete verification failed: {remaining}")

    self.manifest.remove(doc_id)
    return {
        "action": "deleted",
        "deleted": len(old_ids),
    }

The order is still: delete vector records, read and verify, delete manifest last. Chroma's docs state that a collection's delete permanently removes the record's vector, document, and metadata, so the target scope must be limited to exact IDs before executing, and source data or backups should be kept. Chroma Collections

If the system needs soft delete, first set the status to inactive in the business database and filter by metadata at retrieval time; physically delete after background confirmation. This makes rollback easier, but every query must carry the status filter.

8. Four tests guarding the sync invariants

I did not load a real embedding model in tests; instead I implemented an in-memory collection. It follows the same upsert/get/delete interface and is used to verify the sync state machine:

class InMemoryCollection:
    def __init__(self):
        self.rows = {}

    def upsert(self, *, ids, documents, metadatas):
        for row_id, document, metadata in zip(
            ids, documents, metadatas
        ):
            self.rows[row_id] = {
                "document": document,
                "metadata": metadata,
            }

    def get(self, *, where, include):
        ids = [
            row_id
            for row_id, row in self.rows.items()
            if all(
                row["metadata"].get(key) == value
                for key, value in where.items()
            )
        ]
        return {"ids": sorted(ids)}

    def delete(self, *, ids):
        for row_id in ids:
            self.rows.pop(row_id, None)

4 core unit test methods cover 5 behaviors:

  1. First sync correctly creates chunks;
  2. Re-syncing the same document adds no extra records;
  3. When a document shortens, surplus tail chunks get deleted;
  4. After manually deleting one vector record, re-sync repairs the missing item;
  5. After document deletion, neither the in-memory vector store nor the manifest has residue.

Additionally, I added 1 persistence integration test under Python 3.13.7 and ChromaDB 1.5.1, using deterministic 3-dimensional test embeddings to run first creation, content shortening, close-and-reconnect, and exact deletion in sequence. All 4 unit tests and 1 integration test pass; combined runtime is about 1 second.

Test embeddings exist only to avoid downloading models and to reproduce stably; they don't represent BGE-M3's vector quality. The above results also don't prove that server-side deployment, concurrent writes, or large-scale data performance has been validated.

Real projects should add two more kinds of integration tests:

The second kind is especially important. A correct ID set only proves record counts and ownership are correct; it doesn't prove retrieval quality is correct.

9. Detecting deletion when batch-syncing a directory

After single-document sync, directory sync can be organized like this:

Scan source directory to get current_doc_ids
Read manifest to get known_doc_ids

for doc in current source files:
    sync(doc)

for doc_id in known_doc_ids - current_doc_ids:
    delete_document(doc_id)

When scanning, be explicit about:

I prefer: on read failure, keep the old version and alert. Don't mistake "this run failed to read" for "the user asked to delete." Only when the source object is clearly confirmed absent does deletion proceed.

10. Edges to cover before going to production

The code here deliberately implements only the core protocol. When integrated into a real system, at least the following should be added.

10.1 Include processing configuration in versioning

When document content is unchanged but the splitter, cleaner, or embedding model changes, old indexes need updating too. Compute:

index_version = hash(
    content_hash,
    parser_version,
    splitter_version,
    embedding_model,
    embedding_dimension
)

BGE-M3 with the same model path doesn't necessarily mean the same processing configuration; normalization, max length, and pooling method should all be written into an auditable config.

10.2 Batch writes and record progress

Large documents should not submit all chunks at once. Read the vector store's maximum allowed batch size, upsert in batches, and retry failed batches. The Chroma Python Client exposes a method to get the max batch size; production code should not permanently hardcode the batch value. Chroma Python Client

10.3 Define a consistency strategy between sync and query

During an update, queries may see partial new state. Strict scenarios can use "dual version + active_version" switching: fully write the new version first, atomically point the business-side active version from v1 to v2 after verification, then asynchronously clean up v1.

The cost is temporarily doubling storage, and all queries must carry a version filter. For legal texts or auditable knowledge bases, that cost is often worth it; an ordinary personal knowledge base can use this article's single-version reconciliation.

10.4 Don't log only successes

Each sync round should record at least:

run_id
doc_id
old_hash / new_hash
expected_chunks / actual_chunks
upserted / deleted
started_at / finished_at
error

With structured logs, you can answer "when did this old content enter the vector store, which task should have deleted it, and did the deletion pass verification."

10.5 Run periodic full reconciliation

Incremental sync keeps each operation as correct as possible; full reconciliation is responsible for finding long-term drift. Periodically check:

When anomalies are found, output a report first, then decide on auto-repair. Don't give the "reconciliation script" unrestricted delete permission by default.

11. How to split responsibilities in a RAG project

Split the indexing pipeline into five layers:

SourceLoader        Reads files, databases, or object storage
Parser              Extracts and normalizes body text
Chunker             Generates a deterministic chunk sequence
VectorStoreAdapter  Wraps Chroma read/write/delete
IndexSynchronizer   Computes diffs, verifies, commits manifest

This way debugging can first locate the layer:

For HeritageMind, use stable intangible-heritage project primary keys as doc_id, don't mix in craft names; for LexRAG, put laws, chapters, articles, and data versions in metadata; for EduTutor, write course owner and access scope into filter fields. These are migration suggestions only; they don't mean those projects have completed the above changes.

Conclusion

The hard part of maintaining a RAG knowledge base is not turning text into vectors the first time; it's keeping source data, vector records, and version manifest consistent over the long term.

A minimal reliable scheme can be summarized as:

Stable doc_id confirms identity
Content hash judges version
Deterministic chunk_id supports idempotent writes
Set difference cleans residue
Read-back verification confirms results
Manifest committed last
Periodic full reconciliation finds drift

With this in place, "re-running the index script" stops being a gamble and becomes a sync operation that can be explained, verified, and recovered.

References

Comments

Top 2 of 3 from juejin.cn, machine-translated. The original thread is authoritative.

组合搭配函数

“stale_ids = old_ids - desired_ids” — I agree with that line. When a document shrinks from 20 chunks to 12, the set difference directly computes the 8 IDs to delete, which is much cleaner than “overwriting the first 12 counts as an update” — the latter is a fake update, the old tail stays in the store waiting to be recalled. I also agree with the choice to keep the old version and alert on read failure; treating “didn’t read” as “user requested deletion” is the most dangerous misjudgment in a sync engine. One practical question: with chunk_id using document hash plus chunk sequence number, inserting a block at the beginning shifts all subsequent chunk positions, so all must be re-vectorized. In projects like HeritageMind, do you just accept that cost, or have you already switched to content-aware chunking with chunk-level hash reuse?

初学AI的小高

Thanks for the follow-up; this issue is real. The current approach prioritizes idempotent updates, clean removal of old chunks, and resumability after interruption, so it accepts the cost of re-vectorization after chunk positions change. HeritageMind hasn’t implemented chunk-level hash reuse yet; at this stage the data volume is small, so consistency comes first. Once the data grows, stable segmentation and a vector cache can be considered later.

卷土的土

Concurrent sync of the same document needs to prevent old writes from overwriting newer ones.