LLM · Next.js · Agent

Vectorizing Long-Term Chat Memory So It Survives a Paraphrase

By 倾颜 · Jul 10, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Most chat apps bolt on RAG and call it memory, but the real work is in the guardrails: which fields get indexed, which conversations trigger recall, and how the system degrades when the vector store is down. This implementation makes those boundaries explicit, offering a reference design for anyone adding semantic memory to a LangGraph agent without letting it become a single point of failure.

Summary

Keyword matching fails the moment a user rephrases a stored preference. AI Mind v0.4.6 adds a vector index to every active UserMemory, embedding `text` and `tags` via Doubao's embedding model into pgvector. On each eligible chat turn, the user's input is vectorized and run through an ANN search for the top-8 candidates, then squeezed through a six-layer filter chain that enforces a 0.32 cosine-similarity threshold, polarity conflict detection, and a hard budget of three memories totaling 900 characters.

The write path is deliberately narrow: only `active` memories get indexed, and only the `text` and `tags` fields are vectorized. The recall path is conservative by design, preferring to return nothing over injecting irrelevant context. A 1500ms timeout and two layers of try/catch ensure a failed embedding call or a dead pgvector query degrades to zero memories without blocking the chat stream.

The result is a retrieval system that finds "don't be too abstract" when the stored preference says "explain in plain language first," without a single LLM query-rewrite call.

Takeaways

— Only `active` UserMemory documents are vector-indexed; suppressed or inactive memories pass `false` to `store.put()` and stay out of the semantic candidate pool.

— Vector indexing is scoped to `['text', 'tags']` — other fields like `sourceConversationId` and `reason` are excluded to prevent irrelevant semantic matches.

— Semantic recall triggers only on `ordinary_chat` and `tool_assisted_ordinary_chat` paths; hydration, sidebar loads, and agent chains skip it to avoid wasted embedding API calls.

— User input is truncated to 800 characters (first 400 + last 400) and used as-is for vector search — no LLM query rewriting, keeping latency and cost down.

— PostgresStore's vector mode is used instead of hybrid search because hybrid mode would full-text search the entire JSON document, bypassing the field whitelist.

— Top-8 candidates from ANN search pass through six filter layers: eligibility check, score validity, a 0.32 cosine-similarity threshold, polarity conflict detection, stableKey deduplication, and conflict handling.

— The 0.32 threshold was calibrated from real score distributions of the Doubao embedding model on Chinese UserMemory data, not chosen arbitrarily.

— Final injection is capped at three memories and 900 total characters; each memory is clipped to 300 characters to keep token budgets predictable.

— A 1500ms Promise.race timeout and two nested try/catch blocks guarantee that any embedding or pgvector failure returns an empty array and lets the chat continue normally.

— Failure logs are sanitized: they record event name, provider type, error category, and duration, but never raw query text, memory text, or embedding vectors.

Conclusions

The decision to skip LLM query rewriting is a deliberate trade-off: it saves a round-trip and avoids rewrite-induced semantic drift, betting that the embedding model's own generalization is good enough for the recall task.

Scoping vector indexing to explicit fields rather than the whole JSON document is a security and relevance boundary that many RAG pipelines overlook, and it prevents metadata leakage into semantic search results.

The six-layer filter chain is effectively a cost-control mechanism disguised as quality assurance — each layer removes candidates that would waste context-window tokens without improving the answer.

Polarity conflict detection treats the current user utterance as higher authority than stored memory, which is correct for a chat assistant but would be wrong for a system that must enforce persistent rules.

Hard-coding the embedding model ID and version into persisted document metadata creates a future-proofing hook: when the model changes, old vectors are automatically distrusted rather than silently producing degraded results.

Concepts & terms

pgvector

A PostgreSQL extension that adds a vector data type and index methods (IVFFlat, HNSW) for approximate nearest neighbor (ANN) search, enabling semantic similarity queries directly in the database.

Cosine distance

A similarity metric that measures the cosine of the angle between two vectors. Values range from -1 to 1, with 1 meaning identical direction. It is widely used for text embeddings because it captures semantic relatedness independent of vector magnitude.

ANN (Approximate Nearest Neighbor)

A class of algorithms that trade a small amount of accuracy for much faster vector search compared to exact k-NN. pgvector uses ANN indexes so similarity search over millions of vectors completes in milliseconds.

Embedding model

A neural network that converts text into a fixed-length array of floating-point numbers (a vector). Texts with similar meanings produce vectors that are close together in the vector space, enabling semantic search without keyword overlap.

Hybrid search

A retrieval strategy that combines vector similarity search with traditional full-text (keyword) search, often merging results via reciprocal rank fusion. It can improve recall when exact term matches matter, but requires careful scoping to avoid searching irrelevant fields.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗