Vectorized Processing of Long-Term Memory in AI Chat: From "Keyword Matching" to "Finding It Even When Phrased Differently"

This article is organized based on the real implementation of the AI Mind project. GitHub: https://github.com/HWYD/ai-mind Corresponding code version: v0.4.6 Online link: https://ai.hwyblog.cloud/instant-mind

AI Mind is a continuously iterating Next.js AI Chat project. Starting from the most basic local chat, it gradually added streaming protocols, tool calls, MCP, Skill, and Agent capabilities.

If you are interested in this project, or if this article helps you a little, you are also welcome to drop by GitHub and give AI Mind a Star⭐, which would be a great encouragement for me to continue updating.

---

If an AI Chat can only rely on keyword matching to retrieve a user's long-term preferences, it will likely fail—the system won't recognize it when the user phrases things differently.

For example. A user said "Remember I don't eat cilantro," and later asked "My stomach has been upset recently, recommend something easy to digest"—"stomach" and "digestion" have no lexical overlap with "cilantro" and "dietary restriction," so rule-based matching is completely blind. Another example is response style preference: a user said "Explain technical stuff in plain language first, then add professional terms," and next time asked "What is LangGraph Store? Don't be too abstract"—"plain language" and "don't be too abstract" are semantically connected in Chinese, but have zero lexical overlap, so they still can't be found.

The essence of the problem is simple: when a user phrases things differently, the system can't find previously stored memories.

What this version does is also very direct—add a vector index to each long-term memory, so semantically similar text can be automatically recalled. The core approach: vectorize the text and tags of each UserMemory and store them in pgvector; when a user asks a question, vectorize the question as well, and use cosine distance to find the most similar memories. During writes, PostgresStore automatically calls the Doubao embedding model to generate vectors; during searches, pgvector performs ANN retrieval. The entire pipeline is equipped with a 1500ms timeout and failure degradation, so semantic recall won't become a single point of failure for the chat.

Next, let's break down the entire pipeline from three stages: vector writing, vector recall, and failure degradation.

A few key concepts to clarify first:

• UserMemory: Long-term user memory within the current browser session scope, reusable across sessions, such as "user doesn't eat cilantro" or "explain technical stuff in plain language first, then professionally"
• PostgresStore: The PostgreSQL storage backend provided by LangGraph; this version uses its built-in pgvector capability for vector indexing
• embedding: The process of converting text into an array of floating-point numbers (vectors); semantically similar texts have similar vectors

---

1. Overall Architecture: Where Vectorization Fits In

The vector capability resides within the existing user-memory module, with new additions and changes concentrated in three files:

doubao-embeddings.ts (new)  → Text-to-vector translator, calls the Doubao embedding API
provider.ts (modified)      → Adds vector index configuration when creating PostgresStore
retrieval.ts (heavily modified) → Recall pipeline with vector search + multi-level filtering

The entire pipeline is:

[Write] Model extracts candidate memories → Program validates → store.put(value, ['text','tags'])
                                              ↓
                              PostgresStore internally calls embedDocuments() to generate vectors
                                              ↓
                                         Stored in pgvector

[Recall] User input → Truncated to 800 characters → store.search(mode:'vector', query)
                                              ↓
                              PostgresStore internally calls embedQuery() to generate vector
                                              ↓
                              pgvector ANN search → top-8 candidates
                                              ↓
                              6-layer filtering → up to 3 items injected into model

---

2. Vector Write Process: How Memories Enter Vector Storage

2.1 Build the Vector Index First

The prerequisite for writing is that the Store itself has vector capability. In provider.ts (PostgresStore configuration and instance management), an index configuration is added when creating PostgresStore:

PostgresStore.fromConnString(connectionString, {
    index: {
        dims: dimensions,              // Vector dimensions, determined by doubao-embedding-vision
        distanceMetric: 'cosine',       // Cosine distance
        embed: doubaoEmbeddings,        // Embedding model instance
        fields: ['text', 'tags'],       // Only build vector index for these two fields
    },
    schema: 'langgraph_user_memory',
})

Each of the four parameters has a clear responsibility:

• dims: The dimension of the pgvector column, must match the output of doubao-embedding-vision. This value is passed in via the environment variable AI_MIND_USER_MEMORY_EMBEDDING_DIMENSIONS; if not found, startup fails directly—if dimensions are wrong, building the vector index is pointless.
• distanceMetric: 'cosine': Cosine distance. The closer the direction of two vectors, the higher the score. For Chinese semantic similarity, cosine distance is the community-verified default choice.
• embed: A DoubaoEmbeddings instance. PostgresStore automatically calls its embedDocuments() and embedQuery() methods internally during writes and searches, so there's no need to manually manage the vectorization process.
• fields: ['text', 'tags']: This is the most critical security boundary. PostgresStore can index the entire JSON document by default (fields: ['$']), but the UserMemory document also contains fields like sourceConversationId, reason, confidence that shouldn't participate in semantic search. Explicitly declaring ['text', 'tags'] means only the content of these two fields will be vectorized; the rest serve only as filtering and sorting metadata.

2.2 Deciding Whether to Build an Index During Write

The third parameter of store.put() controls whether to build a vector index:

await store.put(
    namespace,                                // ['ai-mind', 'user-memory', 'v1', '<hash>']
    stableKey,                                // Document key
    toStoredValue(nextDocument),              // Complete UserMemoryDocument JSON
    nextDocument.status === 'active'          // Third parameter: index configuration
        ? ['text', 'tags']                    // active → build vector index
        : false                               // inactive/suppressed → don't build index
)

Only memories with active status pass ['text', 'tags']. Suppressed or inactive memories pass false—even if they were previously indexed, the vector won't be updated again. This design ensures: Only currently active, validated long-term memories enter the candidate pool for semantic search.

2.3 Attaching Semantic Metadata During Write

In addition to the vector index, a semantic metadata block is written inside the document of each active memory:

{
  "semantic": {
    "embeddingModelId": "doubao-embedding-vision",
    "embeddingProviderKind": "volcengine-ark-doubao-openai-compatible",
    "semanticIndexFields": ["text", "tags"],
    "semanticIndexedAt": "2026-07-10T12:00:00.000Z",
    "semanticIndexVersion": "user-memory-semantic.v3"
  }
}

This metadata is persisted in the document, not a temporary state in memory. Its purpose: during recall, check which embedding model was used to build this memory's index and whether the index version matches. If the model is upgraded or the index version changes, the old vector index is no longer trusted—it's directly filtered out to avoid cross-version semantic drift.

2.4 Embedding Model: Reused API Key, Independent Model

The vectorization engine is doubao-embeddings.ts (Doubao Embedding client), inheriting from LangChain's Embeddings base class. What it does is extremely simple—POST to Volcengine's /embeddings endpoint, converting an array of text into an array of floating-point numbers:

// Request body
{
    model: 'doubao-embedding-vision',
    input: texts,              // Texts to be vectorized
    dimensions: 1024,          // Vector dimensions
    encoding_format: 'float'   // Return float32
}

After returning, it validates each item: is it an array, is it a number, are the dimensions correct. If any condition is not met, it throws an error directly, to be caught by the upper-level catch for degradation.

The API Key and baseURL reuse the project's existing Doubao chat model configuration, but the model id is fixed to doubao-embedding-vision. This means: Switching models in the chat interface will not affect the embedding model selection. The quality of semantic recall is independent and stable.

---

3. Vector Recall Process: How to Retrieve Relevant Memories When a User Asks

3.1 Trigger Timing

Semantic recall is not triggered for every chat. In chat-orchestrator.ts (stage orchestration and strategy judgment for the main chat chain), the run() method first checks:

// Only ordinary_chat and tool_assisted_ordinary_chat trigger
this.userMemoryContextMessages = isUserMemoryContextEligibleRequest(this.request)
    ? await this.resolveUserMemoryContextMessages(
        session.toolBoundModel ? 'tool_assisted_ordinary_chat' : 'ordinary_chat'
    )
    : []

Excluded paths: Tasklist Agent, Delivery Chain, hydration (restoring session state during frontend page load), sidebar session list loading, conversation switching. These paths don't need long-term memory supplementation; triggering semantic recall would be a pure waste of embedding API calls.

3.2 Query Normalization: Truncate, Don't Rewrite

Before using user input as a retrieval query, only deterministic processing is done, no LLM rewriting:

function normalizeSemanticQuery(latestUserText: string, config): string {
    const normalized = normalizeWhitespace(latestUserText)  // trim + collapse extra whitespace

    if (normalized.length <= 800) {
        return normalized   // Not exceeding 800 characters, return directly
    }

    // Exceeded: keep first 400 characters + last 400 characters
    const head = normalized.slice(0, 400)
    const tail = normalized.slice(-400)
    return `${head}${tail}`.slice(0, 800)
}

Two key decisions:

• No LLM query rewrite. Don't use a model to rewrite "Don't be too abstract" into "Explain in simple language" before searching. An extra LLM call means extra latency and cost, and rewriting might introduce ambiguity. Search directly with the user's original words, letting the embedding model handle semantic generalization itself.
• For over-length truncation, take first 400 + last 400. Users might put key information at the end—"By the way, I don't eat cilantro"—taking only the first 800 would lose it. Taking half from the front and half from the back is a compromise strategy.

3.3 Vector Search: One Call, Top-8 Candidates

The normalized query is handed to PostgresStore for vector search:

const items = await store.search(namespace, {
    limit: 8,                  // Only take top-8
    mode: 'vector',            // Only vector search, no hybrid/text
    query: normalizedQuery,     // Query already truncated to 800 characters
})

PostgresStore internally completes three things automatically: calls embed.embedQuery(query) to convert the query into a vector → performs ANN (Approximate Nearest Neighbor) search in pgvector → returns the 8 most similar items, each with a score (cosine similarity score).

Why not use hybrid search? PostgresStore's hybrid mode simultaneously performs full-text search on the complete JSON of store.value—meaning fields like sourceConversationId, reason that shouldn't participate in search would also be hit, directly bypassing the field whitelist. This version only uses vector mode.

Why topK=8? This is a "sufficient but not wasteful" number. Browser-session-level memory usually only has a few dozen UserMemories; 8 candidates are enough to cover most relevant memories. More candidates won't improve recall quality but will increase the computational load for subsequent filtering.

3.4 Post-Recall Filtering: 6-Layer Security Check

The 8 candidates returned by vector search are not injected directly. In retrieval.ts (semantic recall core pipeline), toVectorSemanticCandidates() applies 6 layers of filtering to them:

Layer	Filter Condition	What It Filters Out
1	isUserMemorySemanticEligible	status≠active, confidence<0.7, semantic metadata mismatch
2	Score validity	Missing score, NaN, negative, greater than 1
3	Score threshold	score < 0.32 (calibrated for doubao-embedding-vision in this version)
4	Conflict detection	User's current input conflicts with memory polarity (e.g., "don't eat cilantro" vs "want to eat cilantro")
5	stableKey deduplication	Same key, only keep the one with higher score or newer updatedAt
6	Conflict handling	Conflicting memories with same type+subject+facet, keep the newer one

How was the score threshold of 0.32 determined? It wasn't a random guess. It was calibrated based on the real score distribution of doubao-embedding-vision in this version's Chinese UserMemory scenario. Semantically related memories typically fall between 0.4 and 0.7, unrelated ones between 0.1 and 0.2. 0.32 is a conservative dividing line—better to recall less than to inject randomly.

Layer 4's conflict detection is worth expanding on. Suppose a user memory is "likes to eat peaches" (polarity=prefer), but the current input contains negation words like "don't eat," "don't want," "don't"—this memory is not injected. Conversely, if the memory is "doesn't eat cilantro" (polarity=avoid), and the current input contains "want to eat," "can eat," it's also not injected. The current user input always takes priority over long-term memory.

3.5 Final Selection: Budget Control

Filtered candidates are sorted by score in descending order and enter selectFromSemanticCandidates() for final truncation:

function selectFromSemanticCandidates(candidates, config): SelectedUserMemory[] {
    const selected = []
    let totalChars = 0

    for (const candidate of candidates) {
        if (selected.length >= 3 || totalChars >= 900) break   // Hard limit
        const text = clipUserMemoryText(candidate.document.text, 300)  // Each ≤300 chars
        if (!text || totalChars + text.length > 900) continue
        selected.push({ score, stableKey, tags, text, type })
        totalChars += text.length
    }
    return selected
}

Three hard limits: maximum 3 items, 300 characters per item, 900 characters total. This limit not only controls token consumption but also prevents too many long-term memories from overshadowing the main content—the current session's short-term context (summary, pinnedDecisions, recent messages) is the primary basis for the response.

---

4. Failure Degradation Process: Semantic Recall Must Not Become a Single Point of Failure

This is a hard bottom line for v0.4.6. The entire execution body of retrieveRelevantUserMemories() is wrapped in try/catch:

try {
    const searchItems = await withSemanticTimeout(
        vectorSemanticSearch(store, namespace, normalizedQuery, input.limit),
        input.timeoutMs  // 1500ms
    )
    const candidates = toVectorSemanticCandidates(searchItems, input.latestUserText, config)
    return selectFromSemanticCandidates(candidates, config)
} catch (error) {
    // Sanitized log: only records event name, provider type, search mode, error category, and duration
    logUserMemoryRetrievalEvent('semantic-retrieval-degraded', {
        degradationKind: error.message === 'USER_MEMORY_SEMANTIC_TIMEOUT' ? 'timeout' : 'failure',
        providerKind: config.semanticEmbeddingProviderKind,
        searchMode: 'vector',
        errorName: error instanceof Error ? error.name : 'UnknownError',
    })
    return []  // Degrade to 0 items, chat continues
}

Three layers of degradation protection:

1. Timeout protection: withSemanticTimeout() implements a 1500ms timeout using Promise race. After timeout, it rejects, never waiting indefinitely.
2. Exception catch-all: Embedding API is down, pgvector query errors, Store connection broken—any exception is caught, returning an empty array.
3. Upper-level secondary catch-all: resolveUserMemoryContextMessages() in chat-orchestrator.ts also has its own independent try/catch, also returning an empty array.

Log sanitization: Failure logs only record event name, provider type, search mode, error category, and duration. Absolutely do not record raw query text, raw UserMemory text, embedding vectors, or provider raw responses. Once this data enters logs, it could be persisted, collected by monitoring systems, or mistakenly displayed to users.

Effect after degradation: 0 UserMemory items injected → normal chat continues → streaming output unaffected → ThreadState unaffected. For the user, it's just "long-term memory wasn't used this time," the same as if the feature wasn't enabled.

---

5. Summary

Above, three pipelines were dissected: vector writing, vector recall, and failure degradation. Together, these three pipelines form the complete picture of this semantic recall capability.

The core decision of the write pipeline is "only index what should be indexed"—text and tags go into the vector index, other fields serve only as filtering metadata. The third parameter of store.put() controls whether to build an index; active memories pass ['text', 'tags'], suppressed memories pass false. PostgresStore internally automatically calls the embedding model to generate vectors, no need to manually manage pgvector tables.

The core decision of the recall pipeline is "better to recall less than to inject randomly"—800-character truncation, top-8 candidates, 0.32 score threshold, 6-layer filtering chain, each step is doing subtraction. The memories finally injected into the model are at most 3 items, totaling 900 characters, serving as supplementary context rather than authoritative answers.

The core decision of the degradation pipeline is "semantic recall must not become a single point of failure"—1500ms timeout, exception catch-all, log sanitization, two-layer try/catch protection. If any link fails, the chat continues as usual, just without long-term memory supplementation this time.

Adding a vector index to long-term memory essentially allows the system to find previously stored preferences even when the user phrases things differently. The technical means aren't complex; what's complex is guarding the boundaries well—which fields go into the index, which paths trigger recall, how to degrade on failure. It's precisely these boundaries that determine whether this capability is "stable and usable" or "problematic from time to time."

---

The natural next directions are hybrid recall (semantic + keyword complement), Memory Inspector UI, account-level memory. But each version only pushes one step forward, and each step first ensures it doesn't break existing stability.

---

Project Link

👉 GitHub: https://github.com/HWYD/ai-mind

👉 Online Experience: https://ai.hwyblog.cloud/instant-mind

If this article or the AI Mind project has been helpful to you, you are also welcome to give the project a Star⭐. This support is very important to me and will give me more motivation to continue organizing the implementation process, design trade-offs, and pitfall reviews for subsequent versions.