When AI Forgets Mid-Chat: Three Token-Management Strategies That Keep Context Alive

AI's "Memory" Comes at a Cost — The Art of Token Management

The memory didn't crash, but the API started returning error 400 context_length_exceeded. This article dives into the principles of large model context windows, comparing three strategies: sliding window, summary compression, and structured storage.

01. Scenario Recreation: AI Suddenly "Amnesia"

User feedback: "The AI was quite smart at first, but by the 50th round, when I mentioned an account I asked it to remember 20 minutes ago, it actually asked me 'What account?'"

Checking the backend logs revealed an error returned when calling the OpenAI API:

{
  "error": {
    "message": "This model's maximum context length is 16385 tokens. 
                 You requested 21000 tokens.",
    "type": "invalid_request_error"
  }
}

Root Cause: The attention mechanism in the Transformer architecture has a computational complexity of O(n²), so every model has a hard context window limit. Our frontend passed too many historical messages, directly blowing up the Prompt.

02. Strategy Comparison and Decision Matrix

Strategy	Principle	Information Retention	Extra Cost	Applicable Scenarios
Sliding Window	Keep only the last N rounds, discard earlier ones	⭐⭐	Zero cost	Short Q&A, customer service
Summary Compression	Use AI to condense history into a summary	⭐⭐⭐⭐	Extra LLM call	Long document analysis, complex reasoning
Structured Memory	Extract entities/intents into a vector database	⭐⭐⭐⭐⭐	Requires vector DB	Personalized assistants, Agents

03. Strategy 1: Sliding Window (Low Complexity)

The frontend maintains a sliding window, taking only the last MAX_ROUNDS rounds when sending a request:

const MAX_ROUNDS = 10; // Keep the last 10 rounds (20 messages)

function buildPrompt(allMessages: Message[]): Message[] {
  if (allMessages.length <= MAX_ROUNDS * 2) {
    return allMessages;
  }
  // Trim
  const trimmed = allMessages.slice(-MAX_ROUNDS * 2);
  // [Fatal Flaw] User asks "What was that plan we talked about earlier?" — AI has no idea what "earlier" means
  return trimmed;
}

Fatal Flaw: If the user asks in round 50, "What was the plan you initially suggested?", the AI already discarded the plan from round 1.

04. Strategy 2: Summary Compression (Medium-High Complexity) — Recommended for Production

Core Process:

1. Trigger Threshold: Triggered when conversation rounds exceed 15, or estimated Tokens exceed 70% of the total window.
2. Compression Action: Call an LLM (a cheaper model like GPT-3.5-turbo can be used) to generate a historical summary.
3. Context Reconstruction: Subsequent requests = [System: Historical Summary] + Last 5 rounds of full conversation.

Prompt Engineering Template:

const SUMMARY_PROMPT = `
You are a conversation summary assistant. Please compress the following conversation history into a summary of no more than 200 words.
Strict requirements:
1. Retain the user's core goals/problems.
2. Retain key conclusions and action suggestions given by the AI.
3. Retain all important entities (dates, amounts, names, accounts, code logic).
4. Discard redundant information like pleasantries and repeated confirmations.

Conversation History:
{history}

Summary:
`;

// Trigger in code
async function compressHistory(messages: Message[]): Promise<string> {
  // Only compress the first half of non-system messages (older history)
  const toCompress = messages.slice(0, -6); 
  const recent = messages.slice(-6);
  
  const summary = await callLLM({
    messages: [
      { role: 'system', content: SUMMARY_PROMPT.replace('{history}', 
        toCompress.map(m => `${m.role}: ${m.content}`).join('\n'))
      }
    ],
    max_tokens: 300,
    temperature: 0.3,
  });
  
  // New architecture: Summary + Recent messages
  return {
    summary,
    recent,
    totalTokens: estimateTokens(summary) + estimateTokens(recent),
  };
}

Cost Consideration: Each compression consumes about 15% more LLM computing resources (but far lower than the Token cost of transmitting the full history). Set a compression interval; don't compress every round.

05. Strategy 3: Structured Memory (Hybrid Memory Architecture)

For assistant-type applications, we need long-term memory. A vector database (like Pinecone, Milvus) or simple relational storage can be introduced:

// Extract structured information
interface MemoryEntity {
  type: 'user_preference' | 'fact' | 'task';
  key: string;
  value: string;
  timestamp: number;
}

// After each conversation, call the structured extraction function
async function extractMemory(messages: Message[]): Promise<MemoryEntity[]> {
  const extractionPrompt = `
  Extract structured memories from the recent conversation, returned as a JSON array:
  [
    { "type": "fact", "key": "user_name", "value": "John" },
    { "type": "user_preference", "key": "coding_language", "value": "Python" }
  ]
  Conversation content: ${messages.slice(-4).map(m => m.content).join('\n')}
  `;
  // Call LLM to extract...
}

The final context sent to the large model becomes: System: Summary + Structured Memory + Last 3 rounds of conversation.

06. Key Point: Transparent User Experience

The frontend interface must indicate the "context truncated" status to the user, otherwise they will think the AI has become stupid.

<div className="context-hint">
  <InfoIcon />
  <span>The current conversation has exceeded {MAX_ROUNDS} rounds. The AI only retains the full memory of the last {MAX_ROUNDS} rounds.
    <a onClick={showFullHistory}>View full history</a>
  </span>
</div>

---

📖 Next Article Preview (Part 04)

No Loss on Refresh, Resume on Disconnect — Persistence and Recovery Mechanisms
Scenario Preview: The user's signal is intermittent on the subway, and the AI gets stuck mid-sentence; the user accidentally hits F5, and a half-hour conversation disappears completely..
Core Highlights: IndexedDB+Dexie.js encapsulation / SSE native Last-Event-ID disconnection resume / Exponential backoff reconnection algorithm / Normalized state (entities+order) + Valtio precise re-rendering

---

👋 The next part is even more exciting, follow me to see it first!

Your 'Like 👍 + Bookmark ⭐' is the best encouragement for tech creators!