LLMs Don't Forget You — They Were Never Listening

When I first started writing LLM interfaces, I had a very natural misconception:

Since I was having a continuous conversation with the AI, the model should remember what was said before.

For example, if I first tell it:

Please remember my name is Xiaolin.

And then in the next round ask:

What is my name?

If it can answer, we easily think "the model remembered me."

But from the perspective of API calls, things are not like this.

Most LLM APIs are inherently stateless. The model is not a chat partner sitting there waiting for you; it is more like a computation service that gets temporarily woken up each time. You send the information needed for this request, it generates a result based on this input, and then this request ends.

The next time you make a request, if you don't include the previous content, it won't know what happened before.

What is Stateless?

Stateless means: every request is independent, and the server does not rely on the previous request by default.

This is very similar to the HTTP protocol.

When you open a webpage, submit a form, or call an interface, these are essentially independent HTTP requests. HTTP itself does not naturally remember "what this user just did." If the server wants to identify who you are, it usually relies on mechanisms like cookies, tokens, sessions, and Authorization headers.

LLM APIs are similar.

When the model receives a request, it can only see the content given to it in this request. It will not automatically know what you said in the last round, nor will it automatically save all chat history.

So the so-called "continuous conversation" is not the model saving memories itself, but the application layer re-sending the necessary historical dialogue to the model each time.

Why are LLMs Designed to be Stateless?

Intuitively, being stateful seems more convenient: the server remembers everything the user said, and continues the chat directly next time.

But in real engineering, this brings a lot of pressure.

LLM services face a large number of user requests. If the server had to save the complete conversation state for every user long-term, several problems would arise:

• Increased server pressure
• More difficult scaling
• Requests are hard to distribute to arbitrary machines
• More complex failure recovery
• Higher privacy and data management costs

The benefit of being stateless is that every request can be handled by any server.

As long as the request itself carries the necessary information, the backend doesn't need to care about "which machine handled this user before." This is very important for high concurrency, high availability, and horizontal scaling.

So statelessness is not a flaw of LLMs, but a very common backend design.

The real problem is: since the model doesn't remember, we have to decide for ourselves what content should be included in this request.

Messages are the Context the Model Sees

In interfaces like Chat Completions, we often pass a messages array:

const messages = [
  { role: "system", content: "You are a rigorous assistant" },
  { role: "user", content: "Please remember my name is Xiaolin" },
  { role: "assistant", content: "Okay, I remember, your name is Xiaolin." },
  { role: "user", content: "What is my name?" }
];

The most critical point here is:

The model doesn't know "my name is Xiaolin" from some hidden memory, but because this sentence is still inside messages.

That is to say, messages is the context the model can see for this turn.

If the second request only sends this sentence:

[
  { role: "user", content: "What is my name?" }
]

The model has no reliable basis to know who you are.

It might guess, or it might say it doesn't know. Either way, it's essentially not that it "forgot," but that you didn't give it context in this request.

A Minimal chatHistory Example

The following minimal demo uses chatHistory to simulate a continuous conversation:

const chatHistory = [
  { role: 'system', content: 'You are a rigorous assistant' }
];

chatHistory.push({
  role: 'user',
  content: 'Please remember my name is Xiaolin'
});

const response = await client.chat.completions.create({
  model: 'deepseek-v4-flash',
  messages: chatHistory
});

chatHistory.push({
  role: 'assistant',
  content: response.choices[0].message.content
});

After the first round of requests ends, the code puts the model's reply back into chatHistory.

Then in the second round, ask again:

chatHistory.push({
  role: 'user',
  content: 'What is my name?'
});

const response2 = await client.chat.completions.create({
  model: 'deepseek-v4-flash',
  messages: chatHistory
});

It looks like the model "remembered," but in reality, the program saved the previous conversation into an array and sent the whole thing again in the next request.

This detail is very important.

Many people, when first writing chat applications, only save user messages and not the model's replies. This breaks half the context. Because the model not only needs to know what the user said in the next round, but also needs to know what it replied in the last round.

The conversation history should be a complete chain:

system rules
user's first input
assistant's first reply
user's second input
assistant's second reply
...

The more complete this chain, the better the model can understand where the current conversation has developed.

chatHistory Cannot Grow Infinitely

Since history is included every time, can we just stuff all conversations into messages permanently?

Simple in theory, unrealistic in engineering.

Because the longer messages gets, the greater the token cost.

When the token count increases, several direct problems arise:

• Higher request costs
• Slower response times
• Easier to exceed the context window
• Old information interferes with new tasks
• Model attention is distracted by irrelevant content

So a chat system cannot just "append history"; it must also learn to "manage history."

This moves from simple API calls into context engineering.

LRU Thinking: The Most Recent Isn't Necessarily the Most Important, But It's Usually More Useful

The concept of LRU can help understand chat history trimming.

LRU stands for Least Recently Used, meaning the content used least recently is prioritized for eviction.

In the context of chat history, it can be roughly understood as:

Recently discussed content is more likely to be used again immediately, while content from too long ago that is unimportant can be prioritized for compression or removal.

For example, if a user has been debugging an error, the error messages, code snippets, and modification records from the last 10 turns are usually more important than a casual chat from much earlier.

But you can't mechanically delete based on time here.

Some information, although old, is very critical:

This project uses Vue 3.
All interfaces go through /api.
Do not modify the user's existing code.
All answers should be in Chinese.

This information cannot be deleted just because it's "old."

So real context management usually combines several strategies:

• Keep recent conversations directly
• Persist important facts long-term
• Periodically summarize long conversations
• Retrieve project materials on demand
• Place fixed rules into the system prompt or project rule files

Statelessness Shifts Responsibility to the Application Layer

After understanding statelessness, many AI applications become much clearer.

Tools like ChatGPT, Cursor, Claude Code, and Codex seem to understand the current task very well, not because the underlying model naturally remembers everything, but because they do a lot of context management outside the model.

For example:

• Current file content
• Relevant code snippets
• Terminal output
• User's recent instructions
• Project rule files
• Tool execution results
• Materials retrieved via RAG
• External context provided by MCP

These things are selectively organized into the model's current request.

So there is a very core saying in AI engineering:

The model is responsible for generation; the application is responsible for providing context.

The stronger the model, the better it can indeed reason; but if the context is wrong, insufficient, or messy, it will still go off track.

From Prompt to Context, to Harness

Early on, people focused heavily on Prompt Engineering, which is how to write a good prompt sentence.

But when you start building real AI applications, you find that one prompt is far from enough.

Because the model itself has several structural limitations:

• Stateless: won't automatically remember history
• Has a context window: can't stuff content in infinitely
• Probabilistic generation: the same input may produce different outputs
• Cannot directly operate the external world: needs tools and a runtime

So later, Context Engineering emerges.

It's not concerned with "how to ask a question," but with "what background, rules, materials, history, and tool results should this request carry."

Further down the line is Harness Engineering.

That is, building a more complete engineering system outside the model: memory, tools, loops, validation, permissions, rules, context compression, making the model's capabilities more stable and controllable.

Statelessness is the starting point of this line.

Because as long as the model doesn't automatically remember, developers must learn to design memory.

Summary

LLM APIs are typically stateless.

They don't naturally remember the previous round of conversation. The "continuous chat" we see is essentially the application layer putting the necessary history into messages each time, allowing the model to see the context again in this request.

So when writing AI applications, you can't just know how to call the interface; you must also think clearly about:

• Which history to bring
• Which content can be deleted
• Which information should be summarized
• Which materials should be retrieved
• Which rules should be permanently saved

Statelessness makes services easier to scale, and it also hands the task of context management to the developer.

True AI engineering capability often hides within these trade-offs.