LLMs Don't Forget You — They Were Never Listening
Treating an LLM as a stateful conversation partner is the root cause of broken multi-turn logic, ballooning API bills, and context-window overflows. Recognizing that the application owns all memory forces a shift from prompt tweaking to context engineering, which is where production AI systems actually succeed or fail.
Most LLM APIs are stateless by design. A model does not sit waiting for the next message; it receives a request, processes the input it is given, and terminates. If a second request omits the history from the first, the model has no basis to know what was said before. The familiar chat experience is an application-layer construct: the client maintains a `chatHistory` array and resubmits the full conversation in every `messages` payload.
This statelessness is not a flaw but a deliberate engineering choice that keeps servers horizontally scalable and avoids the cost of persisting per-user state. The trade-off is that developers must manage context explicitly. A minimal implementation simply appends user prompts and assistant responses to an array, but that array cannot grow indefinitely. Token costs rise, latency increases, and the context window eventually overflows.
Real applications move beyond simple append-only history into context engineering. Strategies include trimming old messages with an LRU-like mindset, persisting critical facts separately, summarizing long exchanges, and injecting project rules or retrieved documents. The model only knows what the application chooses to send, so the quality of an AI feature depends as much on context assembly as on the model itself.
The mental model of a 'conversation' with an LLM is a UX convenience that obscures the stateless request-response reality underneath, and that mismatch causes persistent bugs in multi-turn applications.
Statelessness is a scaling strategy borrowed from HTTP, not a limitation of AI; the real engineering challenge is deciding what to remember, not how to remember it.
LRU is a useful starting heuristic for history eviction, but time-based deletion alone is dangerous because a single early constraint like 'use Vue 3' can be more valuable than the last ten chat turns.
The industry's shift in vocabulary from Prompt Engineering to Context Engineering signals that the bottleneck is no longer how to ask a question but how to assemble the right information for each request.