An LLM Only Predicts the Next Token: the three gaps that turn a chatbot into an Agent
A stronger model leaves all three gaps intact, so agent quality is bounded by context design and tool coverage, not by model choice. Teams shipping LLM features can expect the real engineering to sit on the caller's side: resending history, building tools, and policing the loop.
An LLM inside a chatbot and an LLM inside an Agent can be the same model. This first installment of a ten-part hand-building series puts the entire difference outside the weights: who acts, who reads the result, and when to stop. A chatbot answers one turn and stops; an Agent keeps looping until the job is done.
The core claim is that a model does one thing only: look at the preceding text, score every candidate next token, pick one, append, and repeat until an end token appears. Grammar, common sense, and the shape of reasoning get compressed into parameters during training and frozen afterward; no conversation or correction changes them. Fluency is not verification, which is why hallucinations happen. Temperature tunes how risky the pick is, and thinking mode just drafts before answering while burning more time and tokens.
None of that provides memory. In the demonstration, the model says it remembers the name, forgets it when the next request omits history, then recalls it when the previous turn travels along verbatim. Xiaoji v0.1, a roughly 20-line Python loop, stores the whole conversation in a messages list and sends all of it on every turn. That leaves three gaps no stronger model can close: it doesn't know today's date, can't see a local folder, and can't create a file. The fixes, injecting facts into context and letting the model state an action your program executes, are the eyes and hands in the formula Agent = brain + eyes + hands.
The position that a stronger model leaves all three gaps intact reframes agent capability as context and tool engineering: once you accept that a model only writes continuations, model choice matters less than what goes into the request and what the program can do with the output.
The 'I remember your name' then 'I don't know your name' demo is a useful failure case; the memory users attribute to the model is a caller-side artifact that breaks the moment the app changes how it resends history.
Resending full history puts a concrete price on remembering: every remembered turn adds time, tokens, and pressure against the context window, which is what pushes naive 'just keep everything' designs toward memory and retrieval.
Defining hallucination as fluency without verification predicts when it fails, on unfamiliar territory where a smooth continuation exists anyway, even though the model has no way to know it is wrong.
The brain/eyes/hands decomposition gives teams a debugging vocabulary: when an agent misbehaves, the question is whether it is a context problem, a tool problem, or a missing control loop.
I strongly agree that following along and being correct are two different things. I used to always complain that models make things up and spent ages grinding on prompts, but it turns out they never actually verify anything—they just keep the conversation rolling along.