跪拜 Guibai
← All articles
Artificial Intelligence · Frontend · Backend

An LLM Only Predicts the Next Token: the three gaps that turn a chatbot into an Agent

By 十有八七 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

A stronger model leaves all three gaps intact, so agent quality is bounded by context design and tool coverage, not by model choice. Teams shipping LLM features can expect the real engineering to sit on the caller's side: resending history, building tools, and policing the loop.

Summary

An LLM inside a chatbot and an LLM inside an Agent can be the same model. This first installment of a ten-part hand-building series puts the entire difference outside the weights: who acts, who reads the result, and when to stop. A chatbot answers one turn and stops; an Agent keeps looping until the job is done.

The core claim is that a model does one thing only: look at the preceding text, score every candidate next token, pick one, append, and repeat until an end token appears. Grammar, common sense, and the shape of reasoning get compressed into parameters during training and frozen afterward; no conversation or correction changes them. Fluency is not verification, which is why hallucinations happen. Temperature tunes how risky the pick is, and thinking mode just drafts before answering while burning more time and tokens.

None of that provides memory. In the demonstration, the model says it remembers the name, forgets it when the next request omits history, then recalls it when the previous turn travels along verbatim. Xiaoji v0.1, a roughly 20-line Python loop, stores the whole conversation in a messages list and sends all of it on every turn. That leaves three gaps no stronger model can close: it doesn't know today's date, can't see a local folder, and can't create a file. The fixes, injecting facts into context and letting the model state an action your program executes, are the eyes and hands in the formula Agent = brain + eyes + hands.

Takeaways
— An LLM generates text one token at a time: score every candidate next token, pick one, append it, and repeat until an end token appears.
— Model parameters are fixed after training; chatting with or correcting the model never changes them.
— Hallucination happens because each step picks the most fluent continuation; fluency and correctness are different things.
— Temperature controls sampling randomness: low produces stable output, high produces variety, and minimum temperature does not guarantee identical output for identical input.
— Thinking mode still writes continuations; it just drafts a reasoning pass before the final answer, costing more time and tokens.
— A model stores nothing between requests; a chat app keeps the whole conversation in a messages list and resends all of it on every request.
— messages entries consist of a role (system, user, assistant) and content; the system message is the opening brief.
— Xiaoji v0.1 is a roughly 20-line Python loop: append the user turn, send the whole list, append the model reply.
— Longer conversations make each request slower and costlier and eventually exceed the model's context window.
— Because the messages list is in the program's hands, it can be edited: trimmed, compressed into summaries, or given extra reference material.
— A brain alone cannot know post-training events, cannot see local files, and cannot perform actions; its only output is text.
— The fixes are to put current facts and file contents into messages (context, the eyes) and to let the model state an intended action in text that your own program executes (tools, the hands).
— The series formula: Agent = brain (model) + eyes (context) + hands (tools).
Conclusions

The position that a stronger model leaves all three gaps intact reframes agent capability as context and tool engineering: once you accept that a model only writes continuations, model choice matters less than what goes into the request and what the program can do with the output.

The 'I remember your name' then 'I don't know your name' demo is a useful failure case; the memory users attribute to the model is a caller-side artifact that breaks the moment the app changes how it resends history.

Resending full history puts a concrete price on remembering: every remembered turn adds time, tokens, and pressure against the context window, which is what pushes naive 'just keep everything' designs toward memory and retrieval.

Defining hallucination as fluency without verification predicts when it fails, on unfamiliar territory where a smooth continuation exists anyway, even though the model has no way to know it is wrong.

The brain/eyes/hands decomposition gives teams a debugging vocabulary: when an agent misbehaves, the question is whether it is a context problem, a tool problem, or a missing control loop.

Concepts & terms
token
The unit an LLM predicts one at a time. A token can be a single character, a whole word, or half an English word, depending on the model's vocabulary. API pricing and context limits are counted in tokens.
temperature
A sampling setting that controls randomness when the model picks the next token. Lower values make output more stable and repeatable; higher values let less-likely tokens win occasionally, producing more variety and more drift.
hallucination
When the model writes a fluent, plausible-sounding answer that is wrong. It happens because each step picks the most fluent continuation rather than checking a fact.
context
Everything in the messages list that the program sends to the model on a request; beyond training data, it is the only view the model has. What goes in and how it is arranged directly shapes output quality.
tool (function calling)
A program your own code runs after the model outputs an intent as text, for example 'read the first note in notes'. The model cannot touch a filesystem; the tool performs the action and the result is fed back into the context.
From the discussion
Featured comments
组合搭配函数 1 likes

I strongly agree that following along and being correct are two different things. I used to always complain that models make things up and spent ages grinding on prompts, but it turns out they never actually verify anything—they just keep the conversation rolling along.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗