An LLM Only Predicts the Next Token: the three gaps that turn a chatbot into an Agent
This is Part 1 of the 10-part "Hand-Building an Agent" series. Series guide and table of contents
Figure 1-1 Series map. This installment only covers the yellow box in the middle: the brain.
Preface
The guide left a question: what really separates an Agent from the chatbot I use every day?
When I first came across the word Agent, I assumed it meant "a smarter chatbot" — just a stronger model. Later, when I wrote one myself, I realized I was wrong. The model used inside an Agent can be the same one used in a chatbot. The difference lies entirely outside the model.
To see what's added on the outside, you first have to know what a model alone can and can't do. So this installment doesn't touch Agents at all; it looks only at the brain. By the end you'll have a 20-line program I call Xiaoji v0.1. It can only chat, and the few things it can't do are exactly what the next nine installments will add, one by one.
Key points of this installment:
- An LLM only does one thing: guess the next word from the preceding text.
- The model remembers nothing; a chat app's "memory" comes from the program resending the full history every time.
- A model alone can't do three things. Fill those in and you get the three parts of an Agent.
1. Where chatbots and Agents differ
Consider a concrete task. I have a notes folder containing my study notes, and I want to know which ones mention RAG.
With a chatbot, the process goes: I ask how to find them, it tells me I can use a search command; I run the command myself in the terminal; I paste the results back into the dialog; it compiles them into a list. Three or four round trips, and the one doing the running is me.
With an Agent, I only say "look through notes and tell me which ones mention RAG." It lists the files itself, opens each one and reads it, then tells me which ones.
Figure 1-2 On the left the errand runner is me; on the right it's it.
The difference between the two is who runs the loop of "act, look at the result, decide the next step."
| Chatbot | Agent | |
|---|---|---|
| Who acts | Me | It |
| Who looks at results | I read and paste back | It reads itself |
| When to stop | Stops after one answer | Stops when the job is done |
So my current understanding of an Agent: a program that can act for itself, read results for itself, and decide the next step for itself, until the job is done. Its core is still that LLM; around it are the things that let it "see" and "act", plus a loop that keeps it going around.
So what is the model itself?
2. The LLM only does one thing: guess the next word
What an LLM does is deceptively simple: give it a stretch of text and it guesses what the next word is.
For example, give it "今天天气真" (the weather today is really...), and it scores every word that could follow. "好" (good) is most likely, "不错" (nice) next, with a bit of probability on "热" (hot) and "冷" (cold). Then it picks one, say "好", appends it, turning the text into "今天天气真好" (the weather today is really good). Then with this new sentence it guesses the next word again, appends it again. It writes word by word like this until, at some step, it picks a special token meaning "end", and stops.
Figure 1-3 The model only repeats one motion: look at the preceding text, score, pick one, append.
Tip The "word" I'm using here is more precisely called a token. A token can be a single character, a whole word, or half an English word; how it's split depends on the model's own vocabulary. Thinking of it as a "word" is fine for now; later, when we're computing lengths and prices, the unit is tokens, and I'll switch terms then.
When I'm asking it a question, how is it "continuing the text"? Because in its view, the conversation is also just a stretch of text: first comes "what the user said", and next it's "what the assistant says". The stretch it writes after that is the answer we see.
2.1 It only guesses words — why does it seem to know everything
To guess the next word accurately, memorization isn't enough. To get the word after "法国的首都是" (the capital of France is), it needs geography. To get the word after "小王把钥匙忘在公司了,回到家门口,他发现自己" (Xiao Wang forgot his keys at the office; when he got home, he found he...), it has to understand that without keys you can't get through the door.
During training the model reads enormous amounts of text and does this kind of fill-in-the-blank again and again, adjusting itself whenever it guesses wrong. To guess better, it's pushed into learning grammar, common sense, and the shape of step-by-step reasoning. All of these abilities live in a large pile of numbers inside it, called parameters.
Once training ends, the parameters are fixed. Every later call to the model uses those same fixed parameters. It won't get smarter from having chatted with me, nor will it change next time just because I corrected it once. This point comes up again later.
2.2 Why it says wrong things with a straight face
I used to think the model had a database in its belly and consulted it when a question came in. There is no "look it up" step. The answer is written word by word, and at each step it picks whatever fits most smoothly right there.
Smooth and correct are two different things. Most of the time what's smooth is also correct, because most of what it has read is correct. But on things it has never seen, it can still produce a very smooth passage that reads as real. This is what's usually called a hallucination.
2.3 Why the same question gives different answers each time
Because the "pick one" step involves randomness. The setting that controls the randomness is called temperature.
Figure 1-4 The candidate words are still the same candidates; temperature changes how daring the pick is.
With temperature low, the highest-scoring word is picked almost every time, and answers are stable. With temperature high, words further down also get a chance; answers are more varied and more likely to go off track. For tasks that need stability — writing code, pulling information from documents — go low; for naming things or copywriting, you can go a bit higher.
2.4 What "thinking mode" is about
Many models now have a thinking mode that "thinks" for a stretch before answering. It hasn't changed how it works; it's still writing continuations — it just writes a draft first, then writes the official answer. With the draft sitting in front, the later answer can reference it, so harder questions turn out better, at the cost of being slower and burning more tokens.
3. It doesn't remember you: full history gets resent every time
When I use chat software, it remembers what I said three turns ago. I always assumed the model remembered; it wasn't until I wrote code that I saw the model remembers nothing.
3.1 A small demonstration
The code below makes three consecutive requests. What matters is the difference between the second and third: they ask the same question, and the only difference is whether the previous turn is carried along.
Tip Before running, install the
openaipackage and set three environment variables, as described in the guide, section six. If you only want the idea, you can skip running the code.
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
def chat(messages):
resp = client.chat.completions.create(model=os.environ["LLM_MODEL"], messages=messages)
return resp.choices[0].message.content
# First request: tell it my name
first = chat([{"role": "user", "content": "我叫阿树,记住了吗?"}])
print("第一次:", first)
# Second request: send only the new question, without the previous turn
print("第二次(不带历史):", chat([{"role": "user", "content": "我叫什么名字?"}]))
# Third request: carry the previous Q&A along verbatim
print("第三次(带上历史):", chat([
{"role": "user", "content": "我叫阿树,记住了吗?"},
{"role": "assistant", "content": first},
{"role": "user", "content": "我叫什么名字?"},
]))
My output looks like this; yours will differ in wording:
第一次: 记住了,你叫阿树。之后我就这样称呼你。有什么想聊的?
第二次(不带历史): 我不知道你的名字——你还没有告诉我。……
第三次(带上历史): 你叫阿树。
The first time it says "got it", and the second time it doesn't recognize me. It isn't blowing me off; it genuinely has nowhere to store things. Every request is brand new to the model; all it sees is what was sent this time. The third time it answers correctly because I resent the earlier conversation verbatim.
3.2 Where a chat app's "memory" comes from
The software does it. It keeps the whole conversation in a list; each time I add a new line, it sends the entire conversation from the start, together with the new line, to the model.
Figure 1-5 The longer the conversation, the more is sent each time. On the model's side it's like meeting me for the first time every time.
In the API this list is called messages. Each entry has two things: who said it (role) and what was said (content). There are three roles in use:
| Role | Who's speaking | What it's for |
|---|---|---|
system |
The person who wrote the program | The opening brief: what it is and how it should behave. Usually first. |
user |
Me | What I say |
assistant |
The model | What it said before |
3.3 Xiaoji v0.1: a 20-line chat program
With these in hand, a program that can hold a continuous conversation is very short. Below is all of Xiaoji v0.1. The thing to watch is how the messages list grows a little each pass through the loop.
import os
from openai import OpenAI
client = OpenAI(base_url=os.environ["LLM_BASE_URL"], api_key=os.environ["LLM_API_KEY"])
MODEL = os.environ["LLM_MODEL"]
# The whole conversation lives in this list. The model itself remembers nothing; it all depends on us sending it each time.
messages = [{"role": "system", "content": "你是小记,一个帮我整理学习笔记的助手。回答尽量简短。"}]
while True:
user_input = input("我:").strip()
if user_input in ("exit", "quit", "q"):
break
messages.append({"role": "user", "content": user_input})
resp = client.chat.completions.create(model=MODEL, messages=messages)
reply = resp.choices[0].message.content
messages.append({"role": "assistant", "content": reply}) # its answer is recorded too, to be sent along next round
print("小记:" + reply)
Only three steps actually matter:
- Append what I said to the list.
- Send the whole list to the model.
- Append the model's answer to the list.
Remove step three and it stops recognizing me, just like in the demonstration.
3.4 Three consequences of this
"The model remembers nothing; it all re-sends" is something we'll come back to again and again.
- The longer the conversation, the more is sent each time — slower and pricier. And a model has an upper limit on how much it can read at once, so eventually it won't fit. That's what installments three and four solve.
- The model only sees this list. What goes into the list and how it's arranged directly determines how well the model performs. Everything in the list is what we'll uniformly call context.
- The list is in our hands, so we can edit it: delete the useless parts, compress a long stretch into a summary, put in the material it needs. The memory and retrieval discussed later both do exactly this — put things into this list and take things out of it.
4. With only a brain, there are three things it can't do
Run v0.1 and ask it three questions.
| I ask | Its response | What it shows |
|---|---|---|
| What's today's date? | On my side it directly gave a date, and it was wrong | Everything it knows stops at the day training ended |
| How many notes are in my notes folder? | Says it can't see my folder, asks me to send it the file list | It has never read my stuff, and I didn't send it this time |
| Create a todo.md in notes for me | Says it can't operate my computer, suggests I create it myself | The only thing it can produce is text |
The last two answers are honest. The first is trickier: it doesn't know today's date, but it isn't necessarily going to admit it — as said before, it just wrote a very smooth continuation.
Three shortcomings: it doesn't know new events, it can't see my stuff, and it can't do things.
A stronger model still has all three, because they come out of "it only writes continuations". The model has exactly two sources of information: what it read during training, and what was sent to it in this request. Its only possible action is to output text.
5. Give it eyes, hands, and feet
Now that the shortcomings are clear, so are the fixes.
The first two are "doesn't know". We can't control what it read during training, so put what it needs to know into the request. Add a line to the system message saying "today is [year-month-day]", ask what day it is, and it answers correctly. Same for notes: put the note contents into messages, and it "sees" them. What goes into the request and how — that's the eyes on the map, i.e. context.
The third is "can't do". The model can only output text, so have it say in words what it wants to do, e.g. "I want to read the first note in notes", and our program does the reading for it, then puts what it read back into messages for the model to see. These errand-running programs are called tools — the hands and feet on the map.
Figure 1-6 Three shortcomings, two kinds of fixes.
Put together, this is the skeleton of the whole series:
Agent = brain (model) + eyes (context) + hands (tools)
Eyes and hands work together. Whatever the hands fetch from outside has to go into the context for the brain to see; after the brain sees it, it decides what the hands should do next. This back-and-forth is the loop on the right in Figure 1-2.
Once it can see and act, something still has to manage it — keep it from deleting the wrong file or spinning endlessly. That's the outermost reins on the map, to be covered in installment eight.
Common misconceptions
I got all of these wrong myself once.
| What I used to think | What's actually true |
|---|---|
| An Agent uses a smarter model | The same model, wired up with tools and put in a loop, is an Agent |
| The model remembers what we've chatted about | It doesn't; the program resends the whole conversation each time |
| It answers so fluently, it must be right | Fluency comes from its ability to write; it hasn't checked anything |
| The more we chat, the better it understands me | Parameters are fixed after training; its "understanding me" depends on what this request carries |
| Temperature at minimum means every answer is identical | It's just more stable; there's no guarantee the same input always gives the same output |
Note This article describes general principles. Specific model services differ in implementation details, so your results won't exactly match mine; that's normal.
Summary
- An LLM does one thing: look at the preceding text, guess the next word, write word by word.
- It remembers nothing. A chat app's memory is the program storing the whole conversation in
messagesand resending all of it each time. - A brain alone can't do three things: doesn't know recent events, can't see my stuff, can't act.
- Add eyes (context) and hands (tools), and let it loop around, and you have an Agent.
This version's Xiaoji:
| Xiaoji v0.1 | |
|---|---|
| What it has | A brain (model) and a messages list |
| What it can do | Hold a continuous chat, remember what's been said in this conversation |
| What it can't do | Know today's date, see my notes, touch any file |
Next installment
I asked Xiaoji to count how many notes are in notes, and it said "I cannot view your file".
I could paste the file list to it myself, but then the errand runner is still me. How do I get it to look for itself? The model only outputs text and can't even touch a folder — when people say "calling a tool", who exactly is doing the calling?
The next installment gives it hands and feet.
Next: Hand-Building an Agent (2): growing hands and feet — tool calling and a while loop
Top 1 from juejin.cn, machine-translated. The original thread is authoritative.
I strongly agree that following along and being correct are two different things. I used to always complain that models make things up and spent ages grinding on prompts, but it turns out they never actually verify anything—they just keep the conversation rolling along.