跪拜 Guibai
← Back to the summary

The 34 Concepts That Separate Prompt Tweaking from Building Real Agents

Between "prompt tweaking" and "building an Agent" lies a whole set of concepts you've never heard of. Token, Context, RAG, CoT, MCP, ReAct, Harness… each word makes sense on its own, but strung together they become confusing.

This article arranges 34 core concepts along a main thread: LLM fundamentals → Prompt → Agent loop → Memory & Retrieval → Multi-Agent protocols → Engineering & Production deployment → Security red lines. For each concept, it provides not just a definition, but also "why it matters" and "what to watch out for in practice." An FAQ and a verified learning path are included at the end.

1. Nail Down the Most Fundamental Understanding First: LLM is a Pure Function

1.1 LLM: Text In, Text Out, Nothing More

Models like GPT, Claude, and Gemini are essentially the same thing: a function that receives text and outputs text.

input prompt → output text

This statement might seem like stating the obvious, but it is the most important insight for avoiding detours in the Agent field.

Because it means: An LLM cannot browse the internet on its own, cannot remember the last conversation, and does not know today's date. If you ask it, "Help me check the weather in Shanghai," the "25 degrees, sunny" it gives you is made up, unless you put the weather data into the Prompt. If you told it last time, "My name is Xiao Ming," it still won't remember this time, unless you stuff that sentence back into the context every time.

All the solutions that "give AI a memory" or "let AI look things up"—RAG, Memory, Tool Use—are essentially wrapping a shell around the function, not the model growing these capabilities itself. Remember this, and half of the subsequent concepts will click.

1.2 Token: The Currency of the AI World

LLMs don't see "words"; they see tokens (sub-word units). This is the numerical basis for all billing, context windows, and latency:

So a "1 million token context window" translates to roughly 750,000 Chinese characters. When writing prompts, use English abbreviations instead of piling on long Chinese text—fewer tokens mean faster responses and thinner bills. This habit can save a significant amount of money over a year in production.

1.3 Context Window: How Much It Can See, and "Lost in the Middle"

The Context Window is the maximum number of tokens an LLM can "see" in a single call. The scale of mainstream models in 2026:

Model Series Context Window
Claude Sonnet 5 / Opus 5 1M
GPT-5.6 1.05M
Gemini 3.5 Flash 1M (Pro up to 2M)
xAI Grok 4.5 500K

But a large window doesn't mean it can be used well. Beyond a certain length, LLMs suffer from Lost in the Middle: content at the beginning and end is remembered well, while content in the middle is "forgotten."

Practical principle: Put the most important instructions at the beginning and end of the Prompt, and auxiliary materials in the middle. In long-context scenarios (like stuffing an entire codebase into the window), this detail directly determines the quality of the answer.

2. Prompt Engineering: How You Talk to the Model is the First Craft

2.1 System Prompt and User Prompt

A single LLM call is usually divided into two layers:

The priority design of the System Prompt is the foundation of safety and quality—it should be an immutable rule layer, while the User Prompt is a variable input layer. Later, when we talk about Prompt Injection, you'll see that mixing these two layers is the beginning of disaster.

2.2 Zero-shot / One-shot / Few-shot

The difference between these three terms is simply "how many examples you gave":

Type Number of Examples Applicable Scenarios
Zero-shot 0 Ask directly, no examples given
One-shot 1 Simple format guidance
Few-shot 2~5 Tasks with strict format requirements, significant accuracy improvement 
Translate the following Chinese into English:
Chinese: 你好 → English: Hello
Chinese: 谢谢 → English: Thank you
Chinese: 再见 → English:

The cost of Few-shot is that the examples themselves consume tokens. More examples are not always better—for tasks with strict formatting, 3~5 are enough. More than that wastes context and easily introduces noise.

2.3 Chain-of-Thought (CoT): Let the Model "Think First, Then Answer"

The core of Chain-of-Thought is to make the LLM output the reasoning process first, then give the conclusion, rather than jumping straight to an answer. Two triggering methods:

  1. Few-shot CoT: Include complete reasoning steps in the examples for the model to imitate
  2. Zero-shot CoT: Add "Let's think step by step" at the end of the Prompt

The cost is very real: CoT significantly increases output tokens, meaning slower and more expensive. But its value for math, logic, and multi-step reasoning tasks is also substantial. In production environments, it's usually only turned on for tasks requiring reasoning; keeping it on for simple tasks is just burning money.

3. Agent: The Step from "Conversation" to "Autonomous Action"

3.1 The Three Elements of an Agent

An Agent is not a "smarter LLM," but an autonomous system with an LLM at its core. All three elements are indispensable:

Element Function
LLM Reasoning / Planning / Decision-making
Actions Means to do things (call tools, write code, query databases)
Loop The heartbeat cycle of Perceive → Decide → Act → Observe → Repeat

The key difference in one sentence: A pure LLM is you ask, I answer, a single interaction; an Agent is the three elements plus a continuous loop, running until the goal is achieved or the budget is exhausted.

Note a common misconception: ReAct is just one pattern of Agent, not the definition of Agent. CodeAct, computer-use, and planning agents are all Agents; they just have different loop methods.

3.2 Tool Use / Function Calling

Let the LLM call external functions you've defined. The LLM returns not text, but a structured call instruction:

{
  "function": "search_weather",
  "args": { "city": "Beijing" }
}

Your program executes the function, throws the result back to the LLM, and the LLM continues reasoning based on the result. Think of the LLM as the brain, and Tool Use as its hands, feet, and senses.

Two practical reminders:

  1. Vendor naming differs: Anthropic calls it "Tool Use," OpenAI calls it "Function Calling." API schemas differ, so align them when writing cross-vendor SDKs.
  2. Tool descriptions are Prompts: The model relies on the description to choose a tool. Writing clearly "what this tool does, what the parameters are, what to do on failure" is more effective than any code-level fallback.

3.3 ReAct: The Most Classic Agent Pattern

Reasoning + Acting. The idea is an explicit "Think → Do → See" loop:

Thought → Action (call tool) → Observation (observe result) → Thought → ...

Loop until a final answer can be given. Most Agent frameworks internally implement this pattern. Understanding it is the key to understanding all Agent frameworks.

3.4 Structured Output: The "Protocol Layer" for Communicating with LLMs

Force the LLM to output according to a fixed schema (like JSON), rather than free text. The response_format parameter in various APIs does this.

Agent frameworks almost entirely rely on it to communicate with the LLM—state transfer at each step of the Agent loop and parameter parsing for tool calls all depend on structured output. The first lesson of writing an Agent is: never let the LLM return free text as a protocol; JSON schema enforcement is the baseline.

3.5 Self-Refine: The Simplest "Reflection" Pattern

Actor produces answer → Critic finds problems → Actor sees feedback and answers again

Self-Refine allows the Agent to self-evaluate the previous round's output and revise it, without needing a persistent memory layer. Tools like Cursor and Cline run variants of this every day (one is the Actor writing code, the other is the Critic reviewing code). It is essentially a sibling pattern to ReAct, the only difference being that the "action" is "regenerate the answer" rather than "call a tool."

4. Memory and Retrieval: Giving AI a Real "Memory"

4.1 Two Orthogonal Classifications of Memory

"Memory" is used loosely and confusingly. Breaking it down, it's actually two independent axes:

Temporal Axis:

Content Axis (CoALA Framework):

Type Meaning Example
Working Temporary info storage Current task steps
Episodic Past experiences User preferences stated last time
Semantic Factual knowledge Technical parameters of a company's product
Procedural How to do something Standard process for calling a specific API

Long-term memory can simultaneously contain episodic + semantic + procedural memories. When designing an Agent memory system, first figure out which cell of memory you need; don't jump straight to "let's use a vector database."

4.2 RAG: Giving AI an External Knowledge Base

The core problem RAG (Retrieval-Augmented Generation) solves: The LLM doesn't know your private data, changing data, or the latest data after its cutoff date. It doesn't modify the model; instead, it stuffs relevant data into the context at inference time. Two stages:

Stage 1: Indexing (Ingest)

document → chunk → embed (vectorize) → store in Vector DB

Stage 2: Querying (Query)

question → embed → semantic search → top-K chunks → stuff into Prompt → LLM answers

Key insight about RAG: It's an architecture for "inserting data at inference time," not "training." When data updates, just rebuild the index; model weights don't need to change at all.

4.3 Embedding and Vector DB

Embedding: Converting text/images into N-dimensional vectors, so that things with "similar meaning" are closer in vector space. By default, this refers to dense embeddings. There are also sparse embeddings like BM25 / SPLADE (matching based on literal tokens).

Vector DB (Vector Database): A storage layer for storing and efficiently querying embeddings. Its core capability is ANN (Approximate Nearest Neighbor search), which is hundreds of times faster than brute-force full scans. Representatives: Pinecone, Chroma, Qdrant, Weaviate, pgvector.

4.4 Chunking, Hybrid Search, and Reranking: The Three Cards for RAG Quality

Chunking: Cutting long documents into small segments suitable for embedding (usually 200~1000 tokens). The cutting method directly determines RAG quality—too fine loses context, too long blurs relevance. This is the step beginners most easily overlook, yet it most affects results.

Hybrid Search: Using semantic search + BM25 keyword search together, then merging and sorting. The default standard for production-grade RAG, more accurate than a single method in most scenarios.

Reranking: The first retrieval round fetches top-50, then a more expensive but more accurate cross-encoder model re-ranks them into top-5 for the LLM. Representatives: Cohere Rerank, bge-reranker. Two-stage retrieval (fast first, then accurate) is the standard play for controlling cost and quality.

4.5 Contextual Retrieval

A method proposed by Anthropic in 2024: Add a "context summary of the entire document" to each chunk before embedding them together, solving the problem of "this piece of text alone doesn't tell you what it's about." When your documents are full of references like "as described above" or "this method," this approach is highly valuable.

4.6 Fine-tuning vs RAG: The Golden Rule

Approach Essence Suitable For Not Suitable For
RAG Insert data into context at inference, no weight changes Latest facts, private docs, frequently changing knowledge Extremely complex format requirements
Fine-tune Retrain model, knowledge burned into weights Stably learning a specific format/style/domain terminology Stuffing "latest facts" (will expire and is hard to update)

Golden Rule: In Agent scenarios, exhaust Prompt + RAG first; only consider Fine-tuning when it's truly insufficient. Many teams jump to fine-tuning immediately, only to find afterward that knowledge still expires, hallucinations still exist, and they've added a training pipeline to maintain.

4.7 Reflexion: The "Full Reflection" That Accumulates Lessons Across Trials

The difference from Self-Refine: Reflexion requires a persistent episodic memory store. After an Agent completes a task, it writes the reflection summary into memory; when the next task starts, it retrieves the memory into the Prompt.

Accumulating lessons across trials is the essence of Reflexion. If it's just "this round answered wrong, try again next round," that's Self-Refine; only when lessons can be stored and retrieved for use in future tasks is it called Reflexion.

5. Multi-Agent and Protocols: When One Can't Do It, How a Group Collaborates

5.1 Three Typical Patterns of Multi-Agent

A single Agent has a capability ceiling, hence multi-Agent collaboration:

Handoff: One Agent hands a task to another, involving context transfer and failure handling. The vast majority of bugs in multi-Agent systems occur at Handoff—context gets lost in transfer, or no one takes over on failure.

5.2 A2A: The Protocol Between Agent and Agent

The Agent-to-Agent Protocol, initiated by Google and governed by the Linux Foundation, reached v1.0 in 2026. It is the sister standard to MCP: MCP governs agent ↔ tool, A2A governs agent ↔ agent.

Remember this division of labor, and the protocol understanding in later chapters won't get messy. Single Agent connecting to tools uses MCP; multi-Agent interconnection uses A2A.

6. Claude Code Ecosystem: Raising an Agent Within a Project

6.1 MCP: The USB Interface for LLMs

Model Context Protocol, an open protocol launched by Anthropic in 2024 and donated to the Linux Foundation in late 2025. Think of it as "the USB interface for LLMs"—it standardizes how external tools connect to LLMs, avoiding writing a custom adapter for each vendor.

It standardizes 3 primitives:

Primitive Function
Tools Functions the LLM can call
Resources Data the LLM can read
Prompts Reusable Prompt templates

The architecture is Server/Client mode: Servers are exposed via stdio (local) or Streamable HTTP (remote). When providing the same set of tools to Cursor, Claude Desktop, and a self-built Agent simultaneously, writing one MCP Server is enough.

6.2 Skills / SKILL.md: Claude Code's "Behavior Packs"

A Skill = a folder containing SKILL.md + optional reference files. Key mechanism: Before processing each message, Claude Code scans the frontmatter description of all Skills—if it matches the current context, it's automatically loaded.

So whether the description is well-written directly determines if the skill gets triggered. In practice, starting with "Use when ..." is most effective, as it's the most direct syntax for describing trigger conditions.

6.3 CLAUDE.md, Slash Command, Hooks, and Subagent

The significance of this toolkit: Engineering "rules, processes, safety, and parallelism" rather than relying on ad-hoc instructions in each conversation.

7. Production Deployment: From "It Runs" to "It Can Go Live"

7.1 Eval: An Agent Without Evaluation is Like Code Without Tests

Run a set of test cases against the Agent to quantify accuracy / latency / cost. An Agent without Eval is like code without tests—you just "feel like it's okay."

Common tools: promptfoo, LangSmith, Langfuse. Build an eval set from day one. Run it every time you change a Prompt, switch models, or adjust RAG, to prevent "fixing one thing and breaking three others."

7.2 Observability: The Confidence to Replay When Bugs Occur

Log every step inside the Agent—which LLM call, which tool was called, what the tool returned, what the final answer was. When a bug occurs, you can replay it, rather than guessing what happened in the middle based on a wrong answer.

This is the dividing line between a production Agent and a toy Agent. My biggest pain point in production when tuning Function Calling was the complete opacity of the intermediate process when the toolchain errored—observability is the cure for this.

7.3 Prompt Caching: A Required Course for Saving Money

The LLM caches the prompt prefix. Next time the same prefix is used, you only pay the cheaper cache hit price (Anthropic up to 90% off, OpenAI 50% off).

Long context + repeated query scenarios (in RAG scenarios, system prompts + fixed document prefixes are the same every time) can save a lot of money. The cost is that the cache has a TTL and a minimum length requirement; a trade-off must be made between hit rate and timeliness.

7.4 Computer Use and Browser Use: Handing "Looking at the Screen" to the Agent

Both are slow, expensive, and can fail, but they are an important extension of the Agent's capability boundary—when no API is available, looking at the screen is the last universal interface.

8. Security and Cost: Two Red Lines That Cannot Be Loosened

8.1 Prompt Injection: Malicious Instructions Hidden in Data

Hiding malicious instructions in content the LLM will read (web pages, documents, tool returns) to induce it to ignore the original task.

The root cause is brutal: The LLM cannot distinguish between "system instructions" and "instructions smuggled in data." Everything it reads is just a piece of text.

Defense triad: Least privilege (Agent only gets permissions necessary for the task), Isolate untrusted content (process separately from system instructions), Human review for high-risk actions (deletion, transfer, external sending must pass a human).

8.2 Lethal Trifecta: The Deadly Triangle

A concept proposed by Simon Willison: When an Agent simultaneously possesses the following three capabilities, it can be manipulated by prompt injection to steal and exfiltrate data:

  1. Access to private data
  2. Exposure to untrusted content
  3. Ability to communicate externally

All three together = a dangerous combination. The defense strategy is to break at least one link—most commonly cutting off external communication, or isolating untrusted input. When designing an Agent architecture, running it through this triangle first is more effective than patching a bunch of rules afterward.

8.3 Guardrails: The Rule Layer to Prevent LLMs from Doing Bad Things

Block prompt injection, PII leakage, harmful output, etc. Note that Guardrails are not equal to security—they are a rule layer, and a good attacker can bypass rules. Guardrails are part of the defense line, not the whole thing.

9. Engineering Mindset: Three-Layer Architecture, From "Tweaking Prompts" to "Building Products"

When you move from "tweaking prompts" to "building products," you need to understand the three-layer engineering division of labor—this is also the most core mental model for Agent engineering in 2026:

9.1 Layer 1: Prompt Engineering

Engineering strings. How to write prompts to make the model output better. This is the entry layer, and also the overrated layer—tweaking prompts can only solve problems at the prompt level.

9.2 Layer 2: Context Engineering

Engineering information. What information goes into the window on each LLM call—RAG results, memory, tool definitions, conversation history. Karpathy calls it "the fine art of filling the window with exactly the information useful for the next step."

The 2026 consensus: Context Engineering determines Agent quality more than Prompt Engineering. The same prompt, fed different quality context, yields vastly different results.

9.3 Layer 3: Harness Engineering

Engineering the execution and control layer. Agent loop, tool registration, context management, permissions, security, retries, circuit breakers—all the code that is neither model weights nor the prompt itself.

Simon Willison's formula: Coding Agent = LLM + Harness. The moat of most Agent products is not in the model, but in the Harness—given the same GPT-5.6 or Claude Opus, the Harness design determines whether it's a joy to use or torture.

9.4 Extension: Loop Engineering and Graph Engineering

Moving from linear loops to explicit graphs is the hallmark of a complex Agent system moving towards controllability.

10. Common Pitfalls: The Top Five Traps for Beginners

  1. Using an Agent as if it were an LLM: Calling the API once and thinking you've "built an Agent." Without Tools, without a Loop, it's just a chatbot in a thin shell.
  2. Jumping straight to a vector database: For two or three documents, a few thousand tokens, just stuff them directly into the Prompt. The complexity of RAG is only worth introducing when the knowledge base truly can't fit.
  3. Frantically tweaking Prompts without building Eval: Parameter tuning without an evaluation set is mysticism; you're just memorizing the last example that failed.
  4. Ignoring the Prompt nature of tool descriptions: Writing tool descriptions casually leads the model to pick the wrong tools and pass wrong parameters. Polish tool descriptions like Prompts, and the error rate can drop by half.
  5. Stuffing everything into a long context: Dumping documents, code, history all at once triggers Lost in the Middle and breaks Prompt Caching. Fetch on demand, place in tiers.

FAQ: High-Frequency Questions for AI Agent Beginners

Q1: What's the difference between an Agent and a regular LLM chatbot?

An Agent is an autonomous system of "LLM + Actions + Loop" that can call tools and loop until a goal is met; a regular LLM is a single Q&A. The simple criterion: does it have Tools, does it have a Loop. If it has neither, it's just a chatbot.

Q2: How to choose between RAG and Fine-tuning?

Exhaust Prompt + RAG first; only consider fine-tuning if insufficient. RAG suits latest facts, private docs, frequently changing knowledge; Fine-tuning suits stabilizing format and domain language feel. Fine-tuning cannot solve the "knowledge expires" problem and introduces training pipeline costs.

Q3: What's the difference between MCP and A2A?

MCP is the protocol for agents to connect to tools (the USB interface for LLMs); A2A is the protocol for communication between agents. Single Agent connecting to tools uses MCP; multi-Agent interconnection uses A2A. They are companion sister standards.

Q4: Is ReAct the entirety of Agent?

No. ReAct is just the most classic Agent loop pattern (Think → Do → See). CodeAct, computer-use, and planning agents are also Agents. Understanding ReAct is the key to understanding frameworks, but don't mistake it for the entirety of Agent forms.

Q5: What's the first thing to do when running an Agent in production?

Build an Eval set and Observability first. Without an evaluation set, you can't judge if a change is good or bad; without log replay, you can only guess blindly at a wrong answer when bugs occur. The earlier these two things are done, the more time you save later.

Summary and Learning Path

This guide has traveled from the essence of LLMs all the way to production deployment, with a single main thread: The LLM is a pure function; the Agent is the hands, feet, memory, and control layer fitted onto this function. First understand the economics of Tokens and Context Windows, then understand the Agent loop, then supplement knowledge with RAG, control execution with Harness, guard quality with Eval, and secure the bottom line with safety red lines—all 30+ concepts then fall into their proper places.

If you're ready to get your hands dirty, follow this path to avoid many detours:

  1. First, use Claude Code / Cursor to write a few Skills, feel what Tool Use is.
  2. Build a ReAct Agent with LangChain, run through the "Think → Do → See" loop.
  3. Connect a RAG Pipeline to the Agent, experience the weight of Context Engineering.
  4. Add Eval and Observability; only after this step is it an Agent "ready for production."
  5. Try Multi-Agent collaboration, understand the boundaries of A2A and MCP protocols.

If this helped you, welcome to subscribe to the RSS for continuous AI engineering practical content. Also welcome to share in the comments the pitfalls you stepped into when building your first Agent.

Original tech blog · Open-source project sharing · AI full-stack creation community idao.fun