Token Budgets, Prompt Caching, and the Three Ways to Call an LLM from TypeScript

---

highlight: rainbow theme: cyanosis

1. Basic Concepts

1.1. Token

Token Cost

When calling an LLM, you are not paying by "character count" or "number of messages", but by Token count. A Token is the "minimum semantic unit" obtained after the model segments text—roughly equivalent to "a word" or "a word root". For example, hamburger is segmented into ham + bur + ger, three Tokens, while the is usually 1 Token.

Token ≈ 4 English characters ≈ 0.75 English words ≈ 0.5 Chinese characters (rough estimate; different model tokenizers vary slightly).

Why is Token cost important? Because a single Agent call is usually not as simple as "send one sentence, receive one sentence". Taking Anthropic Claude as an example, fees are divided into three tiers:

Token Type	Meaning	Typical Unit Price (Claude Sonnet Reference)
input_tokens	All prompts you send to the model (system + history + current question)	$3 / 1M
output_tokens	The response generated by the model (including thinking)	$15 / 1M
cache_read_tokens	The part that hits the prompt cache, billed at a lower unit price	$0.30 / 1M
cache_creation_tokens	The part written to the cache for the first time, unit price slightly higher than input	$3.75 / 1M

⚠️ Note: output_tokens are usually about 5 times more expensive than input, so "making the model talk less" saves more money than "making the model read more". This is why Agent systems strictly control the response format (e.g., requiring JSON, requiring brevity) instead of letting the model freely elaborate.

Token Budget/Estimation

When writing an Agent, you need to establish a mental model of a "Token Budget"—the context window is limited, and you must mentally calculate for each call:

Total Token Budget = context_window (e.g., 200K)
            = system_prompt (fixed overhead, ~1-5K)
            + historical messages (grows with conversation, ~1-50K)
            + current input + RAG retrieval snippets (~2-10K)
            + model output (reserved, ~1-4K)
            + thinking (reasoning, ~0.5-32K)
            + intermediate tool call results (~2-20K)

When the total Token approaches 75%~90% of the context_window, context compression needs to be triggered (summarize history, discard irrelevant snippets), otherwise the model will "forget" early information or directly report an error.

To actually estimate the Token count, you can use the tokenizer provided by LangChain:

import { TokenTextSplitter } from '@langchain/textsplitters';

// Estimate how many Tokens a piece of Chinese text roughly consumes
const splitter = new TokenTextSplitter({
  chunkSize: 1000,
  chunkOverlap: 0,
});

const text = 'This is a piece of Chinese text used to estimate the number of Tokens, demonstrating how to budget context space.';
const chunks = await splitter.splitText(text);
console.log(`Approximately ${chunks.length} chunks, each chunk max 1000 Tokens`);

Practical Advice: In the Agent run loop, calculate the currently used Tokens in the context before each turn starts. If it exceeds the threshold, compress—you can implement a token monitor using TokenTextSplitter combined with usage_metadata.

1.2. Prompt

The model itself has no concept of a "task"—it only knows "predict the next Token based on context". You must tell it what to do, how to do it, and what constraints exist through the Prompt.

Message Roles

In LangChain/LangGraph, a conversation consists of a MessageList, where each message carries a role. Common roles:

Role	Produced by	Function	Corresponding Type
system	Developer	Defines the Agent's "personality", capability boundaries, behavioral norms	SystemMessage
user	User	The question / task posed by the user	HumanMessage
assistant	Model	The model's response (including thinking)	AIMessage
tool	Tool	The result returned to the model after tool execution	ToolMessage

Why distinguish roles? Because the model learned during training to "understand semantics based on role stance"—system messages have the highest priority (the model will strictly follow them), user messages are the source of tasks, assistant messages are "what it has said itself" (used to maintain conversation coherence), and tool messages are "feedback from the external world".

Message Types

In LangChain 1.x, all messages come from @langchain/core/messages:

import {
  SystemMessage,
  HumanMessage,
  AIMessage,
  ToolMessage,
} from '@langchain/core/messages';

const messages = [
  // system: Define Agent behavior
  new SystemMessage({
    content: 'You are an assistant that strictly outputs in JSON format. Do not output extra text.',
  }),

  // user: User question
  new HumanMessage({
    content: 'What is the weather like in Beijing today?',
  }),

  // assistant: Model response (may contain tool_calls, indicating the model decided to call a tool)
  new AIMessage({
    content: '',
    tool_calls: [
      {
        name: 'get_weather',
        args: { city: 'Beijing' },
        id: 'call_001',
        type: 'tool_call',
      },
    ],
  }),

  // tool: Tool execution result passed back to the model
  new ToolMessage({
    content: '{"temp": 22, "condition": "sunny"}',
    tool_call_id: 'call_001',
  }),
];

⚠️ The tool_call_id of ToolMessage must strictly correspond to the AIMessage.tool_calls[].id—the model relies on this id to pair the "call" with the "result". Mismatched pairing will confuse the model or cause errors.

Prompt Engineering (ChatPromptTemplate)

Manually assembling message lists is tedious—parameter order is easily wrong, variables are scattered everywhere, and template reuse is difficult. LangChain provides ChatPromptTemplate to make prompts parameterized, templated, and composable:

import { ChatPromptTemplate } from '@langchain/core/prompts';
import { ChatAnthropic } from '@langchain/anthropic';

// ① Basic template: use {variable} as placeholder
const promptTemplate = ChatPromptTemplate.fromMessages([
  ['system', 'You are a {role}, answer in {language}, no more than {maxWords} words.'],
  ['human', '{question}'],
]);

// ② Render message list with variables
const messages = await promptTemplate.formatMessages({
  role: 'Python Tutor',
  language: 'Chinese',
  maxWords: 100,
  question: 'What is a decorator?',
});

// ③ Call the model
const model = new ChatAnthropic({ model: 'claude-sonnet-4-20250514' });
const response = await model.invoke(messages);

Advanced Usage: MessagesPlaceholder for inserting dynamic message lists—reserve a spot in the template to dynamically insert historical conversations, Few-shot examples, RAG retrieval results:

import { MessagesPlaceholder } from '@langchain/core/prompts';

// Reserve {chat_history} placeholder in the template
const templateWithHistory = ChatPromptTemplate.fromMessages([
  ['system', 'You are a customer service assistant, answer user questions based on conversation history.'],
  new MessagesPlaceholder('chat_history'), // ← Dynamic position
  ['human', '{question}'],
]);

// Fill in historical messages when rendering
const messages = await templateWithHistory.formatMessages({
  chat_history: [
    { role: 'human', content: 'I haven\'t received my order yet' },
    { role: 'ai', content: 'Please provide the order number' },
    { role: 'human', content: 'The order number is #12345' },
  ],
  question: 'Where is the shipment now?',
});

Few-shot examples to guide format: Insert a few "question → standard answer" examples into the prompt, and the model will mimic the format and style of the examples when answering. This is particularly effective for structured output and specific styles:

const fewShotTemplate = ChatPromptTemplate.fromMessages([
  ['system', 'You are a sentiment analysis assistant, output in the example format.'],
  // Few-shot examples (user-assistant dialogue pairs)
  ['human', 'This product is amazing!'],
  ['ai', '{"sentiment": "positive", "score": 0.95}'],
  ['human', 'The service attitude was terrible'],
  ['ai', '{"sentiment": "negative", "score": 0.88}'],
  // The actual question
  ['human', '{input}'],
]);

const messages = await fewShotTemplate.formatMessages({
  input: 'It\'s okay, just average',
});
// The model will mimic the previous format and output: {"sentiment": "neutral", "score": 0.5}

System Prompt Writing Tips

The System message determines the Agent's "personality" and capability boundaries. A good system prompt usually contains 5 elements:

Element	Function	Bad Example	Good Example
Role Setting	Give the model a clear "identity"	"You are an assistant"	"You are a Python backend architect with 10 years of experience, specializing in performance optimization"
Capability Boundaries	Tell the model what it can and cannot do	(Not mentioned)	"Only answer technical questions, do not discuss politics or religion"
Output Format	Constrain the response structure	(Not mentioned)	"Answer in Markdown, conclusion first then arguments, no more than 500 words"
Style Anchoring	Set the tone and audience	(Not mentioned)	"Use plain language, target beginners, avoid jargon"
Counter-examples	Prevent common mistakes	(Not mentioned)	"Do not fabricate API names; say 'I'm not sure' when uncertain instead of guessing"

// A complete system prompt example
const goodSystemPrompt = `
You are a Python backend architect, specializing in FastAPI and PostgreSQL.

## Scope of Capabilities
- ✅ Answer FastAPI / SQLAlchemy / PostgreSQL technical questions
- ✅ Review code and provide improvement suggestions
- ❌ Do not answer frontend, mobile, or ops questions
- ❌ Do not discuss programming languages other than Python

## Output Format
- Use Markdown format
- Wrap code blocks with \`\`\`python
- For complex questions, list 3 key points first, then elaborate
- Single response no more than 500 words

## Style
- Easy to understand, targeted at intermediate developers
- Provide concrete examples rather than abstract descriptions

## Notes
- Do not fabricate API names or library names—say so directly when unsure
- Explain reasons and applicable scenarios when recommending solutions
`.trim();

A longer System Prompt is not always better: Experiments show that beyond ~2000 tokens, the benefit gain diminishes (or even decreases due to attention dilution). Put core constraints at the front, details at the back; the model's compliance is highest for instructions at the beginning.

1.3. Context Window

Each time you call the model, there is an upper limit on the total amount of content you can send and receive. Exceeding this limit will cause the request to be rejected or the content to be truncated.

Historical Messages

The largest overhead in the context window is usually historical messages. Imagine an Agent that has been conversing continuously for 50 turns—the previous 49 turns of "user questions + model answers + tool results" all need to be stuffed into the context window. Without any processing, by the 50th turn, it might already be 100K Tokens, approaching the limit.

Common processing strategies:

Strategy	Approach	Applicable Scenarios
Full Retention	Pass all messages as-is	Short conversations (<10 turns), high precision required
Sliding Window Truncation	Keep only the most recent N turns	Simple scenarios, early info will be lost
Summary Compression	Compress old messages into a summary	Long conversations, balancing precision and cost
Retrieval-based	Store old messages in a vector store, retrieve as needed	Ultra-long conversations (hundreds of turns)

A common practice in production environments is a hybrid of summary compression + sliding window: when token usage exceeds 75%, trigger a short-term memory manager to compress early conversations into a summary and inject it into the context.

Token Usage

Every time the model returns a call, it attaches a usage report telling you how many Tokens were used this time. This is the only credible source for Token counts—do not estimate yourself, rely on what the model returns.

import { ChatAnthropic } from '@langchain/anthropic';

const model = new ChatAnthropic({ model: 'claude-sonnet-4-20250514' });
const response = await model.invoke([
  new HumanMessage({ content: 'Hello' }),
]);

// LangChain 1.x: usage is in response.usage_metadata
console.log(response.usage_metadata);
// {
//   input_tokens: 12,
//   output_tokens: 8,
//   total_tokens: 20,
//   input_token_details: { cache_read: 0, cache_creation: 1024 },
//   output_token_details: { reasoning: 0 }
// }

⚠️ Note that Anthropic's input_tokens already includes the cache_read portion. If you directly calculate input_tokens * unit price, you will bill the cache hit portion at full price, leading to double billing. The correct algorithm: (input_tokens - cache_read) * input_price + cache_read * cache_read_price.

Caching Mechanism (Anthropic Prompt Caching)

Anthropic provides Prompt Cache—mark a stable segment of the prompt (like system instructions, long documents, tool definitions) as cache. Subsequent requests with a prefix match will directly reuse the cache, billed at 1/10 the unit price. For long context scenarios (e.g., with long system prompts + conversation history), this can save 80%+ in costs.

import { ChatAnthropic } from '@langchain/anthropic';
import { SystemMessage, HumanMessage } from '@langchain/core/messages';

// ① Long system prompt (assume 5000 tokens, carried with every call)
const longSystemPrompt = `
You are the Apollo AI assistant. Below is the complete product documentation (5000 words):
[A very long document is here...]
`.trim();

const model = new ChatAnthropic({
  model: 'claude-sonnet-4-20250514',
});

// ② First call: use cache_control to mark this segment for caching
const firstCall = await model.invoke([
  new SystemMessage({
    content: longSystemPrompt,
    // Key: mark cache breakpoint (cache the first 5000 tokens here)
    additional_kwargs: { cache_control: { type: 'ephemeral' } },
  }),
  new HumanMessage({ content: 'What does the document talk about?' }),
]);
console.log('First call usage:', firstCall.usage_metadata);
// input_tokens: 5000, cache_creation_tokens: 5000

// ③ Second call: same prefix → automatically hits cache
const secondCall = await model.invoke([
  new SystemMessage({
    content: longSystemPrompt, // Exactly the same
    additional_kwargs: { cache_control: { type: 'ephemeral' } },
  }),
  new HumanMessage({ content: 'How many chapters are in the document?' }), // User question changed
]);
console.log('Second call usage:', secondCall.usage_metadata);
// input_tokens: 5050, cache_read_tokens: 5000 ← Hit!
// 5000 tokens billed at cache_read unit price ($0.30/M), original price is $3/M

Caching Rules:

Dimension	Rule
Matching Method	Prefix match—the first N tokens must be exactly identical to hit
Cache Granularity	Minimum 1024 tokens, maximum 4 breakpoints
Validity Period	5 minutes (ephemeral), expires after that
Best Practice	Put stable, unchanging content at the front (system, long documents), and changing content at the back (user input)

When to use cache: If you have a stable prompt of ≥2000 tokens + multiple calls within the same session, cache is almost mandatory. A common practice is to wrap a prompt builder that automatically appends a cache_control marker when the system message length > 2K.

---

2. Basic LLM Calls

2.1. invoke - Non-streaming Output

invoke is the most basic calling method—you pass in a complete message list and wait for the model to generate a complete response all at once before returning. Suitable for non-interactive scenarios: background batch processing, structured data extraction, tasks that don't require real-time progress display.

import { ChatAnthropic } from '@langchain/anthropic';
import { SystemMessage, HumanMessage } from '@langchain/core/messages';

// 1. Instantiate the model (different providers use different classes, but the API is consistent)
const model = new ChatAnthropic({
  model: 'claude-sonnet-4-20250514',
  temperature: 0.7,
  maxTokens: 1024,
});

// 2. Construct the message list
const messages = [
  new SystemMessage({
    content: 'You are a concise technical assistant, answer in no more than 3 sentences.',
  }),
  new HumanMessage({
    content: 'What is the ReAct pattern?',
  }),
];

// 3. invoke: Block until the complete response is returned
const response = await model.invoke(messages);

console.log(response.content);
// "ReAct = Reasoning + Acting. The model first reasons (Reasoning) to decide the next step,
//  then acts (Acting) by calling a tool, continues reasoning based on the tool result, looping until a final answer is reached."

console.log(response.usage_metadata);
// { input_tokens: 28, output_tokens: 45, total_tokens: 73 }

Instantiation methods for different Providers (API is consistent, only the class name changes):

import { ChatOpenAI } from '@langchain/openai';
import { ChatOllama } from '@langchain/ollama';

// OpenAI
const openaiModel = new ChatOpenAI({ model: 'gpt-4o' });

// Local Ollama (zero privacy leakage, local inference)
const localModel = new ChatOllama({ model: 'qwen2.5:14b' });

Disadvantages of invoke: The user has to wait idly for the model to finish generating before seeing any output. For long answers (e.g., a 2000-word technical document), the user might stare at a blank screen for 10 seconds—poor experience. Therefore, user-facing scenarios should use stream.

2.2. stream - Streaming Output

stream is an incremental return method—every time the model generates a Token (or a small segment), it pushes it to you immediately, and you can render it to the UI in real-time. This is the standard practice for user-facing interactive scenarios.

import { ChatAnthropic } from '@langchain/anthropic';
import { HumanMessage } from '@langchain/core/messages';

const model = new ChatAnthropic({
  model: 'claude-sonnet-4-20250514',
});

// stream: Returns an AsyncIterable, producing chunks one by one
const stream = await model.stream([
  new HumanMessage({ content: 'Explain what a context window is in 200 words.' }),
]);

// Consume chunks one by one using for-await
for await (const chunk of stream) {
  // chunk.content is this small piece of text
  process.stdout.write(chunk.content as string);
  // UI side: append to message bubble to achieve a "typewriter" effect
}
console.log('\n--- Stream ended ---');

Essential differences between stream and invoke:

Dimension	invoke	stream
Return Method	Returns complete AIMessage at once	Returns AIMessageChunk chunk by chunk, can be assembled into a complete message at the end
Time to First Token	High (wait for all generation to finish)	Low (first word appears in tens of milliseconds)
Interruptibility	Difficult (can only abort the entire request)	Easy (can break mid for-await)
Token Usage	Directly in response.usage_metadata	Need to accumulate usage from each chunk (or get it from the last chunk)

⚠️ In stream mode, usage_metadata usually only has the complete value in the last chunk; the usage in intermediate chunks is incremental. You need to accumulate the usage fields (input_tokens / output_tokens) of each chunk in the loop, and the last accumulation gives the true total.

Streaming in Agent Scenarios: When the model decides to call a tool, the stream will first output tool_calls chunks, then the tool executes, the tool result is passed back as a ToolMessage, and the model continues to stream the final answer. LangGraph's createAgent / preModelHook mechanism encapsulates this entire process, detailed in Part 3 Agent Tools.

import { createAgent } from '@langchain/langgraph';

const agent = createAgent({
  llm: model,
  tools: [/* ... */],
});

// Agent streaming execution: automatically handles the loop of "reasoning → calling tool → seeing result → continuing reasoning"
const eventStream = agent.stream(
  { messages: [{ role: 'user', content: 'Check the weather in Beijing and write a poem for me' }] },
  { streamMode: 'updates' }, // Push when each node updates
);

for await (const event of eventStream) {
  console.log(event); // { agent: { messages: [...] } } or { tools: { messages: [...] } }
}

This way, the user can see the whole process of "model is thinking → calling weather tool → writing a poem based on the result", instead of just waiting for a final answer.

2.3. batch - Batch Calls

When you have multiple independent conversations to process in parallel (batch processing, batch classification, batch translation, etc.), batch is much more efficient than looping invoke—it will automatically send requests concurrently, reusing the underlying HTTP connection.

import { ChatAnthropic } from '@langchain/anthropic';
import { HumanMessage } from '@langchain/core/messages';

const model = new ChatAnthropic({ model: 'claude-sonnet-4-20250514' });

// Process 5 independent questions simultaneously
const batchInputs = [
  [new HumanMessage({ content: 'Translate "hello" to Chinese' })],
  [new HumanMessage({ content: 'Translate "world" to Chinese' })],
  [new HumanMessage({ content: 'Translate "good morning" to Chinese' })],
  [new HumanMessage({ content: 'Translate "thank you" to Chinese' })],
  [new HumanMessage({ content: 'Translate "goodbye" to Chinese' })],
];

// batch executes concurrently, returns array of results
const results = await model.batch(batchInputs);

results.forEach((res, i) => {
  console.log(`Q${i + 1}: ${batchInputs[i][0].content} → ${res.content}`);
});
// Q1: hello → 你好
// Q2: world → 世界
// Q3: good morning → 早上好
// Q4: thank you → 谢谢
// Q5: goodbye → 再见

batch vs Promise.all(invoke):

Dimension	Promise.all(inputs.map(invoke))	model.batch(inputs)
Concurrency Control	Completely unlimited, may hit rate limit	LangChain internally has concurrency limits (configurable maxConcurrency)
Error Handling	One failure rejects all	Can configure returnExceptions: true, single failure doesn't block others
Resource Reuse	Each invoke creates a new HTTP connection	Reuses underlying connection pool
Applicability	Few calls (<10)	Large volume calls, batch processing scenarios

// Control concurrency (avoid triggering Provider rate limits)
const limitedBatch = model.withConfig({
  maxConcurrency: 3, // Max 3 requests at a time
});

// Fault-tolerant mode: single failure doesn't block others
const safeResults = await model.batch(batchInputs, {
  returnExceptions: true,
});
safeResults.forEach((r, i) => {
  if (r instanceof Error) {
    console.error(`Q${i + 1} failed:`, r.message);
  } else {
    console.log(`Q${i + 1}: ${r.content}`);
  }
});

A typical application: Using batch mode for batch translation of document paragraphs is 5-10 times faster than invoking one by one (because it reuses the underlying HTTP connection pool, and LangChain has built-in concurrency limit protection).

2.4. Error Retry and Timeout

LLM calls may fail due to network jitter, rate limits, or temporary Provider failures. Production code must implement retry + timeout control:

import { ChatAnthropic } from '@langchain/anthropic';

// ① Configure timeout when instantiating the model
const model = new ChatAnthropic({
  model: 'claude-sonnet-4-20250514',
  timeout: 30_000, // 30-second timeout for a single request
  maxRetries: 3,   // Auto-retry 3 times on failure
});

// ② Use AbortController to interrupt long tasks (especially streaming)
const controller = new AbortController();
setTimeout(() => controller.abort(), 60_000); // Force interrupt after 60 seconds

try {
  const stream = await model.stream(
    [{ role: 'user', content: 'Write a 5000-word novel' }],
    { signal: controller.signal },
  );

  for await (const chunk of stream) {
    process.stdout.write(chunk.content as string);
  }
} catch (err) {
  if (err.name === 'AbortError') {
    console.log('User actively interrupted');
  } else {
    console.error('Call failed:', err);
  }
}

// ③ Custom retry strategy (exponential backoff)
import { RunnableRetry } from '@langchain/core/runnables';

const retryModel = new RunnableRetry({
  bound: model,
  maxAttempts: 5,
  // Exponential backoff: 1s, 2s, 4s, 8s, 16s
  backoffFactor: 2,
  initialDelayMs: 1000,
  // Only retry on specific errors (do not retry business errors)
  retryOnError: (err) => err.message.includes('rate_limit') || err.message.includes('timeout'),
});

Key Principle: Retry only for transient errors (rate limit, timeout, 5xx), do not retry business errors (parameter errors, content violations). Common retry wrapper implementations use exponential backoff for 429/5xx and fail immediately for 4xx.