Hand-Built Agent Memory and Context Compression in 200 Lines of TypeScript
🚀 Welcome to the fifth installment of the "No Frameworks, Hand-Built AI Agent" series.
Even if you haven't read the previous articles, you can start directly from this one. All you need to know is: we already have a minimal AI Agent that can call tools, read and modify files, and execute commands.
However, this Agent still has an obvious problem: every time it handles a new problem, it's like meeting you for the first time, with no memory of what was discussed before. Even if we keep all messages in memory, as the conversation grows longer, it will eventually fill up the model's limited context window.
In this article, we won't switch models or introduce an Agent framework. Instead, we'll build two fundamental capabilities from scratch: using a session record to preserve conversation history, and using context compression to control the length of long-term conversations.
Ultimately, we'll get an Agent that can be continuously queried in the terminal and automatically summarizes early content when the conversation gets too long.
Series Table of Contents:
- No Frameworks, Hand-Built AI Agent: (Part 1) Getting It Running First
- No LangChain, Hand-Built AI Agent: Giving the LLM "Hands" to Read Project Files
- The Core Loop of an AI Agent Is This Simple: Hand-Building an Agent Loop
- How Does Claude Code Modify Code by Itself? The Answer Lies in These 4 Tools
- This Article: No LangChain: Hand-Building Agent Conversation Memory and Context Compression with 200 Lines of Code
- More practical content coming soon...
First, let's "poison" you with a look at the "complete form" we'll eventually craft through this series
Let's First See Why the Current Version Suffers from "Amnesia"
The Agent from Part 4 can already loop and call tools, roughly following this process:
User inputs task
↓
Agent.run() creates a local messages context array inside the function
↓
Model decides to call a tool
↓
Registry executes the tool and puts the result back into messages
↓
Model responds
↓
Program ends
The problem is hidden in src/agent.ts:
async run(prompt: string): Promise<string> {
const messages = [
{ role: "system", content: "..." },
{ role: "user", content: prompt },
];
// Loop calling model and tools here
}
messages is a local variable of run(). After the function returns, it disappears.
Therefore, there is no memory between the following two launches:
npm start -- -prompt "First, read README.md"
npm start -- -prompt "Continue, tell me how to start this project"
After the first run() execution ends, the local context array variable messages is no longer referenced and will be garbage collected by the JavaScript engine. When npm start is executed the second time, it starts a completely new process: its memory has no messages from the previous session, and it only sends "Continue" as new user input to the model. The model naturally doesn't know which file was read before or what step the task was on.
So the problem isn't that the model suddenly "became dumb" or actively "forgot"; it's that the model can only see the context carried by the current request. We neither saved the previous round's messages nor re-included them in the new request, so it naturally cannot continue the previous task.
What We Aim to Achieve by the End of This Article
If you've used Claude Code or Codex CLI, the final effect we're implementing will feel familiar: after starting the powercode program, it no longer exits after answering one question, but stays in the terminal for continuous follow-up queries.
$ npm start
> First, check what files are in the current workspace and tell me the general structure
AI：...
> What are the main contents inside?
AI：...
> Continue, help me see if there's anything noteworthy
AI：...
The "that" and "continue" in the latter two inputs both depend on the previous conversation. powercode will retain the messages of the current session, so the model knows what you just discussed and doesn't need the background re-explained each time.
To achieve this user experience, we will gradually add three fundamental capabilities to the current powercode:
- Use
Sessionto save chat history within the same process; - Use an interactive loop to support continuous input and follow-up queries in the terminal;
- Summarize overly long history to prevent context from accumulating indefinitely;
You can first remember the final effect:
After starting the program once, you can ask continuous follow-up questions; when the conversation gets long, earlier content is automatically turned into a summary, while recent content remains in its original form.
As for how to continue after a process restart, and how to implement a Plan Mode similar to Claude Code or Codex, we'll tackle those separately in the next article.
🚀 Companion source code for this section: powercode 👈Click it
If you encounter problems midway, you can check the source code for troubleshooting. The code for subsequent chapters will also be continuously updated. If this project helps you, feel free to give it a Star ⭐
First, Clarify the Correspondence Between the Main Project and the Article
This article corresponds to powercode. It is the main project from the end of Part 4, with the following core files:
powercode/
├── provider.json # Model configuration, read from project root
├── src/
│ ├── agent.ts # Agent Loop
│ ├── chat.ts # OpenAI compatible client
│ ├── config.ts # provider.json configuration
│ ├── main.ts # CLI entry point
│ └── tools/ # Registry and four tools
└── workspace/ # Workspace where the Agent actually reads and writes
There are two directories here that are easy to confuse:
powercode/src
The Agent source code we are developing. The new Session and Compactor added in this article are written here.
powercode/workspace
The workspace handed over to the Agent for operation. Demo files read, modified, and generated by the tools are placed here.
So, this article always executes npm run build and npm start in the powercode project root directory, but the Agent's four tools only operate on workspace and won't practice on its own src directory. The later compression demo script will centrally place files into workspace/context-demo, without affecting other content in the workspace.
This article will continue development directly on this main project code, without needing to switch to other projects or copies. The following steps will sequentially add session memory and context compression, ultimately achieving the continuous interaction effect shown earlier.
Step 1: Handing Local messages Over to Session
What Exactly Is a Session
Don't overthink "memory" for now.
Session is simply a folder for holding messages: what the user says, the model's replies, and the results returned by tools, all placed inside in chronological order.
Session
├── User question
├── Model's tool call
├── Tool execution result
├── Model's next reply
└── Subsequent user questions
This step only solves one problem: within the same process, the next time the Agent is called, it can still retrieve the previous messages.
It is not yet permanent memory. After the process exits, the Session in memory will also disappear; task state across restarts will be handled later by PLAN.md and TODO.md.
Create src/context/session.ts
import type OpenAI from "openai";
// Message types in a session
export type Message =
OpenAI.Chat.Completions.ChatCompletionMessageParam;
// Session class
export class Session {
readonly createdAt = new Date(); // Creation time
updatedAt = new Date(); // Update time
private readonly history: Message[] = []; // Session history
/**
* Append messages to the session history
* @param messages Messages to append
*/
append(...messages: Message[]): void {
this.history.push(...messages);
this.updatedAt = new Date();
}
/**
* Get the session history
* @returns Session history
*/
getHistory(): Message[] {
return structuredClone(this.history);
}
}
First, look at the two most important methods:
session.append(message); // Save new messages in order
session.getHistory(); // Retrieve the complete history
getHistory() uses structuredClone() to return a deep copy. Regardless of whether the Agent later adds or removes array elements, or modifies the message objects within, it won't affect the history saved internally by the Session.
Modify src/agent.ts to Stop Creating messages Itself
First, look at the original approach:
const messages = [
{ role: "system", content: "..." },
{ role: "user", content: prompt },
];
Now, change this to:
Receive user input
↓
Write to Session
↓
Each time the model is requested, retrieve the complete history from Session
↓
Model replies and tool results continue to be written back to Session
Completely replace src/agent.ts with the following content:
import type OpenAI from "openai";
import { ChatClient } from "./chat.ts";
import type { Session } from "./context/session.ts";
import { Registry } from "./tools/registry.ts";
const MAX_STEPS = 8; // Maximum number of steps
export class Agent {
constructor(
private readonly client: ChatClient, // Chat client
private readonly registry: Registry, // Tool registry
private readonly session: Session, // Session
) {}
/**
* Run the agent
* @param prompt Prompt
* @returns Agent's answer
*/
async run(prompt: string): Promise<string> {
// Append user prompt to session history
this.session.append({
role: "user",
content: prompt,
});
// agent loop
for (let step = 1; step <= MAX_STEPS; step += 1) {
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{
role: "system",
content:
"You are power-code, a development assistant. Please prioritize reading real files; actively run commands to verify results after modifications. Please answer in Chinese.",
},
...this.session.getHistory(), // Append session history
];
// Call the model
const message = await this.client.complete(
messages,
this.registry.getDefinitions(), // Pass tool list definitions
);
if (!message) {
throw new Error("The model did not return a message.");
}
// Append model reply to session history
this.session.append(message);
const toolCalls = message.tool_calls ?? [];
if (toolCalls.length === 0) {
return message.content ?? "The model did not return text content.";
}
// Handle tool calls
for (const toolCall of toolCalls) {
if (toolCall.type !== "function") {
throw new Error(`Tool type not currently supported: ${toolCall.type}`);
}
const name = toolCall.function.name;
console.log(`Round ${step}: AI calls ${name}`);
let result: string;
try {
// Execute tool call
result = await this.registry.execute(
name,
toolCall.function.arguments,
);
console.log(`✓ ${name} execution complete\n`);
} catch (error) {
const reason = error instanceof Error ? error.message : String(error);
result = `Tool execution failed: ${reason}`;
console.log(`✗ ${result}\n`);
}
// Append tool call result to session history
this.session.append({
role: "tool",
tool_call_id: toolCall.id,
content: result,
});
}
}
throw new Error(`Execution exceeded ${MAX_STEPS} rounds, stopped.`);
}
}
After a tool executes successfully, the terminal only displays the tool name and completion status, no longer printing the full results like file content or command logs. The complete result is still written to the Session as a tool message, so the model can continue to use the real results; we are just temporarily omitting the user-facing display. When we upgrade the TUI later, we'll put tool results into expandable and collapsible areas.
The biggest difference between this code and Part 4 is just one thing: messages is no longer the master of memory.
Part 4: Agent creates and saves messages itself
Part 5: Session class saves history, Agent retrieves it each time
Note that the system message is not written into the Session. It's an operational rule that must be included with every request, not task history; so it's fine to re-attach it each round.
Let the Entry Point Create and Pass in the Session
Open src/main.ts and add the import:
import { Session } from "./context/session.ts";
Find the original:
const agent = new Agent(client, registry);
Replace with:
const session = new Session();
const agent = new Agent(client, registry, session);
Don't rush to change the command-line interaction yet. Run a build first:
npm run build
If it passes, it means the Session has been connected to the Agent, but because the entry point still only calls agent.run() once, the effect of continuous conversation isn't visible yet.
Step 2: Let One Process Continuously Receive Tasks
Agent Loop and Terminal Loop Are Not the Same Thing
It's easy to confuse two loops here:
Agent Loop
Model → Tool → Model → Tool → Model answers
Responsible for completing the current single task
Terminal Interaction Loop
User input → Agent Loop → Wait for user input again
Responsible for preventing the program from exiting after completing one task
The previous article implemented the first type of loop. This article needs to add the second type of loop outside it.
Modify src/main.ts
For this step, we'll only add interaction, without introducing --plan and -dir, to keep the changes small.
Completely replace src/main.ts with the following content:
import { createInterface } from "node:readline/promises";
import { stdin as input, stdout as output } from "node:process";
import { resolve } from "node:path";
import { Agent } from "./agent.ts";
import { ChatClient } from "./chat.ts";
import { loadConfig } from "./config.ts";
import { Session } from "./context/session.ts";
import { BashTool } from "./tools/bash.ts";
import { EditFileTool } from "./tools/edit-file.ts";
import { ReadFileTool } from "./tools/read-file.ts";
import { Registry } from "./tools/registry.ts";
import { WriteFileTool } from "./tools/write-file.ts";
async function main() {
const config = await loadConfig();
const client = new ChatClient(config);
const workDir = resolve(process.cwd(), "workspace");
const registry = new Registry();
registry.register(new ReadFileTool(workDir));
registry.register(new WriteFileTool(workDir));
registry.register(new EditFileTool(workDir));
registry.register(new BashTool(workDir));
const session = new Session();
const agent = new Agent(client, registry, session);
const runPrompt = async (value: string) => {
console.log("AI is thinking...\n");
const answer = await agent.run(value);
console.log(`AI：${answer}`);
};
const readline = createInterface({ input, output });
try {
console.log("Entered interactive mode, type exit or quit to leave.\n");
while (true) {
const value = (await readline.question("> ")).trim();
if (!value) continue;
if (value === "exit" || value === "quit") break;
try {
await runPrompt(value);
} catch (error) {
const message = error instanceof Error ? error.message : String(error);
console.error(`This round execution failed: ${message}`);
}
}
} finally {
readline.close();
}
}
main().catch((error: unknown) => {
const message = error instanceof Error ? error.message : String(error);
console.error(`Startup failed: ${message}`);
process.exit(1);
});
Here, workDir is fixed to workspace: the program configuration and Agent source code are still loaded from the powercode root directory, but read_file, write_file, edit_file, and bash only execute within the workspace. In the next article, we'll upgrade this fixed directory to a command-line parameter selectable via -dir, and introduce specific task directories.
Why the Program Keeps Waiting for Input
The key lies in these few lines:
const readline = createInterface({ input, output });
while (true) {
const value = (await readline.question("> ")).trim();
if (!value) continue;
if (value === "exit" || value === "quit") break;
await runPrompt(value);
}
readline is a built-in terminal input tool in Node.js. Think of it as an "operator": responsible for handing the content you type in the terminal to the program.
createInterface() creates this operator. readline.question("> ") first displays a >, then pauses here waiting for your input. Only after you press Enter does the program continue, passing the content to the Agent.
After the Agent answers, while (true) returns to the beginning and waits for your next input:
Display > and wait for input
↓ User presses Enter
Agent processes and answers
↓
Return to loop start
↓
Display > and wait for input again
It won't become an infinite loop running idle, because each round pauses at await readline.question() waiting for you. When you type exit or quit, break ends the loop, and readline.close() closes the terminal input, exiting the program.
Now, running npm start directly will enter interactive mode. After the program answers a question, it won't exit but will continue waiting for your next question.
npm start
During interaction, typing exit or quit will end the program and return to the normal terminal command line.
The key is that these two lines are outside the loop:
const session = new Session();
const agent = new Agent(client, registry, session);
If they were placed inside the while loop, each user input would get a brand new Session, and the effect would revert to "amnesia."
Run a Minimal Verification First
npm run build
npm start
In the terminal, enter sequentially:
> First, read README.md and tell me what this project is
> So, what tools does it currently have?
> exit
The reason the "so" in the second question has a chance to be correctly understood is that the Session within the same process still holds the first round's messages.
Think of it as a colleague who has been sitting in the meeting room the whole time: each time, only a new discussion topic is started, but the meeting minutes haven't been cleared.
After this step is complete, pause and build again:
npm run build
Step 3: When History Gets Too Long, Don't Stuff the Entire Original Content into the Model
Why You Can't Keep the Complete History Forever
Now the Session saves all messages, seemingly already having "memory."
But if the Agent continuously reads files, runs commands, and handles errors, the messages will look like this:
User question
→ Content of README.md
→ Content of package.json
→ First build log
→ Modified file content
→ Second build log
→ A large chunk of failure stack trace
→ New requirements added by the user
If the complete history is sent to the model in its original form every time, two problems arise:
- The input gets longer and longer, increasing cost and latency;
- One large log might push the truly important new question out of the context.
The crudest approach is to keep only the last 20 messages, but that's also unreliable: the 21st message might just happen to hold a key decision, and one tool output could be longer than 20 user messages combined.
So, we separate "what to save" from "what to send this round":
Session.history
Saves the complete history for traceability
Working Memory
Summary of earlier messages + original text of recent messages
Only used as input for the current request
This is context compression. It doesn't delete the original history in the Session, only changes what the model sees for this round.
Write the Model Context Limit into the Configuration
Different models can hold different numbers of tokens, so the limit cannot be hardcoded in the Compactor. First, add contextWindow to the provider.json in the project root:
{
"baseURL": "https://api.example.com/v1",
"apiKey": "replace-with-your-api-key",
"model": "your-model-name",
"contextWindow": 128000
}
contextWindow should be filled with the actual context limit of the current model, based on the model service provider's documentation. When switching models, just modify this together.
Next, modify the configuration type in src/config.ts:
export interface ProviderConfig {
baseURL: string;
apiKey: string;
model: string;
contextWindow: number;
}
Add validation for contextWindow in loadConfig():
if (
!config.baseURL ||
!config.apiKey ||
!config.model ||
typeof config.contextWindow !== "number" ||
!Number.isInteger(config.contextWindow) ||
config.contextWindow <= 0
) {
throw new Error(
"provider.json configuration is incomplete, must provide baseURL, apiKey, model, and a positive integer contextWindow.",
);
}
Finally, include it in the return value:
return {
baseURL: config.baseURL,
apiKey: config.apiKey,
model: config.model,
contextWindow: config.contextWindow,
};
Also, Let ChatClient Return Token Usage
Each time the model responds, the API typically returns the actual number of tokens used for this request in usage.total_tokens. Rather than guessing entirely, it's better to prioritize using this real number; only for new messages added after the last request should we continue using rough estimates.
The previous ChatClient used complete(). After calling the LLM API, it only returned the model's reply message to the Agent, discarding the token usage returned by the API.
Now the Compactor needs to judge whether to compress based on context usage, so we change complete() to completeWithUsage(). The new method still only requests the LLM once, just changing the return value from a single message to:
message Model's reply
totalTokens Actual number of tokens used for this request
From Step 3 onwards, all model requests will uniformly use this new method. Replace src/chat.ts with the following complete version:
import OpenAI from "openai";
import type { ProviderConfig } from "./config.ts";
export interface CompletionResult {
message:
| OpenAI.Chat.Completions.ChatCompletionMessage
| undefined;
totalTokens: number | undefined;
}
export class ChatClient {
private readonly client: OpenAI;
constructor(private readonly config: ProviderConfig) {
this.client = new OpenAI({
apiKey: config.apiKey,
baseURL: config.baseURL,
});
}
getContextWindow(): number {
return this.config.contextWindow;
}
async completeWithUsage(
messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[],
tools: OpenAI.Chat.Completions.ChatCompletionTool[],
): Promise<CompletionResult> {
const response = await this.client.chat.completions.create({
model: this.config.model,
messages,
tools,
});
return {
message: response.choices[0]?.message,
totalTokens: response.usage?.total_tokens,
};
}
}
completeWithUsage() essentially still calls the LLM API only once, just returning the previously discarded usage.total_tokens together with the model message, without making an extra request.
It will be called every time the Agent Loop requests the model. When we later connect the Compactor into agent.ts, the original:
const message = await this.client.complete(messages, tools);
will be replaced with:
const completion = await this.client.completeWithUsage(
messages,
tools,
);
const message = completion.message;
The complete flow is:
Agent prepares messages for this round
↓
completeWithUsage() requests the model
↓
Model returns message + usage
↓
message is written to chat history
usage is written to Session's count snapshot
↓
Next round, Compactor uses it to decide if compression is needed
The Compactor also calls completeWithUsage() when generating summaries, but it only takes the returned message and doesn't save totalTokens. This is because that usage belongs to the temporary "generate summary" request, not the main session's context usage.
So, from Step 3 onwards, complete() is no longer needed; all model requests uniformly go through completeWithUsage().
Add Compression State to Session
Now reopen src/context/session.ts and replace the version from Step 1 with the following complete version:
import type OpenAI from "openai";
// Message types in a session
export type Message = OpenAI.Chat.Completions.ChatCompletionMessageParam;
// Compression state in a session
// Used to record the step number up to which compression has occurred and the summary after compression
export interface CompactionState {
summary: string;
compactedUntil: number;
}
// Token usage returned by the most recent model request
export interface UsageSnapshot {
totalTokens: number;
historyLength: number;
}
// Session class
export class Session {
readonly createdAt = new Date(); // Creation time
updatedAt = new Date(); // Update time
private readonly history: Message[] = []; // Session history
private compaction?: CompactionState; // Compression state
private usage?: UsageSnapshot; // Most recent token usage
/**
* Add messages to the session history
* @param messages Messages to append
*/
append(...messages: Message[]): void {
this.history.push(...messages);
this.updatedAt = new Date();
}
/**
* Get the session history
* @returns Session history
*/
getHistory(): Message[] {
return structuredClone(this.history);
}
/**
* Get the compression state
* @returns Compression state
*/
getCompaction(): CompactionState | undefined {
return this.compaction === undefined ? undefined : { ...this.compaction };
}
getUsage(): UsageSnapshot | undefined {
return this.usage === undefined ? undefined : { ...this.usage };
}
saveUsage(totalTokens: number): void {
this.usage = {
totalTokens,
historyLength: this.history.length,
};
this.updatedAt = new Date();
}
/**
* Save the compression state
* @param summary Summary after compression
* @param compactedUntil The step number up to which compression has occurred
*/
saveCompaction(summary: string, compactedUntil: number): void {
this.compaction = { summary, compactedUntil };
// After compression, the context has changed, old usage can no longer be used.
this.usage = undefined;
this.updatedAt = new Date();
}
}
The Session now saves two pieces of state, but they are responsible for different things:
usage Estimates the length of the current context, decides whether to compress
compaction Records the result of the last compression, decides where to continue compressing from this time
That is, the program will first use usage to judge "is the budget exceeded?". Only when compression is needed will it read compaction to continue processing history not yet covered by a summary.
First, look at usage. It's a count snapshot left by the most recent model request:
totalTokens The actual total number of tokens returned by the API
historyLength Which message in the history this token count covers up to
For example, after the model answers, history has a total of 5 messages, and the API returns totalTokens = 3200. The Session saves:
{ totalTokens: 3200, historyLength: 5 }
This means the first 5 messages already have real token data. Later, the 6th and 7th messages are added, but the model hasn't been requested again, so the API naturally hasn't counted them yet. At this point, the program doesn't need to re-estimate the entire history; it just calculates:
Current context size = 3200 + estimated tokens of the 6th and 7th messages
If the budget isn't exceeded, continue using the current messages; if it is, enter compression. This is where compaction comes into play:
summary The already compressed summary of the earlier conversation
compactedUntil The next index in history to continue processing from
Let's continue this process. Suppose the conversation grows to 10 messages, indices 0 to 9, and the token calculation just found the context is too long. The first compression will summarize the earlier messages 0 to 5, keeping the recent messages 6 to 9 in their original form:
history
[0 1 2 3 4 5] [6 7 8 9]
└── becomes summary ─┘ └─ still kept as original ─┘
summary = Summary of messages 0~5
compactedUntil = 6
compactedUntil = 6 means indices 0 to 5 have already been processed, and next time it should continue from history[6]. The code skips the already compressed part using this line:
const pending = history.slice(compactedUntil);
Compression changes what is actually sent to the model next time, so the old token snapshot is no longer accurate. Thus, saveCompaction() first clears the old usage. After the next model response, the API will return the real token count for the new working memory, and the Session saves a new usage.
Later, the conversation continues to grow and exceeds the budget again. Suppose this time messages 6 and 7 need to be folded into the summary. There's no need to re-summarize messages 0 to 5; just continue updating based on the previous summary:
New summary = Previous summary + messages 6, 7
Recent original = messages 8, 9
compactedUntil = 8
The whole process can be summarized as: usage is responsible for judging "when to compress", summary saves "what was discussed earlier", and compactedUntil records "where compression has reached". Working together, the program can both judge if the context is too long and continue from where it left off during the next compression.
Create src/context/compactor.ts
Compactor is the context organizer used before the Agent requests the model.
import { ChatClient } from "../chat.ts";
import type { Message, Session } from "./session.ts";
const SUMMARY_PROMPT = `
You are responsible for compressing the earlier conversation of an AI programming task.
Please only retain the information truly necessary for completing the subsequent task, and strictly use the following structure:
## User Goal
## Constraints and Key Decisions
## Completed
## Current Issues
## Files Read or Modified
## Next Steps
Do not fabricate conclusions that did not appear in the conversation.
`;
export class Compactor {
constructor(
private readonly client: ChatClient,
private readonly maxContextTokens: number,
private readonly reserveTokens = 16_384,
private readonly keepRecentTokens = 20_000,
) {}
async buildWorkingMemory(session: Session): Promise<Message[]> {
const history = session.getHistory();
const state = session.getCompaction();
const start = state?.compactedUntil ?? 0;
const pending = history.slice(start);
const current = this.withSummary(state?.summary, pending);
const usableTokens = this.maxContextTokens - this.reserveTokens;
const contextTokens = estimateContextTokens(
session,
history,
current,
);
if (contextTokens <= usableTokens) {
return current;
}
const cutIndex = this.findCutIndex(pending);
// If no safe cut point is found, don't generate a summary for now, just truncate overly long tool outputs.
if (cutIndex <= 0) {
return this.withSummary(
state?.summary,
truncateLargeToolOutputs(pending),
);
}
console.log(
`Context approaching limit (approx ${contextTokens} / ${usableTokens} tokens), compressing earlier messages...`,
);
const older = truncateLargeToolOutputs(pending.slice(0, cutIndex));
const recent = truncateLargeToolOutputs(pending.slice(cutIndex));
const summaryInput = [
state?.summary ? `# Previous Summary\n${state.summary}` : "",
`# New History\n${JSON.stringify(older, null, 2)}`,
]
.filter(Boolean)
.join("\n\n");
// The summary request only takes the message, not saving usage to the main Session.
const completion = await this.client.completeWithUsage(
[
{ role: "system", content: SUMMARY_PROMPT },
{ role: "user", content: summaryInput },
],
[],
);
const summary = completion.message?.content?.trim() ?? "";
if (!summary) {
return this.withSummary(
state?.summary,
truncateLargeToolOutputs(pending),
);
}
session.saveCompaction(summary, start + cutIndex);
console.log("✓ Context compression complete, recent messages kept in original form.\n");
return this.withSummary(summary, recent);
}
private findCutIndex(messages: Message[]): number {
let recentTokens = 0;
for (let index = messages.length - 1; index >= 0; index -= 1) {
recentTokens += estimateMessageTokens(messages[index]);
if (recentTokens < this.keepRecentTokens) continue;
// Start keeping from a regular user message to avoid splitting assistant tool calls and tool results.
for (let cut = index; cut < messages.length; cut += 1) {
if (messages[cut].role === "user") {
return cut;
}
}
}
return 0;
}
private withSummary(
summary: string | undefined,
messages: Message[],
): Message[] {
if (!summary) return messages;
const summaryMessage: Message = {
role: "user",
content: `[System-generated summary of earlier session]\n${summary}`,
};
return [summaryMessage, ...messages];
}
}
function estimateContextTokens(
session: Session,
history: Message[],
current: Message[],
): number {
const usage = session.getUsage();
// When there's no real usage yet, estimate the current working memory.
if (usage === undefined) {
return estimateTokens(current);
}
// usage covers the messages before it, only estimate the newly added part after it.
const trailingMessages = history.slice(usage.historyLength);
return usage.totalTokens + estimateTokens(trailingMessages);
}
function estimateTokens(messages: Message[]): number {
return messages.reduce(
(total, message) => total + estimateMessageTokens(message),
0,
);
}
function estimateMessageTokens(message: Message): number {
const bytes = Buffer.byteLength(JSON.stringify(message), "utf8");
// Only used for estimating new messages not yet covered by API usage.
return Math.ceil(bytes / 4);
}
function truncateLargeToolOutputs(messages: Message[]): Message[] {
return messages.map((message) => {
if (message.role !== "tool" || typeof message.content !== "string") {
return message;
}
const content = message.content;
if (content.length <= 8_000) {
return message;
}
return {
...message,
content:
`${content.slice(0, 4_000)}\n\n` +
"...[Middle tool output truncated]...\n\n" +
content.slice(-4_000),
};
});
}
The complete code is placed here first for easy copying. Don't try to understand the entire implementation at once; we'll break it down step by step according to its execution order.
First, look at the ChatClient, Session, and message types imported at the top of the file:
import { ChatClient } from "../chat.ts";
import type { Message, Session } from "./session.ts";
ChatClient is used to request the model to generate summaries, and Session is used to read the complete history and save compression progress.
Next, look at the summary prompt:
const SUMMARY_PROMPT = `
You are responsible for compressing the earlier conversation of an AI programming task.
Please only retain the information truly necessary for completing the subsequent task, and strictly use the following structure:
## User Goal
## Constraints and Key Decisions
## Completed
## Current Issues
## Files Read or Modified
## Next Steps
Do not fabricate conclusions that did not appear in the conversation.
`;
This prompt specifies the content the summary must retain. This way, what's generated isn't a vague summary, but a handover record that allows the task to continue.
Now look at the Compactor's constructor:
export class Compactor {
constructor(
private readonly client: ChatClient,
private readonly maxContextTokens: number,
private readonly reserveTokens = 16_384,
private readonly keepRecentTokens = 20_000,
) {}
The three parameters here control: the model's context limit, the space reserved for the next response, and how many recent original tokens to keep during compression. maxContextTokens will be passed in from provider.json.contextWindow later.
16_384 and 20_000 use Pi Agent's current defaults:
reserveTokens = 16_384 Reserve space for the model's response in this round
keepRecentTokens = 20_000 Keep approximately the last 20K tokens in original form
Why not turn all history into a summary, but specifically keep a recent segment in original form? On one hand, in programming tasks, recent messages usually contain the details the model currently needs most, such as requirements just added by the user, file content just read, command results just executed, and unfinished tool calls. A summary can retain conclusions but inevitably loses specific code and parameters.
On the other hand, this also relates to the common positional bias in long-context models. "Lost in the Middle: How Language Models Use Long Contexts" found that models generally perform better when key information is at the beginning or end of the context; when located in the middle of a very long context, utilization effectiveness can significantly drop. This phenomenon is often called the "primacy and recency effect," with overall performance resembling a U-shaped curve, rather than simply "the later the content, the more the model values it."
Therefore, keeping recent original text first serves to preserve the precise details of the current task; simultaneously, the following arrangement also aligns with the positional bias observed in the paper:
System prompt
Structured summary of earlier content Placed at the front, retaining goals and key decisions
Original text of the most recent conversation segment Placed at the back, retaining precise details of the current task
It should be noted that the paper only explains "why it's worth keeping recent original text," without proving that exactly 20K tokens must be kept. 20_000 is an engineering default adopted by Pi Agent, a compromise between quality, cost, and context space, not a theoretical constant calculated from the paper.
They are not fixed standards that must be used for all models, just a relatively safe starting point for the 128K context model configured in this article. If switching to a model with a smaller context later, these values should be adjusted downwards accordingly.
The easiest point to misunderstand here is: reserveTokens = 16_384 does not require the model to "compress the summary to 16K," nor does the code set a hard 16K limit on output. It just makes the program trigger compression earlier, preventing the input messages from occupying the entire context window.
Taking a 128K context as an example:
Usable input budget = 128_000 - 16_384 = 111_616
When the working memory exceeds about 111.6K tokens, the program starts compressing. This way, the compressed normal request still has a block of space left for the model's response. The current code doesn't pass max_tokens, so the actual maximum output is still determined by the model and service provider.
After compression is triggered, keepRecentTokens = 20_000 will keep approximately the last 20K tokens in original form, with the rest of the earlier content handed to the model to generate a summary. If compression has occurred before, the summary request won't re-carry those earlier original messages but will use:
Previous summary + newly added earlier messages since the last compression
After the model updates the summary, the actual working memory used for the normal response in this round is:
System prompt + new summary + approximately the last 20K tokens of original text
The "20K tokens" here is just a target value, not guaranteed to be exactly 20_000.
What's referred to here as "one round of conversation" isn't simply one question and one answer. It starts from one user message, can include multiple tool calls and tool results in between, until the model gives a final answer; the next user message marks the start of a new round.
For example, when the user asks the model to read a file, one complete round of conversation might contain several messages:
user Please read config.ts
assistant Calls the read file tool
tool Returns file content
assistant Answers based on file content
These four messages belong to the same round of conversation. If the cut happens right between assistant and tool, the model can only see "a tool was called" but not the content returned by the tool, making this record incomplete.
So the core principle here is: prioritize retaining the complete information of one round of conversation, rather than forcibly reaching exactly 20K tokens. 20_000 is just a rough target; to avoid splitting user questions, tool calls, and tool results, the code will appropriately adjust the dividing line, so the final retained recent original text might be slightly more or slightly less than 20K tokens.
The entry point for the entire module is buildWorkingMemory(). It first reads the Session and judges whether the current context has exceeded the budget:
async buildWorkingMemory(session: Session): Promise<Message[]> {
const history = session.getHistory();
const state = session.getCompaction();
const start = state?.compactedUntil ?? 0;
const pending = history.slice(start);
const current = this.withSummary(state?.summary, pending);
const usableTokens = this.maxContextTokens - this.reserveTokens;
const contextTokens = estimateContextTokens(
session,
history,
current,
);
if (contextTokens <= usableTokens) {
return current;
}
Here, current is the "working memory" prepared to be sent to the model for this round. If compression has occurred before, it consists of "previous summary + recent messages not yet compressed"; if not, it's the complete history.
reserveTokens must be deducted from the context limit in advance, because the entire window cannot be stuffed with history; space must be left for the model's response in this round. When the budget isn't exceeded, the method returns directly here without generating a summary.
If the budget is exceeded, buildWorkingMemory() will truly enter the summary process. The so-called "local compression" here doesn't mean local code understands and rewrites the chat content itself. The local code is only responsible for three things: estimating size, splitting old and new messages, and calling the LLM. The actual summary text is still generated by the LLM.
Taking a 128K context just reaching the critical point as an example, the entire process roughly is:
Current working memory approx 112K
↓ Compression triggered
Local program keeps approx last 20K original text
↓
Remaining approx 92K earlier content
↓ First model call
Model generates a summary of the earlier content
↓
Local program composes "new summary + approx last 20K original text"
↓ Second model call
Model answers the user based on the new working memory
The 92K here is just an approximate value for ease of understanding, coming from 128K - 16K - 20K. The actual split also considers the previous summary, system prompt, and complete conversation boundaries, so it won't exactly equal this number.
If compression has occurred before, what the first model call receives isn't those already compressed original messages either, but "previous summary + the earlier messages newly designated for this round." The model will generate an updated summary based on the old summary.
Therefore, only the round that triggers compression typically calls the model twice. When compression isn't triggered, buildWorkingMemory() directly returns the current messages, and the Agent only calls the model once for the normal response.
Only when the budget is exceeded does compression need to be considered. However, before looking at the code below, three actions must be distinguished:
Estimate Roughly calculate message size using "UTF-8 bytes ÷ 4", does not modify messages
Split Divide messages into "earlier content" and "recent content" groups, also does not modify messages
Compress Have the model generate a shorter summary of the earlier content
Why is "splitting" needed? Because we don't want to turn all history into a summary. The most recent conversation contains the code currently being processed, tool results, and user requirements; keeping the original text is more accurate. Only earlier content is suitable for becoming a summary.
cutIndex is the array index between these two groups of messages. For example:
pending = [msg0, msg1, msg2, msg3, msg4, msg5]
cutIndex = 4
Earlier content = pending.slice(0, 4) = [msg0, msg1, msg2, msg3]
Recent original = pending.slice(4) = [msg4, msg5]
It's not a token count, nor does it truncate message content; it just tells the program "from which message to start keeping the original text." Now, write the code for finding the dividing line:
const cutIndex = this.findCutIndex(pending);
// If no safe cut point is found, don't generate a summary for now, just truncate overly long tool outputs.
if (cutIndex <= 0) {
return this.withSummary(
state?.summary,
truncateLargeToolOutputs(pending),
);
}
findCutIndex() returning 0 means no position was found that can both retain recent original text and avoid splitting a complete conversation. For example, there might currently be only one very long round of tool calls, with no next user message to serve as a dividing line.
In this situation, the current version doesn't generate a summary for now, only calling truncateLargeToolOutputs() to check tool results. If a tool result exceeds 8,000 characters, only the beginning and end are kept. This "truncating tool output" is not the same as the earlier token estimation: token estimation only calculates size, whereas here the message content is actually shortened.
After finding a safe dividing line, first tell the user in the terminal that compression has started:
console.log(
`Context approaching limit (approx ${contextTokens} / ${usableTokens} tokens), compressing earlier messages...`,
);
This log only appears when actually preparing to call the model to generate a summary. If it's just a normal conversation, or there's no safe cut position currently, it won't falsely report "compressing."
After finding the cut position, first truly split the messages into two groups:
const older = truncateLargeToolOutputs(pending.slice(0, cutIndex));
const recent = truncateLargeToolOutputs(pending.slice(cutIndex));
Now the two variables respectively hold:
older Earlier messages before the dividing line, next to be summarized
recent Recent messages after the dividing line, continue to keep in original form
Both groups of messages go through truncateLargeToolOutputs(), just to prevent any single tool result from being exceptionally long. It does not delete the original history in the Session.
Next, assemble the content to be given to the summary model:
const summaryInput = [
state?.summary ? `# Previous Summary\n${state.summary}` : "",
`# New History\n${JSON.stringify(older, null, 2)}`,
]
.filter(Boolean)
.join("\n\n");
During the first compression, there's no state.summary yet, so the input only contains the batch of earlier messages older. During the second compression, the input will simultaneously contain:
# Previous Summary
Even earlier content already compressed
# New History
The older messages prepared to be added to the summary this time
JSON.stringify(older, null, 2) just converts the message array into text the model can read. .filter(Boolean) removes the empty "previous summary" during the first compression, and .join("\n\n") connects the remaining parts with two newlines.
After preparing summaryInput, call the model to generate the new summary:
// The summary request only takes the message, not saving usage to the main Session.
const completion = await this.client.completeWithUsage(
[
{ role: "system", content: SUMMARY_PROMPT },
{ role: "user", content: summaryInput },
],
[],
);
The system message here tells the model what information the summary must retain, and the user message puts in the real history just assembled. The last parameter is passed [], meaning this time the model is only needed to summarize text, not allowed to call file or command tools.
The token usage returned by this request belongs to the temporary "generate summary" request and cannot be treated as the main session's context usage, so only message is taken here.
Next, check if the model successfully returned a summary:
const summary = completion.message?.content?.trim() ?? "";
if (!summary) {
return this.withSummary(
state?.summary,
truncateLargeToolOutputs(pending),
);
}
If summary is an empty string, it means this compression failed. At this point, the compression progress is not updated, and the original summary and all uncompressed messages continue to be used, avoiding accidentally discarding history.
After the summary succeeds, save the result and return the new working memory:
session.saveCompaction(summary, start + cutIndex);
console.log("✓ Context compression complete, recent messages kept in original form.\n");
return this.withSummary(summary, recent);
}
Why save start + cutIndex here, instead of just cutIndex? Because cutIndex is relative to pending, while start is the starting point of pending within the complete history.
For example, last time compression already reached history[6], so start = 6. This time, the first 2 items from pending are compressed, meaning cutIndex = 2, so the new position is:
compactedUntil = 6 + 2 = 8
Finally, withSummary(summary, recent) returns "new summary + recent original text." The complete history in the Session is still not deleted; only the working memory actually sent to the model for this round is changed.
Next, look at how findCutIndex() decides where to cut the history:
private findCutIndex(messages: Message[]): number {
let recentTokens = 0;
for (let index = messages.length - 1; index >= 0; index -= 1) {
recentTokens += estimateMessageTokens(messages[index]);
// Less than 20K, continue
if (recentTokens < this.keepRecentTokens) continue;
// Start keeping from a regular user message to avoid splitting assistant tool calls and tool results.
for (let cut = index; cut < messages.length; cut += 1) {
if (messages[cut].role === "user") {
return cut;
}
}
}
return 0;
}
It first calculates backwards from the end, finding the approximate position corresponding to "about the last 20K tokens"; then, from this position, it searches forward for the first user message, using it as the starting point for the recent original text.
This is done so that the retained content starts from a new round of user questioning. For example:
Earlier content, enters summary
─────────────────────────────┐
user Please read config.ts │
assistant Calls read tool │ Same round of conversation, entirely enters summary
tool Returns file content │
assistant Answers based on content │
─────────────────────────────┘
Recent content, kept as original
─────────────────────────────┐
user Please continue modifying config │ Starts from the next round of user questioning
assistant ... │
─────────────────────────────┘
If cut directly in the middle based on token count, the recent messages might only have a tool result left, without the corresponding preceding assistant tool call; or only the tool call is kept, but the tool result is lost. Not only would the model struggle to understand what happened in the next round, but the OpenAI-compatible interface might also directly throw an error due to incomplete tool message sequence.
So the core of findCutIndex() is not to pursue exactly keeping 20K tokens, but to find the next user message, keeping tool calls, tool results, and final answers on the same side, ensuring the semantic and message structure of one round of conversation is complete.
Now look at how withSummary() assembles the final working memory:
private withSummary(
summary: string | undefined,
messages: Message[],
): Message[] {
if (!summary) return messages;
const summaryMessage: Message = {
role: "user",
content: `[System-generated summary of earlier session]\n${summary}`,
};
return [summaryMessage, ...messages];
}
}
This method wraps the summary into a message and places it before the recent original text. What the Agent ultimately gets is "summary of earlier history + recent message original text."
The several functions after the Compactor class are responsible for calculating tokens. First, look at how it prioritizes using the real usage returned by the API:
function estimateContextTokens(
session: Session,
history: Message[],
current: Message[],
): number {
const usage = session.getUsage();
// When there's no real usage yet, estimate the current working memory.
if (usage === undefined) {
return estimateTokens(current);
}
// usage covers the messages before it, only estimate the newly added part after it.
const trailingMessages = history.slice(usage.historyLength);
return usage.totalTokens + estimateTokens(trailingMessages);
}
When usage exists, only the newly added messages after it are estimated; when there's no real usage yet, the entire current working memory is estimated. This is the purpose of saving historyLength earlier.
The specific estimation method is as follows:
function estimateTokens(messages: Message[]): number {
return messages.reduce(
(total, message) => total + estimateMessageTokens(message),
0,
);
}
function estimateMessageTokens(message: Message): number {
const bytes = Buffer.byteLength(JSON.stringify(message), "utf8");
// Only used for estimating new messages not yet covered by API usage.
return Math.ceil(bytes / 4);
}
Here, the message is converted to JSON, and tokens are roughly converted based on the UTF-8 byte count. It's only responsible for estimating new messages not yet counted by the API, not aiming for perfect consistency with every model's tokenizer.
Finally, look at how exceptionally long tool outputs are handled:
function truncateLargeToolOutputs(messages: Message[]): Message[] {
return messages.map((message) => {
if (message.role !== "tool" || typeof message.content !== "string") {
return message;
}
const content = message.content;
if (content.length <= 8_000) {
return message;
}
return {
...message,
content:
`${content.slice(0, 4_000)}\n\n` +
"...[Middle tool output truncated]...\n\n" +
content.slice(-4_000),
};
});
}
When a tool reads a file or executes a command, a single result might be longer than the entire chat. Here, the beginning and end of the output are kept, omitting the middle part, to prevent one tool message from instantly filling the context.
At this point, the execution process of Compactor has been broken down. It does not modify the complete history saved by the Session, only responsible for generating a shorter working memory before each request.
This step draws on Pi Agent's hybrid counting approach: prioritize trusting the usage returned by the model API, only estimating the new messages after the usage. This way, there's no need to introduce different tokenizers for each model, and it's more reliable than re-estimating the entire history from scratch.
maxContextTokens now comes from provider.json.contextWindow, no longer hardcoded. Dividing the byte count by 4 after JSON.stringify is just a cheap approximation used for newly added messages.
Connect Compactor into src/agent.ts
The previous Compactor can already generate working memory. Now directly replace src/agent.ts with the following complete code:
import type OpenAI from "openai";
import { ChatClient } from "./chat.ts";
import { Compactor } from "./context/compactor.ts";
import type { Session } from "./context/session.ts";
import { Registry } from "./tools/registry.ts";
const MAX_STEPS = 8;
export class Agent {
private readonly compactor: Compactor;
constructor(
private readonly client: ChatClient,
private readonly registry: Registry,
private readonly session: Session,
) {
this.compactor = new Compactor(
client,
client.getContextWindow(),
);
}
async run(prompt: string): Promise<string> {
this.session.append({
role: "user",
content: prompt,
});
for (let step = 1; step <= MAX_STEPS; step += 1) {
const memory = await this.compactor.buildWorkingMemory(this.session);
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{
role: "system",
content:
"You are power-code, a development assistant. Please prioritize reading real files; actively run commands to verify results after modifications. Please answer in Chinese.",
},
...memory,
];
const completion = await this.client.completeWithUsage(
messages,
this.registry.getDefinitions(),
);
const message = completion.message;
if (!message) {
throw new Error("The model did not return a message.");
}
this.session.append(message);
if (
completion.totalTokens !== undefined &&
completion.totalTokens > 0
) {
this.session.saveUsage(completion.totalTokens);
}
const toolCalls = message.tool_calls ?? [];
if (toolCalls.length === 0) {
return message.content ?? "The model did not return text content.";
}
for (const toolCall of toolCalls) {
if (toolCall.type !== "function") {
throw new Error(`Tool type not currently supported: ${toolCall.type}`);
}
const name = toolCall.function.name;
console.log(`Round ${step}: AI calls ${name}`);
let result: string;
try {
result = await this.registry.execute(
name,
toolCall.function.arguments,
);
console.log(`✓ ${name} execution complete\n`);
} catch (error) {
const reason = error instanceof Error ? error.message : String(error);
result = `Tool execution failed: ${reason}`;
console.log(`✗ ${result}\n`);
}
this.session.append({
role: "tool",
tool_call_id: toolCall.id,
content: result,
});
}
}
throw new Error(`Execution exceeded ${MAX_STEPS} rounds, stopped.`);
}
}
The complete code is placed here first for direct replacement. Below, we'll step-by-step see what changed in this version of Agent compared to the previous one; the tool execution part is unchanged, the focus is on how working memory and token usage are connected into the Agent Loop.
First, look at the new import:
import { Compactor } from "./context/compactor.ts";
Agent also needs to hold a Compactor instance:
private readonly compactor: Compactor;
The constructor creates it using the current model's context limit:
constructor(
private readonly client: ChatClient,
private readonly registry: Registry,
private readonly session: Session,
) {
this.compactor = new Compactor(
client,
client.getContextWindow(),
);
}
Entering the Agent Loop, originally it directly read the Session's complete history:
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{
role: "system",
content: "...",
},
...this.session.getHistory(),
];
Now it's changed to first let the Compactor build the working memory for this round:
const memory = await this.compactor.buildWorkingMemory(this.session);
const messages: OpenAI.Chat.Completions.ChatCompletionMessageParam[] = [
{
role: "system",
content:
"You are power-code, a development assistant. Please prioritize reading real files; actively run commands to verify results after modifications. Please answer in Chinese.",
},
...memory,
];
The model call must also switch to completeWithUsage(), and after saving the reply, record this real usage:
const completion = await this.client.completeWithUsage(
messages,
this.registry.getDefinitions(),
);
const message = completion.message;
if (!message) {
throw new Error("The model did not return a message.");
}
this.session.append(message);
if (
completion.totalTokens !== undefined &&
completion.totalTokens > 0
) {
this.session.saveUsage(completion.totalTokens);
}
Here, it's crucial to append(message) first, then saveUsage(). Because historyLength needs to record which message this usage has covered up to.
Tool results are still written to the Session as usual:
this.session.append({
role: "tool",
tool_call_id: toolCall.id,
content: result,
});
They determine what content is actually in the Session for the next compression.
Now build:
npm run build
If the build passes, the logical relationship becomes:
Session saves complete history
↓
Compactor selects working memory
↓
Agent adds system prompt and requests the model
↓
Model reply and real usage written back to Session
↓
New tool results temporarily estimate tokens
How to Observe Context Compression
Normal small tasks might not grow long enough to trigger the Compactor. If readers are only asked to manually chat for dozens of rounds in the terminal, it's not only troublesome, but the timing of triggering compression would differ for everyone.
Here, a dedicated demo script is added. It will automatically create a large file, split the file content into multiple complete rounds of messages placed into the Session, and then trigger a real summary request. After the summary is complete, it will also call the model one more time, demonstrating how the Agent uses "summary + recent original text" to continue answering.
Create src/demo-compaction.ts
import { mkdir, readFile, writeFile } from "node:fs/promises";
import { resolve } from "node:path";
import { ChatClient } from "./chat.ts";
import { loadConfig } from "./config.ts";
import { Compactor } from "./context/compactor.ts";
import { Session } from "./context/session.ts";
const DEMO_DIR = resolve("workspace/context-demo");
const DEMO_FILE = resolve(DEMO_DIR, "large-context.txt");
const BEFORE_FILE = resolve(DEMO_DIR, "context-before.json");
const SUMMARY_FILE = resolve(DEMO_DIR, "compressed-summary.md");
const AFTER_FILE = resolve(DEMO_DIR, "context-after.json");
function formatBytes(bytes: number): string {
return `${(bytes / 1024).toFixed(1)} KB`;
}
async function main() {
const config = await loadConfig();
const client = new ChatClient(config);
const session = new Session();
const blocks = Array.from({ length: 14 }, (_, index) => {
const title = `# Demo Block ${index + 1}`;
const content = `context demo block ${index + 1} `.repeat(300);
return `${title}\n${content}`;
});
await mkdir(DEMO_DIR, { recursive: true });
await writeFile(DEMO_FILE, blocks.join("\n\n"), "utf8");
const largeContent = await readFile(DEMO_FILE, "utf8");
const sections = largeContent.split("\n\n");
for (const [index, section] of sections.entries()) {
const toolCallId = `demo-read-${index + 1}`;
session.append(
{
role: "user",
content: `Please read section ${index + 1} of the demo file.`,
},
{
role: "assistant",
content: null,
tool_calls: [
{
id: toolCallId,
type: "function",
function: {
name: "read_file",
arguments: JSON.stringify({ path: DEMO_FILE }),
},
},
],
},
{
role: "tool",
tool_call_id: toolCallId,
content: section,
},
{
role: "assistant",
content: `Section ${index + 1} has been read.`,
},
);
}
session.append({
role: "user",
content: "Please summarize in one sentence what the demo file just read mainly contains.",
});
console.log(`Demo file created: ${DEMO_FILE}`);
console.log(`Constructed ${sections.length} rounds of historical messages.\n`);
const beforeContext = JSON.stringify(session.getHistory(), null, 2);
await writeFile(BEFORE_FILE, beforeContext, "utf8");
// A smaller demo budget is used here to ensure compression triggers stably.
// This does not modify the model's real contextWindow in provider.json.
const compactor = new Compactor(
client,
16_000,
4_000,
4_000,
);
const memory = await compactor.buildWorkingMemory(session);
const state = session.getCompaction();
if (!state?.summary) {
throw new Error("Context compression was not triggered, please increase the demo file content and retry.");
}
const afterContext = JSON.stringify(memory, null, 2);
await Promise.all([
writeFile(SUMMARY_FILE, state.summary, "utf8"),
writeFile(AFTER_FILE, afterContext, "utf8"),
]);
console.log(`Complete history: ${session.getHistory().length} messages`);
console.log(`Working memory for this round: ${memory.length} messages`);
console.log(`Compressed up to history[${state.compactedUntil}].\n`);
console.log("Files before and after compression:");
console.log(
`Before compression ${formatBytes(Buffer.byteLength(beforeContext, "utf8"))} ${BEFORE_FILE}`,
);
console.log(
`Summary ${formatBytes(Buffer.byteLength(state.summary, "utf8"))} ${SUMMARY_FILE}`,
);
console.log(
`After compression ${formatBytes(Buffer.byteLength(afterContext, "utf8"))} ${AFTER_FILE}\n`,
);
const completion = await client.completeWithUsage(
[
{
role: "system",
content: "You are an assistant used to verify context compression, please answer briefly in Chinese.",
},
...memory,
],
[],
);
console.log(`AI：${completion.message?.content ?? "No content returned"}`);
}
main().catch((error: unknown) => {
const message = error instanceof Error ? error.message : String(error);
console.error(`Demo failed: ${message}`);
process.exit(1);
});
This script doesn't have the Agent actually execute read_file 14 times. Instead, it locally splits a large file into 14 rounds of already completed tool call records, then hands them to the real Session and Compactor. This way, it retains the complete structure of "user question → assistant calls tool → tool returns → assistant answers," while only needing two real model requests: one to generate the summary, and one to use the summary to continue answering.
To allow the before-and-after changes to be directly opened and viewed, the script will also generate in workspace/context-demo:
large-context.txt Original demo large file, not modified by Compactor
context-before.json Complete Session messages before compression
compressed-summary.md Summary generated by the LLM
context-after.json Summary + recent original text, i.e., the working memory after compression
What should truly be compared are context-before.json and context-after.json, because the Compactor compresses the chat context, not the source file large-context.txt. compressed-summary.md is convenient for separately checking what key information the model actually retained.
These files are only additionally exported by the demo script for ease of observation. During normal powercode operation, the Compactor still only saves the summary in Session memory and does not automatically write debug files into the user's project.
The demo parameters 16_000 / 4_000 / 4_000 are only used to make the effect easier to appear, without modifying provider.json or impersonating the model's real context limit. When running powercode officially, the Agent still uses the model's real contextWindow and the Compactor's official defaults.
Now build and run:
npm run build
node dist/demo-compaction.js
The terminal will see output similar to the following:
The specific token count, message count, and summary position might vary slightly, but as long as you see "compressing" and "compression complete," it means the summary process has actually executed.
The files generated by the script are only located in workspace/context-demo and will not modify the source code of the practice project. After the demo is complete, you can directly delete this demo directory when it's no longer needed.
When Does Compression Occur During Normal Operation
After the demo, return to the official configuration. The actual compression threshold is:
contextWindow in provider.json
- reserveTokens reserved 16,384
= Threshold triggering compression
For example, when contextWindow = 128000, compression is triggered when the working memory exceeds about 111616 tokens. Normal small tasks are unlikely to reach this length; this is normal. Don't deliberately fill in a smaller real model limit just to see the compression effect sooner.
After compression is triggered, the Agent will make an additional summary request without tools. Think of it as pausing a meeting midway to organize a page of handover notes, then continuing the discussion with the notes:
Earlier messages
↓ First model call
Generate structured summary
↓
Summary + approx last 20K original text
↓ Second model call
Continue answering the current question
This summary must at least retain:
What the user ultimately wants to accomplish
Which constraints cannot be violated
What has already been done
Which files have been read or modified
What problem is currently encountered
What is the next step planned
After the summary is generated, approximately the last 20K tokens are still kept in original form. This way, the model can both know the previous task background from the summary and see the recent code, tool results, and user requirements.
It needs to be emphasized again: compression only changes the working memory sent to the model for this round, and does not delete Session.history. If auditing, debugging, or restoring complete chat records is needed later, the history can continue to be written to JSONL, SQLite, or other persistent storage; this is not part of this minimal implementation.
A Few Easy Pitfalls
1. Treating Session as Permanent Memory
The current Session only exists in the memory of the Node.js process:
Process is alive: Session exists
Process exits: Session disappears
This article solves continuous conversation within the same process, not cross-process recovery. Therefore, don't see the name Session and mistakenly think it already permanently saves sessions like a database.
Cross-process recovery at least requires a persistence method, such as saving chat records, or writing truly important task state into workspace files. The next article will continue to address this issue.
2. Treating the Summary as Absolute Fact
The summary is generated by the model and might miss a number, a constraint, or even miswrite a "discussed plan" as a "decided plan." So, the summary is suitable for shortening context but should not be treated as an absolutely accurate task ledger.
This is also why the Compactor doesn't turn all messages into a summary but continues to keep a recent segment of original text. Acceptance criteria and task progress that truly cannot be lost should later be written into external files.
3. Filling contextWindow with a Uniform Default Value
Different models have different context limits. provider.json.contextWindow must be filled with the actual specification of the current model, not deliberately filled incorrectly to observe compression more easily.
If filled too large, the program might not compress in time before the request exceeds the model's limit; if filled too small, it will cause the Agent to generate summaries prematurely, increasing request count and information loss.
4. Treating Token Estimation as a Precise Result
usage.total_tokens is the real usage returned by the API, but "UTF-8 bytes ÷ 4" is only a rough estimate for newly added messages. The actual token count will differ for different models, different languages, and different message structures.
This hybrid calculation's goal is to judge "is it close to the limit" at low cost, not to implement a counter perfectly consistent with the service provider's tokenizer.
5. Thinking Truncating Tool Outputs Deletes Session History
truncateLargeToolOutputs() processes the copy returned by getHistory(), only shortening overly long tool results in the working memory for this round. The original messages saved internally by the Session still exist and are not rewritten by this function.
What We've Added to the Agent So Far
This article didn't suddenly give the model a larger context window, nor did it introduce a mysterious "memory model." We just split message management into several parts with clear responsibilities:
Session
Saves complete messages within the same process
Terminal Interaction Loop
Allows the user to continuously input and ask follow-up questions
UsageSnapshot
Records the tokens and message position actually covered by the most recent API request
Compactor
Judges when compression is needed and generates "summary + recent original text"
Working Memory
Decides the messages actually sent to the model for this round
Thus, the entire flow becomes:
User continues asking follow-up questions
↓
Messages appended to Session
↓
Compactor judges based on usage if context is approaching the limit
├─ Not reached: continue using current working memory
└─ Reached: first have the model generate a summary, then keep recent original text
↓
Agent answers using the organized working memory
The current powercode can now, like Claude Code or Codex CLI, sustain a conversation within a single launch; when the context grows long, it no longer stuffs the entire history into the model forever. More importantly, we've already separated "complete history" from "content sent to the model for this round," so when continuing to expand persistence, task planning, or different compression strategies later, there's no need to overturn the Agent Loop.
But once the process exits, the Session in memory still disappears. In the next article, we'll solve another type of problem: writing goals, plans, and execution progress into PLAN.md, TODO.md, and adding a simplified Plan Mode, so a new process can also find where the task left off.