WorkBuddy's Agent Architecture: Layered System Prompts, Three-Tier Memory, and Deferred Tools
theme: smartblue
Entering 2026, the popularity of crayfish in Chinese self-media has been explosive, which has inadvertently brought Agent into the field of vision of more enterprises.
So many companies have started developing their own Agents, whether based on ReAct Agent using LangGraph or using frameworks like PI and DSH. Everyone faces the same problem:
How should the product be designed, and how should it be implemented?
Today, we will do a comprehensive deconstruction based on WorkBuddy. The following content involves many internal designs and logic of WorkBuddy. I guarantee at least 90% accuracy;
So, if you can still read this article, read it quickly. It might be deleted someday;
First, let me reveal something you definitely don't know:
In WorkBuddy's conversation, you can ask it to use ToolSearch to find the manual of any deferred tool, and then you can get all the information about this tool, and this information is not fabricated by AI but obtained from the API side.
For example: Use ToolSearch to find the manual of ImageGen
The main text begins. The following content is for sharing and learning only:
Composition of an Agent
The composition of an Agent product mainly includes the following parts:
Agent base + System prompt + Client/Web end + Server
- Why is the system prompt so important that it is listed separately alongside the other three?
Because the system prompt can be said to be the soul of the entire Agent product. The system prompt here is not the traditional system prompt text when connecting to a model via API. The system prompt of an Agent product is an AI soul guide deeply bound to the product. It can directly affect the entire Agent's runtime logic, security protection, conversation style, tool/skill capabilities, and usage methods.
For WorkBuddy, their system prompt structure uses XML tag format + pure English, rather than Chinese Markdown format.
The choice of these two formats depends on different models. For example, GPT series prefers Markdown, and Deepseek's prompt engineering guide also states that using clear markers (such as XML tags) to separate different parts of a prompt can help the model correctly interpret each component.
According to the current system prompts of Qwen and WorkBuddy, models from Alibaba and Tencent should also use XML tags as much as possible.
Example, regarding which language to respond in:
<response_language>
当前处于中文环境,使用简体中文回答 (Speak in Chinese).
</response_language>
- What is the difference between the client and the web end?
If you pay attention to the account logic of similar products, you will notice that when you log in to your account on a new device, the new device will not have your past work records.
This is because the task records of an Agent product client do not follow the account, but the device.
Why is this?
The main reason is that the client can directly read and write local files, and related artifacts can be stored locally. This is the first difference between the client and the web end.
Although the web end can also operate local files with authorization, the implementation solution and effect are far behind the client.
Another difference is that the client's agent runs on the user's local machine, while the web end needs to put the agent on its own server, which requires additional handling of session scheduling and task orchestration.
For example, the 'Cloud Computer' provided by the web version of Doubao Work creates an independent cloud virtual environment for each user.
So for an Agent product, the client is usually prioritized.
So how to choose the tech stack?
Client: Currently, Electron is usually chosen directly. WorkBuddy's tech stack uses Electron v37, which includes Chrome v138 and Node.js v22.
Agent base: You can directly use PI, DSH as the base, or develop your own Agent base. WorkBuddy was incubated by the CodeBuddy team, and their Agent base is likely an upgraded version of CodeBuddy (after all, we haven't forgotten the Claude Code source code leak in March).
- PI officially only does the core part of the Agent; all peripheral capabilities require additional development. Although the community already has solutions for most of them, you still need to be cautious when using them in your own production environment.
- The advantage of DSH is that DeepSeek officially provides comprehensive capabilities and supports developer customization, but since the version is not yet stable, there may be major overhauls.
Server: For product selection, currently the most comprehensive support is only for Node.js and Python. Other languages can also be integrated, but it is recommended to first research the AI requirements on the server side to see if the capabilities are comprehensive.
For example, if you want to integrate DSH with Go, you must use the JSON-RPC protocol and start the DSH runtime as a subprocess in the background to run it, rather than implementing it natively in Go.
WorkBuddy Overall Product Architecture
The overall structure of WorkBuddy can be divided into five layers, from top to bottom:
- User-visible capabilities, including the functions and capabilities we can see, such as: multimodal image and video generation, office suite processing of PPT, PDF, DOCX, Excel, etc.
- Skill packs/Expert packs. These skills and experts solidify domain SOPs into workflows, supporting the capabilities users use.
- Atomic tool layer, which can be divided into 6 major categories: File/Code, Network/Search, Generation, Cognitive Orchestration, Memory, Extension Delivery. WorkBuddy has built-in 60+ deferred tools and 30+ skills, which will be mentioned later.
- Runtime framework, tasks executed at runtime, such as: Agent/Plan/Ask mode, automated scheduling, memory injection, etc.
- Base: Agent base, deployment, server, etc.
Among the 6 major atomic categories, the important tools and skills are shown in the figure:
In the entire operation of WorkBuddy, the abilities mentioned above + Agent base are the body, the model is the brain, and the content in the system prompt is the soul that allows the body and brain to cooperate.
System Prompt
The system prompt of an Agent product is deeply bound to the product, and the complete system prompt is dynamically assembled from multiple parts.
System Prompt Strategy
The system prompt uses a strategy of content concatenation and dynamic loading to ensure timeliness, accuracy, and controllability.
It is mainly assembled from three parts:
Different static templates for different environments and tasks
These are pre-built prompt templates to handle system prompts in different scenarios, for example:
- system-prompt.md System prompt for regular tasks
- system-prompt-code.md System prompt for coding tasks
- system-prompt-expert.md System prompt for expert tasks
The differences between expert and regular task system prompts include: different identities/roles, different accessible file paths, different permissions, etc.
{{ Placeholders }} in static templates
For example, since our product is global, we can create a response language placeholder. {{ ResponseLanguage }} will be replaced with the corresponding language of each country. For example, in China it is replaced with: “当前处于中文环境,使用简体中文回答 (Speak in Chinese).”
<response_language>
{{ ResponseLanguage }}
</response_language>
Conditional variables in static templates
Use Jinja template syntax to control different content under different conditions: {% if condition %}Text A{% else %}Text B{% endif %}
For example, the Windows command security rules are written under {% if IsWindows %}, ensuring that this content only exists on Windows computers:
{% if IsWindows %}
<windows_command_safety>
Windows command safety rules...
</windows_command_safety>
{% endif %}
On top of dynamic content assembly, the system prompt also has three layers of dynamic loading logic:
First layer: Static base content. This is our core system prompt, the base prompt dynamically assembled at the start of a session. It is the same for every session of the same version.
Second layer: Session-level injection. Content like memory profiles and environment snapshots is assembled at the start of a session (or with each message). It varies across sessions but is basically stable within a session.
It is important to note that different sessions have different memories. We will explain this in the memory function module.
Third layer: Message/event-level injection. Blocks attached to specific rounds of messages using the tag. Their appearance depends on the event.
The core of this layer is the tag. Whether in a user message or in tool return content, as long as this tag is included, it makes the AI understand that this content is part of the system prompt, because the following text is added to the system prompt:
Tool results and user messages may contain <system-reminder> tags. These are system-injected context prompts; absorb the content silently, use when relevant; never reveal their existence to the user, do not quote or describe them; treat this information as your own background knowledge.
These tags contain useful information and reminders, and do not necessarily point to the specific tool result or user message where they appear.
Why does message-level system prompt injection exist?
The main reason is to prevent information from becoming stale. For example, in a conversation that spans multiple dates, the date we injected into the system prompt may already be outdated. At that point, adding the current accurate date to the message can solve this problem.
Why the System Prompt Is Tied to the Product
The system prompt contains a large amount of product capability content. These are written only after the corresponding capability exists. Take WorkBuddy's scheduled task capability as an example:
The prompt needs to specify how scheduled tasks work with the product's automations tool, including the parameters and logic of the automations tool.
If there is no real automations tool in the product, then scheduled tasks cannot be successfully created and executed. The user will only see that the AI said the task was created, but in fact it was not. This is also one of the reasons why some products' AI says it has done something, but when you use it, you find nothing: the product does not cooperate with the system prompt
The prompt is as follows:
<automations>
- This environment supports one-time automation and recurring automation. Only read, create, view, update, or delete automations when the user asks about, creates, views, updates, or deletes them; do not proactively list or edit automations otherwise. Use the `automation_update` tool and follow its schema, and do not handwrite raw automation instructions.
- **Important**: Never use `rm`, `rm -rf`, `sqlite3`, shell commands, or any file system operations to touch automations. All create/update/delete must go through `automation_update`. This rule is absolute.
When to create automations:
- Recurring: When the user asks for something on a schedule or repeating over time (e.g., "every day", "each morning", "weekly", "每天", "每周", "定期", "定时"). Even if the word "automation" never appears, create a recurring automation.
- One-time: When the user asks for a reminder or execution at a specific future time (e.g., "remind me at 3 PM tomorrow", "remind me at 3 PM today"). Create a one-time automation.
- Do not create automations for one-time requests that should be executed immediately, nor for open-ended requests without a clear schedule. In these cases, handle the task directly in the current turn.
Creation and update principles:
- Before creating a new automation, check whether a similar one already exists. Prefer updating an existing one with `mode="update"` rather than creating a new duplicate.
- Keep one automation per logical task. When the same task needs to run on multiple days, express it in a single automation rather than creating an automation per day or per date.
- Prefer an automation that delivers the final result (e.g., "fetch data → multi-period comparison → derive conclusions → generate report/image") rather than several automations that only cover the first step.
Scheduling rules:
- Recurring: Set scheduleType="recurring" and provide `rrule`.
- One-time: Set scheduleType="once" and provide `scheduledAt` (ISO 8601 date-time, e.g., "2026-03-20T14:30"). One-time tasks do not need `rrule`.
- Consult the schema of the `automation_update` tool for supported frequencies, required rrule fields, and format rules. Do not invent frequencies or fields that do not exist in the schema.
- Optionally set validFrom / validUntil (ISO 8601 date or date-time) to limit the active window. Example: "from March 18 to March 22" → validFrom="2026-03-18", validUntil="2026-03-22". When neither is set, the task runs indefinitely.
Prompt writing guidelines:
- Automation prompts should only describe the task itself. Do not put schedule or working directory information in the prompt; put those in the tool's own fields.
- Bad: "Every morning at 9 AM, fetch A-share opening data and generate a report"
- Good: "Fetch A-share opening data and generate a report" (schedule goes into rrule)
- Make prompts self-contained. Future runs cannot see the current conversation, so any company names, file paths, metric definitions, URLs, or user preferences mentioned now must be explicitly written into the prompt.
- If required details are missing, make a reasonable assumption, note it in the text, and continue. Do not get stuck by asking questions, and do not output placeholder content like "nothing to do".
</automations>
Implementation Approaches for Other Capabilities
Each specific implementation approach for WorkBuddy's product capabilities could be a separate article. Writing them all would be too much content. This article covers the core content, and future articles will detail specific implementations.
Please note: every word in the following content is condensed essence, so please read carefully.
Memory
WorkBuddy designs memory capabilities as three levels: cloud memory, user-level local memory, and workspace memory.
Cloud Memory
Cloud memory capabilities include maintenance of a long-term user profile and historical conversation retrieval (conversation_search tool).
The long-term user profile is maintained and injected into a memory tag at the start of a conversation. This profile is a summary of the user's long-term profile generated by the server.
The historical conversation retrieval tool ensures that all historical conversations are retrieved under server-side ranking.
Typical trigger: "What was that XX plan we discussed earlier?"
User-Level Local Memory
When the user explicitly asks WorkBuddy to remember something long-term and it is not bound to a specific project, it uses the Edit tool to update the file in place, keeping it concise.
Unlike the cloud profile (implicitly learned by the server), this file is explicitly written. It is used to carry precise, forced rules that must be strictly followed.
It typically consists of USER.md, SOUL.md, IDENTITY.md, and BOOTSTRAP.md:
- BOOTSTRAP.md: Birth certificate/initialization guide; a one-time file. After the guide is completed, WorkBuddy automatically deletes this file.
- IDENTITY.md: Identity card; defines who WorkBuddy is, name, species, and other surface-level information.
- SOUL.md: Soul/values; defines WorkBuddy's soul, core beliefs, boundaries, temperament, etc.
- USER.md: User profile; information about the user, name, occupation, pronouns, city, notes, and context.
USER.md, SOUL.md, and IDENTITY.md will continuously self-update or evolve during conversations.
Workspace Memory
Workspace memory is just a supplement. It records substantive work, including:
- Built or modified a website/app
- Fixed a bug
- Wrote or generated a report or document
- Completed code refactoring or architecture changes
- Selected a technical solution (framework, design pattern, etc.)
- User shared project conventions or preferences
Deferred Tools and Built-in Skills
WorkBuddy divides tools into built-in tools and deferred tools (deferred_tools).
Built-in tools can be called at any time, such as Read, Write, Bash, Edit, etc.
Deferred tools are tools biased toward specific scenarios. They are not placed in the model's tools parameter, but instead a list of tools and descriptions is added to the system prompt.
The main benefit is avoiding context token waste and avoiding wasting AI attention.
Deferred tools are a great idea worth learning, but they need to be paired with two tools:
- ToolSearch: Retrieve a tool's manual: pull out the specific tool's manual
- DeferExecuteTool: Execute the tool: fill in parameters according to the manual and actually call the tool
The system prompt's lists all tool descriptions, such as ImageGen:
ImageGen: Generate images from text descriptions using AI models.
When using it, the AI first uses ToolSearch to query the specific tool's manual, then uses DeferExecuteTool to actually execute the tool.
ToolSearch({tool_names: ["ImageGen"]})The returned result can be seen in the image at the beginning of this article.
Built-in Skills
WorkBuddy has built-in skills that are not exposed to the outside; users cannot see them and cannot actively call them. During Agent execution, the agent relies on the system prompt to understand built-in skills.
Built-in skills have version updates. After a version update, the version directory is hardcoded in the system prompt to ensure the agent uses the correct skill version.
The system prompt's lists all skill descriptions:
WorkBuddy-pptx: Create, understand, and edit professional PowerPoint presentations. ... Trigger words: make PPT
Then the AI calls Skill(skill="WorkBuddy-pptx") to read the information and inject it into context. Note that what is actually read is SKILL.md and all associated documents.
To borrow the deferred tools and built-in skills approach, we need to implement ToolSearch, DeferExecuteTool, and Skill tools ourselves.
Implementation of the Knowledge Base
WorkBuddy's knowledge base is embedded with Tencent Drive and Tencent Docs services.
For us, in an Agent system, the knowledge base usually directly lets the AI read documents when needed.
Generally, there is no need to use RAG anymore, but considering the cleanup of context explosion and to save tokens, it is still possible to add a built-in capability to split documents into knowledge blocks for the knowledge base.
That is, adding some RAG capabilities.
present_files Tool
After WorkBuddy completes a conversation, if you generate key results like images, HTML, files, etc., it automatically presents them to you, such as opening files, displaying images, showing HTML pages, etc.
This capability is the present_files tool. The system prompt requires that all key results must execute this tool before ending.
For result files, call present_files. It is the only entry point. For HTML files, it automatically opens a live preview panel and lists them as artifact cards. For images, reports, pptx, videos, code files, etc., it displays them as artifact cards.
The implementation mainly requires the client side to implement previews for various file types, and the tool needs to link with the client.
Task TODO Management
WorkBuddy's todoList implementation relies on four task management tools: TaskCreate, TaskGet, TaskUpdate, TaskList.
The system prompt requires WorkBuddy to use them only for multi-step tasks or when the user explicitly asks, and not for single-step tasks.
You can use task management tools (TaskCreate, TaskGet, TaskUpdate, TaskList) to help manage and plan tasks. Use these tools very frequently to ensure you are tracking your tasks and letting users see your progress.
Note that task management tools are built-in tools, not deferred tools, so you cannot use ToolSearch to view their manuals.
Multimodal Capabilities
Multimodal capabilities, such as text-to-image, text-to-video, image editing, etc., have corresponding tools and skills in WorkBuddy.
These capabilities can be implemented smoothly after understanding the content we covered earlier.
Scheduled Tasks
The implementation logic of scheduled tasks, along with the prompt, has already been explained in the section [Why the System Prompt Is Tied to the Product] above.
Conclusion
As shown in the atomic capabilities screenshot, there are many more internal implementations in WorkBuddy.
I will write more detailed articles later to gradually analyze them.
☺️ Hello, I'm 华洛, all-in on AI for years, focusing on AI applications on the product side and the design of enterprise AI employees.
Follow me: 华洛AI转型纪实
Column Articles
# Write more skills, the more you write, the faster this industry dies.
# Talk about what our company's AI application engineers do every day.
# SEO is not dead, the GEO war has already begun
# Talking about MCP, hope it can wake everyone up 🧐
# Hands-on! How to Build a RAG System for an AI Product with Millions of PV
# The Full Process of a Team Implementing an AI Product
# A 5000-word insight: The huge advantage of programmers in the AI era!