Agent Skills Are Not Prompts: A Complete Engineering Guide to Writing, Evaluating, and Operating Them

1. High-Quality Skills: Structure, Writing, and Anti-Patterns

1.1 Three-Layer Skill Architecture: On-Demand Loading as Systems Engineering

The wrong approach is to cram all team knowledge and execution details into a single, ultra-long Markdown file. This practice not only increases the context cost of a single call but also severely blurs trigger boundaries.

Skill architecture is essentially an on-demand loading, system-level routing mechanism. This is why Anthropic's official documentation lists longer resource files and executable scripts as items to be "lazy-loaded after triggering." For high-frequency agents, strict layering is not a supplement but a threshold for system performance and accuracy.

Architecture Layer	Engineering Role	Consequence of Poor Writing
Metadata	Routing Trigger	Grabs the wrong task, or remains silent when needed
Instructions	Contract & Process Boundary	Disordered execution, unstable output structure
Resources/Scripts	Offloading Deterministic Actions	Hallucinates outdated knowledge, random execution each time

1.2 Where Skills Come From: Based on "Fault-Tolerance Patches," Not "Encyclopedias"

Building a comprehensive knowledge asset first and then turning it into a Skill is a typical wrong path.

Truly high-yield Skills are born only from scenarios that are "high-frequency, have clear standards, and are easily done wrong repeatedly." The knowledge during model training is static and frozen. If this generation gap is not addressed, the system usually suffers from continuously outputting old SDK writing styles and ignoring edge-case exceptions.

Google's article "Closing the knowledge gap with agent skills" points out: Model knowledge is fixed at the training point, but SDKs and best practices are constantly changing; this continuously creates a knowledge gap that the model itself struggles to eliminate. What Google put into the Gemini API developer skill is not a complete encyclopedia but a capability overview, current SDK information, basic examples, and links to official documentation. This design itself already illustrates the role of a Skill: not to replace documentation, but to guide the Agent to the correct approach.

Anthropic's documentation "Skill authoring best practices" points out: A Skill is not a one-time prompt at the conversation level but an on-demand loaded capability unit: metadata is resident but lightweight, main instructions load only after triggering, and resource files and scripts are read or executed as needed. The purpose is clear: do not permanently press all methods into the context; only bring in the part truly needed for the current task.

1.3 Core Writing Principles: Boundaries Over Details

Writing the Description as a "promotional spot" for humans to read is the primary culprit behind false triggers.

• Write Trigger Conditions First: The description is not a promotional spot but a routing entry point. Once the description is written too broadly, subsequent evaluation results almost certainly worsen because it grabs the wrong task at the entry point.
• Write Steps Next: High-quality Skills rarely win by having a "very complete background introduction"; they win more through step order, decision points, and output contracts.
• Externalize Volatile Information: If content changes rapidly, do not hardcode it into SKILL.md. SDK versions, interface parameters, model lists, capability matrices—once this information is hardcoded, it will soon start consistently outputting outdated practices.
• Script Deterministic Logic: Do not leave stably executable actions to the model's improvisation. Data processing that can be stably run with Python/Bash should never be generated ad-hoc by the model.
• Must Run on Real Tasks: A Skill that looks reasonable on paper may bring no gain at all once online.

1.4 Anti-Patterns and Fatal Traps

Many Skills fail unspectacularly: they don't error out, nor do they fail immediately. More common is: they start consistently producing old practices, grabbing the wrong tasks, or simply bringing no visible gain. The following are repeatedly verified anti-patterns:

• Only writing "this skill is good at X" without stating when not to use it
• Cramming all team knowledge into one super-long SKILL.md, leading to high loading costs and blurred boundaries
• No example inputs/outputs, causing the model to know the principle but not the concrete implementation
• No exit conditions, causing loop-type skills to keep fixing and modifying indefinitely
• Treating scripts as magical black boxes, neither describing inputs/outputs nor handling errors
• Directly automating high-risk actions without human review checkpoints
• Skill overlapping with the agent's role: routing, executing, and reviewing simultaneously, losing focus
• Never doing evaluations, relying only on "feeling the model can use it"
• Skills frequently misfiring

1.5 Skill Pitfall Avoidance Guide

《Lessons from Building Claude Code: How We Use Skills》

Core Writing Principles

1. The Description Field is a Trigger Written for AI, Not a Summary for Humans

• Tip: When Claude decides whether to invoke a Skill, it scans the descriptions of all Skills. Therefore, the description should not just be "this is a skill for writing code," but should explicitly state "in what scenario, upon encountering what request, this skill should be triggered."

2. Focus on Writing "Gotchas"

• Tip: Create a dedicated Gotchas (common errors/pitfalls) section in the document. This is the highest-value content in the entire Skill.
• Practice: Record the most common mistakes Claude makes when using this skill, continuously update this list, and explicitly tell it "do not do X."

3. Don't Write Fluff (Provide Counter-intuitive/Convention-breaking Knowledge)

• Tip: Claude already knows a lot of code knowledge and conventional defaults. High-quality Skills should provide information that can "break Claude's inherent thinking."
• Example: For a frontend design skill, don't teach it how to write HTML, but teach it to "improve aesthetics," telling it to avoid using the default Inter font and clichéd purple gradients.

4. Leave Room for AI, Don't Be Overly Rigid

• Tip: Because a Skill needs to be reused repeatedly, overly rigid instructions will limit the AI's performance. Provide necessary information and boundaries but give the AI space to be flexible based on specific situations.

Architecture and Context Management (Designing Skills as Systems)

5. Use the File System for "Progressive Disclosure"

• Tip: A Skill should not be written in a single giant Markdown file; it should be a folder. Break down large tasks so Claude reads specific files at the appropriate time.
• Practice: Put detailed API documentation in references/api.md; put output templates in assets/. The main file only provides guidance, telling Claude "where to go and what file to read when needed."

6. Provide Ready-Made Scripts and Code Libraries for AI

• Tip: Instead of having the AI write boilerplate code from scratch every time, provide ready-made helper scripts or class libraries directly in the Skill folder.
• Benefit: This allows Claude to spend its computation and time on "business logic orchestration" and "high-level analysis," directly calling the low-level scripts you've written, greatly increasing the success rate.

Advanced Feature Design (Interaction and Memory)

7. Design an Elegant "Initialization Configuration" Process (Setup)

• Tip: Many skills require user context (e.g., which Slack channel should the daily report be sent to?).
• Practice: Place a config.json in the directory. If the configuration is empty, instruct Claude to use the AskUserQuestion tool to proactively ask the user (even offering structured multiple-choice questions) to gather information before executing.

8. Give Skills "Memory"

• Tip: Use files (like append-write .log files, JSON, or even SQLite databases) to let the skill remember historical states.
• Example: After writing a daily report, save it to standups.log. The next day, before running, Claude reads the history log first, so it knows what has changed compared to yesterday.
• Note: Store data in a stable environment variable directory (like ${CLAUDE_PLUGIN_DATA}) to prevent data from being wiped during skill upgrades.

9. Use On-Demand Hooks

• Tip: For some extreme or subjective restrictions, don't make them globally effective. Instead, write them as Hooks that only take effect when that specific skill is invoked.
• Example: Write a /careful skill that intercepts dangerous commands like deleting databases or force-pushing code; or write a /freeze skill that prohibits the AI from modifying files outside a specific directory.

2. Five Most Common Skill Patterns

2.1 Optimal Structures for Different Business Flows

Don't use flat, straightforward instructions for every task. Based on Google Cloud Tech, Anthropic, and internal engineering practices, the following five design patterns are the optimal structures for handling different business flows.

Design Pattern	Core Engineering Value	Typical Application Scenarios
1. Tool Wrapper	Isolates implementation details, loads CLI/SDK context on demand.	New SDK imports, initialization, calling methods
2. Generator	Uses assets/ templates to enforce output structure.	Specs, PR descriptions, test plans
3. Reviewer	Decouples "what to check" from "how to check," introduces multi-level scoring.	Code review, compliance pipeline blocking
4. Inversion	Blocks the Agent's "blind guessing," forcing a shift to a question-collection mode.	Ambiguous requirement clarification, environment initialization config
5. Pipeline	Enforces sequential SOP execution, sets strong validation checkpoints for intermediate results.	Extraction -> Validation -> Assembly -> Backfill

Pattern 1: The Tool Wrapper

A Tool Wrapper provides your agent with on-demand context for a specific library. Instead of hardcoding API conventions into the system prompt, you package them into a skill. Your agent only loads this context when it actually uses that technology.

Sharing an excellent, typical Tool-type Skill: playwright-cli: https://github.com/microsoft/playwright-cli

Pattern 2: The Generator

While a Tool Wrapper applies knowledge, a Generator ensures output consistency. If you encounter the problem of an agent generating different document structures each run, a Generator solves this by orchestrating a fill-in-the-blanks process.

It utilizes two optional directories: assets/ for output templates and references/ for style guides. The instructions act as a project manager.

They instruct the agent to load the template, read the style guide, ask the user for missing variables, and fill in the document. This is very practical for generating predictable API documentation, standardized commit messages, or scaffolding project architectures.

Sharing two excellent, typical Generator-type Skills, great for drawing diagrams, five-star recommendation!!!

• excalidraw-diagram-skill: https://github.com/coleam00/excalidraw-diagram-skill
• draw.io: https://github.com/softaworks/agent-toolkit/blob/main/skills/draw-io/README.md

Pattern 3: The Reviewer

The Reviewer pattern separates what to check from how to check. Instead of writing lengthy system prompts detailing every code smell, store modular scoring criteria in a references/review-checklist.md file.

When a user submits code, the agent loads this checklist and systematically scores the submitted code, grouping results by severity. This is an efficient method to automate PR reviews or catch vulnerabilities before a human reviews the code.

Sharing code-review-skill: https://github.com/awesome-skills/code-review-skill

Pattern 4: Inversion

Agents naturally tend to guess and immediately generate answers. The Inversion pattern subverts this dynamic. Instead of the user asking a question and the agent executing, the agent plays the role of the interviewer.

Pattern 5: The Pipeline

For complex tasks, skipping any step or ignoring any instruction is unacceptable. The Pipeline pattern enforces a strict sequential workflow with critical checkpoints. By implementing explicit gating conditions (e.g., requiring user approval before moving from docstring generation to final assembly), this flow ensures the agent cannot bypass complex tasks and present unverified final results.

For a deeper understanding of these patterns, it is recommended to read the original text: 《5 Agent Skill design patterns every ADK developer should know》

2.2 Choosing the Right Pattern

In practice, which pattern to choose depends on your specific needs. Here are some selection suggestions:

• Need the Agent to correctly use a specific tool or framework → Choose Tool Wrapper
• Need stable output of a specific format of document or content → Choose Generator
• Need to check and review existing content → Choose Reviewer
• Need to collect information first before generating a solution → Choose Inversion
• Need to strictly execute complex tasks step-by-step → Choose Pipeline

It is important to note that these patterns are not mutually exclusive. In actual projects, you may combine multiple patterns to meet complex requirements. For example, a Pipeline-mode Skill might internally call multiple Generator and Reviewer sub-skills.

3. How to Create a High-Quality Skill Closed Loop

Writing a document and directly throwing it at a large model to run is the crudest workshop-style practice. Industrial-grade Skills require a rigorous 8-step closed loop.

Applying TribalScale's pattern to skill design can be summarized into a practical 8-step closed-loop method:

TribalScale《5 AI Agent Design Patterns Every Developer Needs to Know》

Step 1: Select Scenario: Pick tasks that are high-frequency, reusable, and have verifiable results.

Don't force open-ended tasks that heavily rely on ad-hoc judgment and are almost never repeated into a skill. First, define task characteristics, latency requirements, cost budgets, and human involvement needs, then choose a pattern.

Step 2: Define Boundaries: Clearly state what the skill does and does not do.

A good skill must have exclusive boundaries; otherwise, multiple skills will hit simultaneously, causing trigger chaos. OpenAI explicitly recommends that the description should clearly state "should trigger" and "should not trigger."

Step 3: Choose Pattern: Not all skills need multi-Agent setups.

Both OpenAI and Anthropic recommend maximizing the single Agent's capability first, before introducing multi-Agent setups for complex logic, tool overload, or clear task division.

Step 4: Write Description: This is the key to trigger quality.

The description should not just say "does X," but should state "under what input signals, to solve what problem, using what constraints, avoiding what false triggers."

Step 5: Write Process: Write the process as imperative steps executable by the model, not loose suggestions.

OpenAI suggests breaking dense SOPs into smaller, clearer actions and ensuring each step corresponds to a specific output or action.

Step 6: Equip Resources: Put templates, FAQs, policy documents, example inputs/outputs, and deterministic scripts into the skill directory, instead of repeatedly explaining them in chat.

Step 7: Add Guardrails: Any skill potentially involving high-risk actions, private data, or irreversible write operations must have guardrails and human-in-the-loop.

OpenAI explicitly lists escalating high-risk actions and failure thresholds to human handling as a key practice.

Step 8: Evaluate: A skill is not finished once written.

4. Skill Evaluation and Governance: From Writable to Verifiable

Judging solely by whether the final generated answer "looks good" is a meaningless, ineffective evaluation. The existence of a Skill directly changes the Agent's global workflow. A poorly written Skill will silently lower the success rate of originally correct tasks.

4.1 Four-Layer Evaluation Framework: Rejecting the Black Box

Evaluation of a Skill must be split into four layers:

1. Trigger Layer: Measure Trigger Precision, Recall, and False Trigger Rate.
2. Execution Layer: Measure intermediate step compliance rate and tool call accuracy.
3. Outcome Layer: Measure final task completion rate and format contract adherence.
4. System Layer: Measure latency, Token cost, and inter-skill conflict rate.

Direct Judgment: If an evaluation system cannot locate whether a problem stems from "trigger failure" or "execution drift," this evaluation system is invalid. This is also why OpenAI strongly advocates Trace Grading as the core method for system-level defect localization.

4.2 Evaluation Process and Release Standards

1) First, collect real failure samples.

Don't first create a batch of "seemingly reasonable" test questions. First, collect four types of samples: historical failed tasks, high-frequency repetitive tasks, boundary scenarios, and clearly inapplicable scenarios. OpenAI emphasizes reproducible evaluations and dataset-based iteration in its evals articles; the direction is very clear.

2) Must do A/B testing.

A Skill cannot be evaluated in isolation; you must look at the difference "before adding / after adding." Google's public case is convincing precisely because it provides a before-and-after comparison: after adding the Gemini API developer skill to the same model, the success rate increased from 28.2% to 96.6%. Without A/B testing, judgments about "better results" are generally unreliable.

3) Scoring must be layered.

Prepare at least three types of scoring: rule-based scoring checks format, schema, fields, and code executability; trace scoring checks if steps were followed correctly and tools were called correctly; human or model-based scoring checks explanation quality, structure, and business acceptability. OpenAI's approach in "Testing Agent Skills Systematically with Evals" is very practical: break a single run into traces and artifacts, then score across dimensions like outcome, process, style, and efficiency.

4) Test for conflicts before going live.

Just because a single Skill runs well doesn't mean there are no problems when multiple Skills coexist. At least three types of conflicts must be tested: whether two Skills will compete for the same type of task, whether a general Skill will overpower a more specialized Skill, and whether a Skill will induce incorrect tool calls or invalid context loading. For Skills, this is not a supplementary check but a release threshold.

Dimension	Minimum Requirement
Trigger Quality	Recognition of target scenarios is significantly better than the no-Skill baseline
Outcome Benefit	Target task success rate is stably improved
Negative Impact	Does not significantly degrade performance on neighboring tasks
Maintenance Cost	Volatile information is externalized, not reliant on high-frequency manual rewriting
Risk Control	No high-risk scripts, hardcoded credentials, or unauthorized access

If it requires high-frequency manual maintenance, the design is flawed. If its benefits are unstable, the abstraction boundary might be wrong. If it introduces high-risk scripts or unauthorized access, it should not go live directly. Anthropic's enterprise guidance lists script execution, network access, MCP references, hardcoded credentials, and out-of-bounds file access as review items.

4.3 Continuous Governance Mechanism

When a Skill becomes increasingly large and its core value is merely "stably executing a piece of logic," it should be downgraded to a Tool or static script. Forcing purely computational or purely I/O work to remain in the Prompt Context only makes evaluation harder and maintenance more expensive.

Status	Trigger Condition	Handling Method
Retain	Stable triggering, clear benefit, controllable maintenance	Continue use, include in regression evaluation
Split	Overly broad description, bloated instructions, mixed responsibilities	Split into multiple narrower Skills
Upgrade	Core value has become stable execution	Refactor into a script, Tool, or MCP
Retire	Long-term non-triggering, insufficient benefit, excessively high risk	Take offline and retain historical records

The most important sentence here is not the methodology, but the boundary: when the core value becomes stable execution, the Skill should exit, and a tool should take over. Continuing to keep such problems within a Skill will ultimately only make the context heavier, evaluation harder, and maintenance more expensive.

5. Skill Ecosystem Operations and Engineering Management

Only adding new ones without maintenance will eventually turn high-value internal reusable capabilities into a pile of massive, noisy junk directories.

5.1 Directory Categorization and Listing Thresholds

Skills must be managed as internal middleware products. Do not push personal "prompt fragments" directly to the public directory.

Without clear listing constraints, the system typically faces severe context pollution. Relevant governance measures have been incorporated into the best practices of leading architectures: it must be ensured that Skills in the public directory have "clear trigger boundaries, no generalized fluff, no hardcoded credentials, and no unauthorized scripts."

Directory Categorization

Once you have more than a dozen Skills, the biggest risk is no longer "not enough" but "hard to find."

The "How We Use Skills" sharing provides a useful observation: actively used Skills naturally tend to cluster into a few categories, such as Library & API Reference, Product Verification, Data Fetching & Analysis, Business Process & Team Automation, Code Scaffolding & Templates.

The value of this categorization lies not in the names themselves, but in turning a "file list" into a "navigation structure." Without categorization, all subsequent recommendations, evaluations, and retirements become harder.

Category	Core Value	Priority Actions
Library / API Reference	Reduce misuse of SDKs, internal libraries	Internal SDKs, design systems, CLIs
Product Verification	Turn "done" into "verified"	UI flow verification, end-to-end checks
Data Fetching / Analysis	Lower the barrier to data retrieval and analysis	Metric queries, monitoring localization, funnel analysis
Process Automation	Solidify repetitive cross-system processes	Weekly reports, ticket creation, post-release checks
Scaffolding / Templates	Stably output project convention structures	Page templates, service templates, test templates

These five categories cover the areas where most teams are most likely to generate reuse value: reducing misuse, improving verification rates, compressing repetitive processes, and stabilizing output structures.

Listing Thresholds

Not all useful personal Skills are suitable for entering the team's shared directory. The threshold for the shared directory should be significantly higher than for personal use, because it affects more people and is more likely to create system noise. A more stable approach is to give public Skills a minimum listing standard:

1. Has been used in real tasks, not just a paper design
2. Trigger boundaries are clear, not generalized experience
3. Has clear benefits, not just "feels convenient"
4. Volatile information is externalized, not reliant on frequent manual rewriting
5. No high-risk scripts, unauthorized access, or untrusted external links

Anthropic repeatedly emphasizes real usage, discoverability, and security boundaries; the most direct landing point for these requirements in ecosystem operations is the public directory threshold.

Skill Reuse Rate

Many internal knowledge systems fall into the same trap: only looking at additions, not usage. This is especially dangerous for Skills. Because a Skill is not a static document; it participates in triggering, occupies context, and affects routing.

Quantity itself has no value; being correctly reused does.

Metric	Focus Point
Activation Rate	Whether the written Skill is actually being used
False Trigger Rate	Whether it grabs tasks it shouldn't
Task Benefit	Whether the target task stably improves after adding the Skill
Reuse Scope	Whether it has been used beyond the original author
Maintenance Lag	Whether the description and resources are long-term expired
Conflict Rate	Whether it competes for tasks or interferes with other Skills

This set of metrics is the same system as the evaluation framework discussed earlier: evaluation answers "is a single Skill worth existing," while operations answer "is it worth continuing to occupy a spot in the public directory."

5.2 Isolation and Recommendation Paths

A Skill ecosystem is not a file repository; it cannot just do "ingestion." What truly determines whether an ecosystem comes alive is the recommendation path: who sees which Skill in what scenario, which Skill is discoverable by default, and which Skill is only exposed to specific teams.

The Claude Code documentation has already implemented mechanisms like automatic discovery, additional directories, access restrictions, and direct invocation, indicating that the platform layer itself supports the idea that "not all Skills are laid flat for everyone."

Tier	Role
Default	High-frequency, low-risk, strong reuse, discoverable by default
Specialized	Targeted at specific teams or tasks, exposed by directory or permission
Archive	Low-frequency, pending retirement, or only retaining traceability value

The benefit of this approach is direct: the public directory won't become increasingly noisy; high-value Skills are easier to find; low-frequency Skills won't disappear in a one-size-fits-all manner.

Skill ecosystem operations are not about "ingesting more Skills," but about turning scattered techniques into organizational assets. A team that can only write Skills but not operate them will typically end up with three results: the directory grows larger, but no one knows which to use; truly valuable Skills are drowned in noise; old Skills don't retire, and the system becomes increasingly untrustworthy.

Appendix A: An Engineered, Reusable Skill Template

This template removes all redundant polite language, forcing a core focus on boundary division, contract delivery, and exception circuit-breaking.

---
name: target-skill-name
description: [Trigger] Mandatory trigger upon detecting {Signal A}, used to solve {Problem Type B}. Strictly prohibited from triggering upon encountering {Situation C}.
---

### 1. Purpose & Boundary

- **Applicable Scope**: ...
- **Prohibited Scenarios**: ...

### 2. Execution Workflow

- **Step 1. [Judgment]**: If variable X is missing, call AskUser to request it; strictly prohibit self-fabrication.
- **Step 2. [Loading]**: Read `references/api_v2.md` to get the latest Schema.
- **Step 3. [Execution]**: Call `scripts/validate.py` to verify the data.

### 3. Output Contract

- **Must Include**: ...
- **Prohibited**: Absolutely do not generate default purple gradients, do not use deprecated API syntax.

### 4. Fallback Strategy

- Upon encountering an unseen error code, stop inference, output the error as-is, and hand over to a human.

Appendix B: A Skill Evaluation Case Study

Below is a directly implementable evaluation case study. The data is example data; the purpose is not to prove the real-world performance of a specific product, but to walk through the complete process of how to do an evaluation, how to read the results, and finally how to decide whether it can go live.

Scenario

Assume a team has created an SDK integration Skill.

Its goal is simple: when the Agent needs to generate code based on a new SDK, prioritize using the currently recommended import, initialization method, calling path, and minimal example, avoiding the continued output of old version writing styles.

This scenario is suitable as an evaluation case study for simple reasons:

• The task boundary is clear, making it easy to define "right" and "wrong"
• Without the Skill, the model easily continues to output old writing styles
• After adding the Skill, the before-and-after comparison is usually very obvious

Google's public Gemini API developer skill is a similar scenario: it is designed around SDK usage, examples, and official documentation entry points, and provides before-and-after comparison results.

Evaluation Goal

This Skill is not meant to make answers "look more complete," but to reduce three types of errors:

• Using deprecated imports
• Using incorrect initialization methods
• Outputting code that does not conform to the current SDK's recommended structure

The real question is not "does the generated code look like it," but:

After adding the Skill, does the Agent more stably write the current correct usage.

Evaluation Set Design

The evaluation set does not start from "imaginary questions" but from breaking down real usage scenarios.

Category	Count	Example Task
Basic Code Gen	20	Generate a minimal chat example using the SDK
Streaming Output	15	Generate a calling example supporting stream
Tool Calling	15	Generate an example supporting function/tool calling
Doc Q&A Wrapper	10	Wrap a document Q&A function based on the SDK
Error Fixing	20	Fix a piece of code using old SDK writing styles
Boundary Scenarios	10	User description is vague, requiring the Agent to judge first, then generate
Inapplicable Scenarios	10	Unrelated to this SDK, should not trigger the Skill

Total evaluation set: 100 items.

"Inapplicable scenarios" cannot be omitted. Skill evaluation tests not only "is it good when used," but also "will it trigger randomly when it shouldn't."

Evaluation Method

Use A/B comparison:

• Group A: No Skill
• Group B: With Skill
• Model, system prompt, tool environment, and test questions are all kept identical
• Each sample is run 3 times to reduce single-run variance

Evaluation is split into four layers:

Evaluation Layer	Focus Point	Metrics
Trigger Layer	Whether it triggers when it should, and doesn't when it shouldn't	Trigger Precision / Recall
Execution Layer	Whether it follows the Skill-specified process	Step compliance rate, whether correct resources are read
Outcome Layer	Whether the code result is correct	Task success rate, code executability rate
System Layer	Whether side effects are introduced	False trigger rate, average latency, average token cost

Scoring Rules

Rule-Based Scoring (suitable for clearly determinable items):

• Whether the current import is used
• Whether the current initialization method is used
• Whether required parameters are included
• Whether the code can run
• Whether the output meets the specified format

Trace Scoring (suitable for process checks):

• Whether the Skill-specified resource was read first
• Whether tools that shouldn't have been called were called
• Whether key steps were skipped to generate directly

Human Scoring (suitable for structure and usability checks):

• Whether the code is easy to reuse
• Whether the comments are sufficient
• Whether the explanation is consistent with the code

OpenAI's approach in "Testing Agent Skills Systematically with Evals" and trace grading essentially involves explicitly separating these dimensions, rather than just looking at the final answer.

Example Results

Below is a set of example results, mainly used to illustrate how to read the numbers.

Trigger Layer Results

Metric	No Skill	With Skill
Trigger Precision	-	0.91
Trigger Recall	-	0.87
Inapplicable Scenario False Trigger Rate	-	0.08

The first thing to look at here is not "is the trigger rate high," but "is the false trigger rate high." Because even if a Skill performs well on the outcome layer, if its false trigger rate is high, it will start grabbing tasks it shouldn't once in production.

Outcome Layer Results

Metric	No Skill	With Skill
Task Success Rate	62%	86%
Code Executability Rate	68%	90%
Old SDK Writing Style Occurrence Rate	31%	6%
Output Format Compliance Rate	74%	93%

This set of results indicates that the benefit brought by the Skill is mainly reflected in two points:

1. Old writing styles decreased significantly
2. Executability rate improved significantly

This is also the most typical value source for SDK-type Skills: it doesn't make the model "explain better," but makes the model take fewer old paths.

System Layer Results

Metric	No Skill	With Skill
Average Latency	5.2s	5.9s
Average Token Cost	1.00x	1.12x
Conflict Rate with Other Skills	-	0.05

A very real phenomenon can be seen here: after adding a Skill, cost and latency usually increase slightly. As long as the benefit clearly outweighs this cost, it's not a problem. What's truly troublesome is unstable benefits, non-decreasing false triggers, and high conflict rates.

How to Read This Result

The most common mistake in such case studies is focusing only on the success rate.

Looking only at the success rate is insufficient; at least three other things must be considered simultaneously:

1. Is the false trigger rate high? Frequent triggering in inapplicable scenarios usually means the description was written too broadly.
2. Did old writing styles truly decrease? This is more important than "the answer looks better" because it proves the Skill changed the method path.
3. Are the system side effects acceptable? If cost, latency, and conflict rates increase too quickly, it indicates this Skill is not yet suitable for direct launch.

Evaluation Conclusion

Based on this set of example data, this Skill can enter the next stage, but cannot yet be considered "complete."

The reason is straightforward:

• The outcome benefit is already clear enough
• The false trigger rate can still be further reduced
• The conflict rate with other Skills still needs further testing

A more reasonable disposition is not "launch and be done," but:

1. Narrow the description to continue suppressing false triggers
2. Add more inapplicable scenario samples
3. Continue conflict evaluation in a multi-Skill coexistence environment
4. Include in regression evaluation to prevent old writing styles from being brought back

What this case study truly illustrates is not "how much the success rate improved," but how Skill evaluation should be done: not just looking at the final answer, not just testing single-turn output, not launching right after writing, but splitting the trigger, execution, outcome, and system layers apart for examination.

About OpenTiny NEXT

OpenTiny NEXT is an enterprise-level intelligent front-end development solution. Based on two core technologies, Generative UI and WebMCP, it intelligently upgrades existing traditional products like the TinyVue component library and TinyEngine low-code engine, building new products oriented towards Agent applications, such as front-end NEXT-SDKs, AI Extension, TinyRobot intelligent assistant, and GenUI. This enables AI to understand user intent and autonomously complete tasks, accelerating the intelligent transformation of enterprise applications.

Welcome to join the OpenTiny open-source community. Add the WeChat assistant: opentiny-official to participate in front-end technology discussions~ OpenTiny Official Website: https://opentiny.design GenUI SDK Code Repository: https://github.com/opentiny/genui-sdk (Welcome to star ⭐) TinyRobot Code Repository: https://github.com/opentiny/tiny-robot (Welcome to star ⭐) NEXT SDK Code Repository: https://github.com/opentiny/webmcp-sdk (Welcome to star ⭐)

If you also want to co-build, you can enter the code repository, find the good first issue tag, and participate in open-source contribution together~ If you have any questions, feel free to leave a comment for discussion!