After reading this article, you will know how to test the Skill created in the previous article.

That's right, Skills can be tested too!

At the same time, we will get a truly runnable script. Every time you modify the Skill later, you can use it to re-check and confirm that the original capabilities haven't been broken.

In the previous article, we created a working Skill: meeting-notes-formatter. It can organize messy meeting notes into a clearly structured meeting summary that's easy for the team to share.

However, creating a Skill that works once is different from truly trusting it, using it, and recommending it to your colleagues.

A Skill running successfully once might just mean the prompt happened to be right this time. Only when it meets the same set of standards across different inputs and multiple runs can it truly become reliable infrastructure.

We use Skills precisely to improve consistency. And testing is how we achieve that consistency.

So, grab a cup of coffee. ☕

Let's see how to make this Skill one you can "truly trust, use, and be willing to recommend to colleagues."

Why Testing Is Always the Easiest to Skip

This situation is actually quite common:

We finish writing a Skill, paste in a messy chunk of meeting notes, and it quickly generates a beautiful summary. Nothing seems wrong, so we think: this Skill is done.

But when you recommend it to a colleague, a week later, the colleague uses a different phrasing, and the Skill might not trigger at all; or it does trigger, but invents an action item that no one ever confirmed. Even with the same input, the results on Monday and Friday can be completely different (slight differences are acceptable, after all, this is AI).

This doesn't necessarily mean the Skill is broken. More accurately, it was never tested in the first place.

Testing a Skill requires answering three questions, and none can be missed:

• Does it trigger correctly when it should? And can it stay quiet when it shouldn't?
• Is its output correct? Does the structure meet requirements, and is there any fabricated information?
• Is it consistent enough? Across multiple runs, do the results maintain the same structure?

Missing any one of these means the Skill still carries a strong element of chance.

But before starting testing, we need to determine: to what extent does this Skill actually need to be tested?

Three Levels of Testing Intensity

Not every Skill needs a complete testing system. Testing investment should match its importance.

• Manual Testing: Directly input a request to the Agent, observe whether it triggers, and whether the final output is correct. Fast, requires no extra preparation, great for an initial check.
• Scripted Testing: Automate the test cases. Every time you modify the Skill later, you can re-run the same tests. This is the approach used later in this article.
• Programmatic Evaluation: Run a full Eval against a fixed test set, measuring results through APIs, run logs, and scorers provided by the specific host or model platform.

A Skill you use yourself clearly doesn't need the same testing rigor as a Skill released to thousands of users.

Here's a simple rule of thumb to remember:

Personal Skill: manual testing is usually enough; Team Skill: better to add scripts; Production-grade Skill: requires programmatic evaluation.

Let's start with manual testing, then hand off the repetitive work to a script.

But before officially starting, there's a very practical tip that can save a lot of time.

First, Solve a Truly Difficult Sample

People who are truly good at creating Skills usually don't start by preparing twenty test cases.

They first take a genuinely difficult input, like the messiest meeting notes they can find, and continuously tweak the Skill until the Agent can handle it correctly. Then, they take this effective experience and expand it to more types of test cases.

This approach provides faster feedback. Solving one truly difficult failure case is often more valuable than watching ten simple cases all pass.

So, get one very bad sample working stably first, then expand test coverage.

Let's continue using meeting-notes-formatter and complete the three tests in order.

Test 1: Does It Appear at the Right Time?

As mentioned in the previous two articles, when an Agent decides whether to load a Skill, the first things it sees are the name and description in the Frontmatter. Among these, automatic matching mainly relies on the description.

Therefore, trigger testing is essentially checking our description: does it cover the expressions real users might use, without making the scope too broad?

First, we need to prepare two lists.

Should Trigger: Content a real user might input.

- "Organize these meeting notes"
- "Clean up the phone call notes"
- "Write meeting minutes based on the following content"
- "Summarize our meeting just now"
- Directly paste a chunk of messy meeting notes without any command

Should Not Trigger: Requests that seem somewhat close but are actually unrelated.

- "What's the weather like in San Francisco today?"
- "Help me write a Python function"
- "Create an expense report"

The second list is just as important as the first. A Skill that triggers for almost any request is just as problematic as one that never triggers.

Here, we can do a quick check by directly asking the Agent:

"Under what circumstances would you use the meeting-notes-formatter Skill?"

The Agent's response usually rephrases the description. If its answer doesn't cover the expressions in the "Should Trigger" list at all, the problem is likely in the description, not the Agent itself.

However, this method only checks how the Agent understands the description; it cannot replace actual trigger testing. For formal testing, you still need to input the positive and negative requests from the lists one by one and observe whether the Skill is actually invoked.

Correct triggering is just the first step. Next, we need to see if the content it actually generates is correct.

Test 2: Is the Output Correct?

This is where the success criteria we defined in the previous article come into play.

We required this Skill to:

• Always generate four sections: Attendees, Decisions Made, Action Items, Open Questions;
• Not fabricate information not present in the original notes;
• Provide an owner for each action item, marking it as "Unassigned" if unconfirmed;
• Produce a final result shorter than the original meeting notes.

Functional testing means running a known input and then checking the output against these rules.

For example, we can use this messy meeting note:

"Discussed product launch with Sarah and Tom. We decided to postpone to March 15th. Sarah will finish the press release by next Friday. Tom needs to talk to legal, but doesn't know when it will be done. We are still discussing whether to include the mobile features in v1."

The correct output should contain all four sections, list Sarah and Tom as attendees, record the March 15th decision, and assign the press release owner to Sarah.

There's another very critical requirement: it must not invent a meeting date, because the original notes didn't provide one at all.

This last point is easily overlooked.

The most common problem with Skills is often not a missing section, but very confidently adding a piece of information that doesn't exist in the original text.

Doubao (ByteDance's AI model) hallucinates quite a lot. I asked it some questions about Compose, and it even fabricated functions to fool me.

So, the test set should include at least one use case where "the correct answer is 'not stated'."

After the first two tests pass, there's one last test left. It determines whether we have a demo or a truly usable tool.

Test 3: Does It Work Every Time?

Run the same input five times in a row.

Then check if all five outputs use the same structure.

If the first run generates all four sections, but the third run drops "Open Questions," then even if each individual output looks fine on its own, this Skill still cannot be considered reliable.

What we're checking here isn't identical wording. Specific phrasing can vary; we care about the same result shape: same sections, same formatting rules, same handling of missing information.

If the structure drifts across multiple runs, it usually means the instructions aren't strict enough. Move the key rules to a more prominent position and write more explicitly what counts as truly complete.

This can also be seen as part of an Agent Harness. Of course, it's not the entirety of Agent Harness, but in Agent development, we do spend a lot of time writing success criteria, repeatedly running samples, and checking for regressions.

Consistency is the very reason Skills exist. If you can only keep one set of tests, prioritize repeated regression tests with clear success criteria, avoiding the trap of only verifying "it's the same every time" without verifying "it's correct every time."

However, manually repeating these operations quickly becomes tedious. Let's automate them.

Using a Script to Complete All Three Tests

Manual testing is great for initial checks.

But the Skill will continue to be modified later, and we want to be able to re-check everything in seconds after each change, rather than typing in a dozen requests again.

Therefore, we can add a small but truly runnable test Harness to the Skill directory. It reads the actual SKILL.md in use, rather than a separate copy, and then completes the three tests described earlier.

meeting-notes-formatter/
├── SKILL.md
├── references/
│   └── formatting-rules.md
├── assets/
│   └── meeting-template.md
└── tests/
    └── test_skill.py   ← Test Harness

The full script is available here: test_skill.py

This script only uses the Python standard library. Set the API Key and run:

export ANTHROPIC_API_KEY="sk-ant-..."
python tests/test_skill.py --mode all

This uses a test script written for the Anthropic API. If using Codex, OpenAI API, or other Agent hosts, you need to replace the corresponding model call and run observation methods, but the testing approach itself remains the same.

After running, you'll get results similar to this:

=== 1. TRIGGERING ===
  [PASS] (expected: trigger    ) Format these meeting notes for me
  [PASS] (expected: stay quiet ) What's the weather in San Francisco?
  ...
  -> 9/9 trigger cases correct

=== 2. FUNCTIONAL ===
  [PASS] happy_path_with_owners: required sections present
  [PASS] happy_path_with_owners: no invented date
  ...
  -> 6/6 functional checks passed

=== 3. CONSISTENCY (5 runs) ===
  run 1: sections = ['Action Items', 'Attendees', 'Decisions', 'Open Questions']
  ...
  -> PASS: all runs produced the same structure

==================================================
TOTAL: 16/16 checks passed

Regarding its implementation, note a few points:

• Trigger tests only send the description as routing information to the model, then ask "Should this Skill trigger?" Thus, it directly tests the most important text in automatic matching;
• Functional and consistency tests take the body of SKILL.md as instructions, then use ordinary code to check the output against some success criteria, such as whether sections are complete, whether content is too long, and whether an ISO 8601 date appeared when no date was provided;
• The script reads the real SKILL.md, so it doesn't maintain a separate copy of instructions that could easily get out of sync with the actual Skill.

It's not a complete production-grade Skill Eval, much less a universal Skills API for all platforms. It's just a test Harness you can run quickly after each modification. For example, its check for "fabricated dates" mainly relies on the YYYY-MM-DD format and a few "not stated" type keywords, which might still miss fabricated dates in natural language like "March 10th"; if this is a critical risk, stricter checks or manual review are needed.

Also, its trigger tests simulate routing logic and cannot fully replace end-to-end testing in a real Agent host. Different hosts may have differences in Skill discovery, context injection, tool calls, and permission handling.

However, it's already sufficient for quick regression. The test cases are just some lists at the top of the script; expanding them later is simple: add an expression, add a check, and re-run.

When a test fails, it also tells us where to adjust.

How to Interpret Different Failure Results

Testing isn't about seeing a row of green PASS results; it's about pointing out problems.

Should trigger, but didn't, i.e., undertriggering. Usually, first check if the description is too vague or lacks expressions real users would use. Add more specific trigger phrases; for technical Skills, include precise keywords and file types.

Unrelated requests also trigger, i.e., overtriggering. Here, first check if the description's scope is too broad. Further clarify the task scope and specify which adjacent scenarios should not use it.

Triggers correctly, but output is wrong or inconsistent. Here, the problem is usually not in the description, but in the specific instructions. Tighten the operational steps, move key rules to a more prominent position, and clearly state what a correct result should look like.

Testing isn't meant to pass everything on the first run. What truly matters is: the moment something regresses, we can discover it in seconds and know where to start checking.

A Skill is inherently a document that changes continuously. After release, real inputs will bring situations we didn't anticipate, and we'll come back to refine the instructions. A test suite makes this cycle faster and safer.

To Be Continued

Now, we have created a Skill and preliminarily verified that it can trigger correctly, generate results meeting requirements, and maintain structural consistency across multiple runs.

Only one last step remains: how to take it out of its own directory and give it to others to use.

Next, we will continue discussing:

• How to correctly share a Skill;
• How to manage versions to avoid updates breaking other people's workflows;
• How to deploy within a team;
• How to avoid issues that silently degrade a Skill's reliability over long-term use.

Actually, the original problem that started this short series was that we always had to repeat the same requirements in every conversation. Now, we can create a workflow, prove it's reliable enough, and confidently hand it over to others.

That's what we ultimately want to do.