Alibaba Open-Sources Skill-Up, a Test Harness for Agent Skills
Agent Skills are becoming the new unit of developer knowledge, but without testing they rot silently. Skill-Up gives teams a way to gate Skill changes in CI the same way they gate code changes, catching regressions before they mislead an agent in production.
As teams distill coding conventions and domain knowledge into reusable Agent Skills, the question of what makes a Skill good has remained subjective. Skill-Up provides a concrete answer: a YAML-driven test harness that executes Skills against defined cases inside actual agent runtimes, then reports pass/fail results with repair guidance. Failed cases produce actionable fix pointers rather than opaque logs.
The tool fits into a standard project layout—a SKILL.md definition alongside an evals/ directory containing test cases and optional fixtures. It supports filtering by case name, baseline comparisons, multi-format reporting (JSON, HTML, JUnit), and iteration runs for stability checks. An auto-detect mode can consume existing Anthropic evals.json files directly.
Installation works through a companion Skill called Skill-upper or a shell script, and the tool targets Claude Code, Codex, and Qoder CLI as execution backends. The output is a structured report that turns Skill quality from a debate into a measurable, repeatable pipeline step.
Skill quality has been a hand-wavy conversation; Skill-Up shifts it to a deterministic, test-driven discipline that mirrors how software testing matured.
The tool's design assumes Skills are stable enough to be tested like functions, which implies the ecosystem is crossing a threshold where Skills are treated as build artifacts rather than ad-hoc prompts.
By targeting multiple agent engines, Skill-Up sidesteps vendor lock-in and forces Skill authors to confront cross-runtime compatibility early.