Alibaba Open-Sources Skill-Up, a Test Harness for Agent Skills

Background

In 2026, the popularity of Skills remains undiminished and has become standard in the AI-assisted programming field. In daily development, whether distilling personal experience or standardizing a team's best practices, "distilling" knowledge into a Skill is already routine. But the question that follows is: who defines whether a Skill is "good" or "bad"? And what are the criteria for judging it?

With this confusion, I recently reviewed a large amount of technical material and have been searching for a quantifiable evaluation system. Coincidentally, Alibaba recently open-sourced an evaluation tool called Skill-Up—it is specifically aimed at Agent Skill developers, supports verifying the functional correctness of Skills in real Agent Engine environments such as Claude Code, Codex, and Qoder CLI, can turn failed test cases into precise repair guidance, and supports continuous regression in both local and CI environments.

Thus, the following practice was born.

What is Skill-Up

Skill-Up is a command-line evaluation tool for Agent Skills. The specific directory structure is as follows

my-skill/
  SKILL.md              # Your Skill definition
  evals/                # Evaluation directory
    eval.yaml           # Evaluation entry configuration
    cases/              # Test case directory
      basic-test.yaml   # An evaluation case
      edge-case.yaml    # Another evaluation case
    fixtures/           # Test resources (optional)
      repos/            # Repository templates
      scripts/          # Evaluation scripts

Installation

Using Skill-upper, which is the recommended method

# Codex, global installation
npx skills add https://github.com/alibaba/skill-up/tree/main/skills/skill-upper -g -a codex -y

You can also install via the command line, which allows you to run purely with commands, also very satisfying

curl -fsSL https://raw.githubusercontent.com/alibaba/skill-up/main/install.sh | bash

Key commands

# Run all test cases
skill-up run ./evals/eval.yaml

# Run only matching test cases
skill-up run ./evals/eval.yaml --include-case-name "basic-*"

# Exclude specific test cases
skill-up run ./evals/eval.yaml --exclude-case-name "*-old" --exclude-case-name "*-deprecated"

# Specify Engine and model
skill-up run ./evals/eval.yaml --engine codex --model openai/gpt-4

# Temporarily override the number of parallel test cases
skill-up run ./evals/eval.yaml --parallelism 4

# Enable baseline comparison
skill-up run ./evals/eval.yaml --baseline

# Generate reports in multiple formats
skill-up run ./evals/eval.yaml --format json --format html --format junit

# Run 3 consecutive rounds, writing artifacts to iteration-1/ through iteration-3/
skill-up run ./evals/eval.yaml --iteration 3

# Auto-detect mode (directly consumes Anthropic evals.json)
skill-up run --auto
skill-up run --auto --engine codex
skill-up run ./my-skill/ --auto

This indicates successful installation It can also be invoked through Codex

Create and Run the First Set of Evaluations

Demo evaluation report results