A Single Orchestration Skill Turns Four Separate AI Calls Into One UI Test Pipeline

Hello everyone, I'm Kuangshi.

In the first three articles, we handed over the three stages of UI automation to AI one by one. "Running tests" got ui-test-executor, "fixing scripts" got ui-failure-diagnoser, and "generating reports" got ui-report-generator.

Each stage performs well on its own. But when you actually use them, a new problem emerges.

To run a complete round of testing, you first call the execution skill once, check the results, then call the failure diagnosis skill once, fix and rerun the execution skill once, and finally call the report generation skill. One chain, a person has to call AI four times, and in between must judge the timing for connecting them.

The work is done by AI, but the chain is strung together by a person. Human attention is still tied to this chain.

Foreword

This series has been doing one thing: using Agent Skills to automate each stage of UI automation testing step by step.

By the previous article, the stage-level Skills were all in place. Execution has evidence, failures have conclusions, reports have judgments — all single-point capabilities are present.

What was missing was something to string them together. When to execute, whether to diagnose after a failure, whether to rerun after fixing, how many rerun rounds, how to calculate results — these decisions between stages previously relied on on-the-spot human judgment.

So for the final article of this series, I created ui-pipeline-scheduler, a Skill for unified orchestration of the entire chain. It doesn't play as a player, only as a conductor, orchestrating the previous Skills into an automated pipeline.

This article focuses on this Skill, starting from the problems it faces, to how it solves them, and then running it in practice.

Why Orchestration Is Needed When Independent Execution Skills Already Exist

Breaking down the matter of "manual chaining", the problems often concentrate in a few main areas.

1. One chain, Skills need to be executed four times separately.

Execution, diagnosis, rerun, report — each step is initiated separately. The tester becomes a messenger between AIs, switching windows a dozen times to run one chain.

2. Connection timing relies on human monitoring.

After running, you need to check if there are failures; only if there are failures do you invoke diagnosis. After fixing, you need to check if it's fixed; only if it's fixed is it worth rerunning. The "next step" at each stage requires human judgment. If the person walks away, the chain breaks.

3. No rules for retry rounds.

Fix one round, run one round; if it fails, fix again and run again. When is it enough? Without an upper limit, it's an infinite loop; setting an arbitrary limit still wastes time on unfixable cases.

4. Multi-round results conflict with each other.

Reruns usually only run the failed cases, so new execution results overwrite the old ones. With 8 cases in the first round, after rerunning 3, the result file only contains 3. Using this directly to generate a report gives completely incomplete numbers.

5. Midway errors scrap the entire chain.

If the diagnosis stage throws an error, everything after it breaks, and the previous run results are left uncollected. Either start over from scratch or manually piece things together.

6. Cannot connect to CI.

A pipeline requires a single entry point, one command. Several skills are manually operated and simply cannot be implemented in CI.

These issues share a common trait. The sequence is fixed, the conditions are clear, the boundaries are distinct. All are tasks AI excels at.

So, can AI take over the step of "stringing the chain" as well?

The answer is, yes. And once this step is taken over, the value of the previous Skills truly connects into a whole.

Introduction to the ui-pipeline-scheduler Skill

Simply put, ui-pipeline-scheduler is an orchestration-layer Skill that automatically strings the four stages of execution, diagnosis, retry, and reporting into a closed loop.

It doesn't do the work itself; it only schedules the preceding Skills according to rules.

Input:

• Test project directory (containing tests/, pages/, pages.yaml)
• Execution parameters (priority, tags, browser, parallel count, etc.)
• Retry limit (default 2 rounds)

Processing logic:

1. Environment self-check: Confirm project structure integrity and Python environment readiness;
2. First-round execution: Call ui-test-executor to run the specified scope, back up first-round results;
3. Conditional diagnosis: Only call ui-failure-diagnoser if there are failures; if all pass, skip directly to report;
4. Targeted retry: Only rerun failed cases, verify after fixing, loop judgment;
5. Result merging: Merge first-round and all retry-round results into one complete dataset;
6. Final report: Call ui-report-generator, integrate all artifacts to produce the report.

Output:

• A final HTML test report integrating all information from execution, diagnosis, and retries
• Archived results and diagnosis reports from each round

Applicable scenarios:

• One-click full UI automation run (execution + diagnosis + retry + report)
• Complete regression closed loop before version release
• Automatic diagnosis and repair of failed cases followed by automatic retry verification
• Connecting to CI/CD, single entry point triggering the entire test pipeline

Its entire design follows two principles.

Zero intrusion, only acts as conductor. It does not modify any code, input parameters, or output parameters of the preceding sub-Skills; it only does three things: pass parameters, read artifacts, control sequence. This principle is even more important when viewed in reverse: the three sub-Skills can be called independently at any time; fitting into the pipeline does not affect standalone use, and standalone use does not depend on the pipeline.

Stop when needed, never idle. Every loop cycle has clear exit conditions; if it can't be fixed, stop the loss. What a pipeline fears most is not slowness, but meaningless idling.

In terms of process, it solidifies into a five-stage chain.

Execution → Diagnosis → Retry → Merge → Report

Let's look at how it dismantles the previous problems step by step.

1. One command, five stages

For the user, there is only one entry point. One sentence: "One-click full run of P0 smoke tests, automatic failure diagnosis and repair retry, finally generate report." The rest — execution, diagnosis, retry, merge, report — the five stages proceed in sequence on their own.

Previously, a person had to execute AI four times separately; now, the person only says one sentence.

2. All-green express lane

If the first round passes completely, it's the best-case scenario for the pipeline, and also the scenario that should least be wasted. In this case, diagnosis and retry are skipped directly, going straight to the report.

No stages are run empty just for "process completeness"; every stage's trigger has an actual condition.

3. Diagnosis and repair, targeted retry

If the first round has failures, it enters the diagnosis stage. After fixing, it doesn't rerun the full set; it only pulls out the failed cases for targeted rerun — fast, and without introducing variability to already-passed cases.

After running, check the results again; if fixed, continue converging; if there are still failures, enter the next round of judgment.

4. Circuit breaker safety net, absolutely no infinite loops

Three conditions trigger immediate shutdown. Retry round limit reached; this round of diagnosis fixed none, so rerunning is futile; or all cases passed, mission complete.

After shutdown, if failures still remain, they won't be silently swallowed; they will be clearly marked in the results: these cases remain unfixed after N rounds of repair, with a list of cases attached. These cases are where human intervention is needed.

5. Multi-round merging, numbers without distortion

This is the most hidden pitfall of the entire pipeline, and also the most valuable detail of the orchestration layer.

Retries only run failed cases, so the result file gets overwritten, leaving only a small set. Directly using this to generate a report, out of 8 cases in the first round, only 3 remain in the report — all numbers are distorted.

So results from each round are archived. Before the final report is generated, a merge is performed, using the complete first-round results as the base and overwriting each case with the latest status from each retry round. Not a single case is lost, and all statuses are the latest.

Skill Practical Demonstration

Still the shop-lab project. Select ui-pipeline-scheduler from the skill list, and input one command.

/ui-pipeline-scheduler One-click full run of shop-lab's P0 smoke test cases,
automatic failure diagnosis and repair with retry, finally generate a complete report

The detailed execution process is as follows:

The UI automation full process execution is complete, and the process results are as follows:

Open the latest generated test report to confirm the effect:

At this point, we have connected "Execution → Diagnosis → Retry → Merge → Report" through the one-click scheduling UI automation test pipeline skill ui-pipeline-scheduler.

In the vast majority of scenarios, you only need to call this skill.

Summary

First, let me share my genuine feelings.

Writing this series up to the fourth article, what I most want to share is actually a realization. A single Skill is capability; orchestration is productivity. Execution, diagnosis, report — each stage viewed alone only saves a period of time. Once strung into a pipeline, what changes is the way of working — the tester transforms from "an operator monitoring every stage" to "a decision-maker who sets parameters and reads the report." This role change is worth far more than the hours saved.

Another layer of insight lies in the design. The easiest wrong direction for an orchestration layer is to grow fatter and fatter, absorbing the logic of several Skills, ultimately becoming a monolithic block that no one can disassemble. ui-pipeline-scheduler does the opposite: zero intrusion, only passing parameters, reading artifacts, controlling sequence, keeping sub-skills completely independent. They can be combined because each is specialized; only by being specialized do they qualify for combination.

Of course, boundaries must be clearly stated. The pipeline automates the process; the right of judgment still rests with the person.

What AI is responsible for	What the person is responsible for
Five-stage sequential scheduling, conditional triggering	Defining execution scope (what to run, how many rounds)
Diagnosis and repair loop, targeted retry	Reviewing circuit-breaker-marked "unfixable" cases
Circuit breaker stop-loss, never idle	Confirming real defects, driving fixes
Multi-round result merging, numbers without distortion	Reading the final report, making release decisions

A special reminder. Circuit breaking is not failure; it's division of labor. The pipeline explicitly tells you "these cases cannot be fixed by the machine," which is precisely a sign of its reliability; the remaining judgment inherently belongs to the person. Don't treat a one-click pipeline as a shield for unattended operation; the ⚠️ mark in the report is always worth clicking open to take a look.

Reviewing the entire process.

Pain points. One chain calls AI four times, connection relies on human monitoring, no rules for retries, multi-round results conflict, midway breakage scraps the entire chain, cannot connect to CI.

Solution. ui-pipeline-scheduler solidifies the process into a five-stage closed loop of "execution, diagnosis, retry, merge, report." All-green express lane, failures enter loop, circuit breaker has safety net, merging without distortion, single entry point with one command.

Effect. Previously, one complete regression required a person to monitor and string together four stages. Now, start with one sentence, come back to directly read the final report; the intermediate loops, stop-loss, and merging are all automatic, and CI can connect directly.

Boundaries. AI is responsible for orchestration and execution; the person is responsible for parameters and decisions. The pipeline paves the road; whether to walk it and how to walk it is still up to the person.

Everyone can develop skills based on the ideas provided in this article. If you need ready-made tutorials and skills, you can also join "Kuangshi . AI Evolution Society" to obtain them. It contains various nanny-level illustrated tutorials and video tutorials for AI technology implementation, including practical tutorials for the full AI testing process (nanny-level hand-feeding tutorials; follow the steps, zero foundation can quickly get started, currently containing over 30 Agent Skills for full-scenario AI testing).

Friendly reminder, "AI Testing" is only one of the eight major skill sections of the AI Evolution Society.

At this point, the four puzzle pieces of "run, fix, report, string" are assembled, and the entire chain of UI automation from execution to decision-making is officially running through.

Connect this chain into enterprise CI/CD; code is submitted, tests run automatically, reports are pushed automatically. Let your pipeline truly flow. Buddy, have you learned it? See you in the next series.