跪拜 Guibai
← Back to the summary

A 5-Agent Pipeline That Replaces the Flakiest Parts of UI Automation

Almost every team that has done UI automation has experienced this death loop:

3 days to write scripts, 2 weeks to fix them.

This is the reality of UI automation. It is also the most "fragile," most "expensive," and most in need of AI empowerment in the entire testing system.

So, can we use Agent Skills to string together the entire chain of UI automation, from page parsing to script maintenance?

The answer is yes. And the effect is much better than you might imagine.

The core idea can be summed up in one sentence: AI is responsible for parsing, generation, enhancement, and maintenance; humans are responsible for verification and decision-making.

But there is a key issue here: you cannot create a "universal Skill"; you must split responsibilities into "specialized Skills."

This article will give you a complete view of how a set of 4+1 Agent Skills strings together the entire UI automation process.

1. The Pain of UI Automation Goes Far Beyond Grabbing Elements

When many teams mention the pain points of UI automation, their first reaction is "grabbing elements is slow." But in reality, grabbing elements is just the tip of the iceberg.

From page parsing to script maintenance, every link in UI automation has maddening aspects:

Link Pain Point Typical Manifestation
Page Parsing Manually grabbing elements one by one is slow and prone to omissions A page with 100+ elements, checking attributes and copying locators one by one, can't be finished in 3 days
Script Writing POM + test cases + data, all handwritten from scratch Page object classes, test cases, and test data are all manually written, extremely inefficient
Script Execution Lack of waits, lack of exception handling, always failing Demo-level scripts can run, but fail on CI—timeouts, popups, iframes, sporadic failures
Visual Verification DOM assertions cannot detect style issues Elements are present, text is correct, but layout is misaligned, styles are broken, and all assertions pass
Long-term Maintenance Frontend redesigns cause large-scale script failures Element attributes change, components are replaced, previous location strategies are all scrapped
Team Collaboration Everyone has different styles, quality is uneven Element grabbing methods, location strategies, naming conventions are all written differently, maintenance cost is extremely high

A single Skill cannot solve the pain points of these six links.

A pitfall many beginners easily fall into: wanting to make a "universal Skill"—input a page URL, directly output a perfect script. One skill takes care of page parsing, element location, script generation, visual assertions, and execution maintenance, resulting in bloated logic, difficult maintenance, and limited scalability.

image.png

The correct approach is to split by responsibility, with each Skill doing only one thing, and doing it to the extreme.

2. The 4+1 Skill Full-Process Architecture

First, look at the big picture. The entire AI-empowered chain for UI automation consists of 4 core Skills + 1 optional Skill, forming a complete closed loop (actually, this isn't the entire closed loop yet; 7 other core skills like execution, self-healing, and report generation haven't been discussed):

Skill Core Responsibility Pain Point Solved
ui-page-parser Structured parsing of page elements Slow manual element grabbing, fragile locators, non-standard page information
ui-testscript-generator Batch generation of UI test scripts Slow manual coding, difficult POM standard implementation, non-uniform location strategies
ui-testscript-enhancer Script robustness enhancement Lack of wait mechanisms, exception handling, popup blocking, failure screenshots
ui-visual-assert Visual assertions and multi-browser adaptation Insufficient DOM assertions, high multi-browser compatibility costs
ui-auto-maintainer (optional) Intelligent maintenance and self-healing Script failures due to page changes, high maintenance costs
image.png

These Skills form a complete closed loop: Parse → Generate → Enhance → Adapt → Maintain, which can be used in series or called independently.

Page URL / DOM Structure / Test Case Description
  │
  ▼
ui-page-parser ──→ Standardized Page Object Definition (pages.yaml)
  │
  ├──→ ui-testscript-generator ──→ POM Classes + Test Scripts + Test Data
  │         │
  │         ▼
  │   ui-testscript-enhancer ──→ Robustness Enhancement (Wait+Exception+Popup+Captcha+Screenshot)
  │         │
  │         ▼
  │   ui-visual-assert ──→ Visual Assertions + Responsive Adaptation + Multi-browser Compatibility
  │         │
  │         ▼
  │   ui-auto-maintainer ──→ Page Change Detection + Locator Self-healing + Baseline Update
  │
  └──→ pages.yaml can also be directly used for frontend component documentation generation, accessibility audits

Why split it this way?

Three principles:

  1. Single Responsibility: Each Skill only does one type of core action (parse, generate, enhance, adapt, maintain), avoiding functional coupling.
  2. Closed-loop Connection: The output of the previous Skill is the input of the next Skill, forming a complete automation chain.
  3. Flexible Reuse: Each Skill can be called independently. For example, if you only want to sort out page elements, just use ui-page-parser alone; if you only want to enhance script robustness, just use ui-testscript-enhancer alone.

Next, let's break down the positioning and role of each Skill one by one.

3. Breakdown: The Positioning and Role of Each Skill

Skill 1: ui-page-parser — Structured Parsing of Page Elements

Positioning: The UI data preprocessing Skill, the foundation of the entire chain.

All subsequent script generation, enhancement, and visual assertions depend on the output of this step. If the parsing is inaccurate, everything that follows is unreliable.

img

What problem does it solve?

In the traditional model, test engineers need to open the browser DevTools, check element attributes one by one, copy XPath / CSS Selectors, analyze page loading timing, and sort out interaction flows. Facing single-page applications built with modern frontend frameworks, the dynamic generation and asynchronous loading of elements make this process particularly painful.

Manually sorting out a system with 20 pages, where each page averages 100 interactive elements, might take 3-5 days, and it's easy to miss dynamic elements, iframe nesting, and Shadow DOM.

Core Capabilities:

Input:

Output:

Standardized pages.yaml, containing basic information for each page, an element inventory (including multi-level locator strategies), interaction flows, and page states.

Core Value:

Replaces manual element grabbing and page structure analysis, compressing hours or even days of manual labor into minutes. The parsing logic itself is repetitive and specialized; encapsulating it as an independent Skill serves as standard input for all subsequent Skills and can also be reused in non-testing scenarios (like frontend component documentation generation, accessibility audits).

Skill 2: ui-testscript-generator — Batch Generation of Page Objects and Test Scripts

Positioning: Based on structured page definitions, batch generates POM classes, test scripts, locator strategies, and test data in one go.

img

What problem does it solve?

You have the pages.yaml in hand, but to turn these structured page definitions into actually runnable test scripts, the traditional way still requires manually writing POM classes from scratch, writing test cases one by one, and constructing test data one by one. A login page might be okay, but if it's 20 pages, with dozens of elements per page, and each element needs to cover positive/boundary/exception/security scenarios, the workload is explosive.

Core Capabilities:

A Key Design Decision: Merging Data Generation and Script Generation

Data construction and script writing in UI testing are highly coupled—the test data for the same form field directly drives the corresponding page operation steps. Therefore, merging the two into one comprehensive Skill fits the actual workflow of UI automation better than splitting them into independent Skills. Users only need to input page definitions and get a complete, runnable project in one step.

img

img

img

Input:

  1. pages.yaml output from ui-page-parser
  2. Team UI framework specifications (framework selection, directory structure, locator strategy priority, wait mechanism, assertion strategy)
  3. Test data rules (optional, for custom data construction rules)

Output:

Core Value:

Replaces manual writing of POM and test cases from scratch, completing the full output of "data + pages + test cases" in one step. Particularly suitable for rapid implementation, small projects, and beginners, offering quicker onboarding, simpler operation, and ready-to-use generation in one step.

Skill 3: ui-testscript-enhancer — Script Robustness Enhancement

Positioning: Automatically enhances basic scripts, evolving them from "demo-level" to "production-level."

img

What problem does it solve?

The scripts generated by ui-testscript-generator have standard structures and complete test cases, but lack the robustness logic required for production environments. Specifically: no intelligent waits (or hardcoded sleep), no popup blocking, no automatic iframe switching, no exception retries, no failure tracing, and captchas that completely block execution.

Demo-level scripts can run, but fail in various ways once they hit the CI environment.

Core Capabilities (Six Enhancement Dimensions):

Enhancement Dimension Problem Solved
Intelligent Wait Completion Replaces hardcoded sleep, automatically supplements page load waits, element visibility waits, Ajax async waits, animation transition waits, element state change waits
Popup and Interference Handling Automatically detects unexpected popups (ads, permission requests) → tries to close → continues execution; Toast messages automatically captured and asserted
iframe/Shadow DOM Handling Automatically identifies iframe nesting → switches context → locates internal elements; Shadow DOM penetration locating
Exception Retry and Fault Tolerance Element not found automatically retries 3 times → saves screenshot → logs; page crash automatically refreshes and recovers; network timeout automatically retries
Failure Tracing Enhancement Automatic screenshots (full-page screenshot on failure), automatic screen recording (Trace files), automatic network request log recording
Login Captcha Recognition Image captchas, slider captchas, text click captchas, math problem captchas, automatically recognized and processed

Input:

Output:

Enhanced UI automation test scripts + enhancement configuration files (e.g., enhanced_base_page.py, a base class encapsulating all enhancement logic).

Core Value:

Filters out the "fragility" problems of AI-generated scripts. A @retry_on_failure decorator can automatically retry failed test cases, and a ddddocr integration can prevent captchas from blocking the automation process. Scripts are no longer just "able to run," but "run stably."

Skill 4: ui-visual-assert — Visual Assertions and Multi-browser Adaptation

Positioning: Adds visual assertion capabilities to UI scripts, enabling cross-browser, cross-resolution compatibility verification.

img

What problem does it solve?

Traditional DOM assertions can only verify "is the element present" and "is the text correct," but cannot detect "does the page look right." Elements exist, text is correct, but the layout is misaligned, colors are wrong, responsive design is broken—DOM assertions still pass everything.

Even more troublesome is multi-browser compatibility. The same script runs fine on Chromium, but on Firefox, click coordinates shift; on WebKit, style rendering is inconsistent, and the adaptation cost explodes.

img

Core Capabilities (Three Dimensions):

Dimension One: Visual Assertions.

Dimension Two: Responsive Adaptation.

Dimension Three: Multi-browser Compatibility.

Input:

Output:

Cross-browser test scripts with visual assertions + baseline image library + difference comparison report.

img

Core Value:

Goes beyond traditional DOM assertions to achieve intelligent verification of "does the page look right." No longer requires manual, naked-eye checking across every browser and resolution; the Skill automatically runs all combinations and outputs a difference report.

Skill 5 (Optional): ui-auto-maintainer — Intelligent Maintenance and Self-healing

Positioning: A long-term operations Skill, responsible for page change perception, automatic locator strategy repair, and visual baseline updates.

What problem does it solve?

The number one killer of UI automation is not technical difficulty, but maintenance cost. Frontend redesigns are the norm—element attributes change, layouts are refactored, components are replaced, and previous locator strategies all become invalid. Without maintenance, scripts quickly become "one-off projects."

In the traditional model, maintenance relies entirely on manual work: periodically run scripts → discover large-scale failures → investigate one by one → manually fix locators → update baselines. Once this cycle starts turning, the maintenance workload grows larger and larger, eventually crushing the entire automation project.

Core Capabilities:

Input:

Output:

Core Value:

Solves the ultimate pain point of "high maintenance cost" in UI automation. After a frontend redesign, there's no longer a need to manually investigate failed scripts one by one; the Skill automatically detects changes, automatically repairs locators, and automatically updates baselines, enabling sustainable operation.

4. Beyond the 4+1 Core Architecture, Expand as Needed

The 4+1 Skills above cover the core closed loop of UI automation testing. If your team has special scenario requirements, you can supplement on top of this foundation:

Extension Skill Core Responsibility Applicable Scenario
Cross-platform Adaptation Skill Specifically handles script migration and adaptation for Web / H5 / Mini Programs Multi-platform business lines, needing one set of scripts to cover multiple platforms
Performance Testing Linkage Skill Automatically generates Lighthouse performance test configurations based on UI scripts Frontend performance monitoring, CI pipeline automatically runs performance regression
Accessibility Testing Skill Automatically generates axe-core accessibility audit scripts based on POM Compliance requirements, implementation of information accessibility standards

Core Principle: First implement the four most core Skills to achieve a closed loop, then expand based on the team's actual scenarios, avoiding over-design.

5. Full Process Chain Review

Stringing the entire chain together in a command-line style looks like this:

# 1. Start Parsing (one entry URL, automatic full-site traversal)
/ui-page-parser Please capture all pages of http://localhost:3000/

# 2. First Stop: Page Parsing
   ├─ BFS crawler traverses the entire site → discovers all pages
   ├─ Requires authentication? → Launch Chrome, CDP reuses login state
   ├─ Extract elements page by page → generate structured definition for each page
   ├─ Automatic screenshots → save a visual snapshot for each page
   └─ Output pages.yaml ← Standardized page object definition

# 3. Second Stop: Script Generation
   └─ pages.yaml → ui-testscript-generator
      ├─ Intelligent test data construction (positive/boundary/illegal/null/injection)
      ├─ Automatic POM class generation (including intelligent locator strategy)
      ├─ Automatic test case generation (including multi-dimensional assertions)
      └─ Output pages/ + testcases/ + data/ ← Complete project structure

# 4. Third Stop: Robustness Enhancement
   └─ Basic scripts → ui-testscript-enhancer
      ├─ Intelligent wait completion (replaces hardcoded sleep)
      ├─ Popup blocking + automatic iframe switching
      ├─ Exception retry + failure screenshot/screen recording
      ├─ Automatic captcha recognition
      └─ Output enhanced scripts ← Demo-level evolves to production-level

# 5. Fourth Stop: Visual Assertions and Multi-browser Adaptation
   └─ Enhanced scripts → ui-visual-assert
      ├─ Visual assertions (full-page + local screenshot comparison)
      ├─ Responsive adaptation (desktop/tablet/mobile)
      ├─ Multi-browser compatibility (Chromium/Firefox/WebKit)
      └─ Output cross-browser scripts + baseline image library ← Visual-level verification

# 6. Fifth Stop (Optional): Intelligent Maintenance
   └─ Production scripts → ui-auto-maintainer (triggered periodically)
      ├─ Page change detection (DOM diff)
      ├─ Locator strategy self-healing
      ├─ Visual baseline update
      └─ Output maintenance report + repair suggestions ← Sustainable operation

6. AI Does the Work, Humans Do the Checking

There is a key issue that must be clarified here: All results generated by AI are not ready to use directly; they require manual verification.

This set of 4+1 Skills can help you complete actions like "parsing, generation, enhancement, adaptation, maintenance," compressing weeks or even months of manual labor into minutes or hours. But the following things cannot be done by AI and still require human oversight:

Things AI is Responsible For Things Humans are Responsible For
Automatic full-site page traversal Confirm whether the traversal scope is complete (are any key pages missed)
Element locator strategy derivation Verify whether the locator strategy is reasonable (check against the real page)
Intelligent test data construction Confirm whether data rules cover core business scenarios
Script robustness enhancement Confirm whether wait strategies and exception handling match real interactions
Visual assertions and multi-browser adaptation Confirm whether baseline images are accurate and tolerance thresholds are reasonable
Page change detection and self-healing Confirm whether the self-healed locator is correct (not just "happens to find it")

To put it plainly, AI is responsible for doing the manual labor from "0 to 80," and humans are responsible for the quality check from "80 to 100." This way, it's efficient without losing control over quality.

Special Reminder: During the debugging and initial use phase of the Skills, it is recommended to open the real webpage, pick a few key pages and key elements, and use developer tools to check whether the AI-generated pages.yaml, POM classes, and locator strategies match the actual elements on the page, ensuring the authenticity and accuracy of the parsing results.

7. Skill Source Code and Complete Tutorials

You can develop Skills yourself based on the ideas and architecture provided in this article. If you need ready-made tutorials and Skills, you can also join "Kuangshi . AI Evolution Club" to get them. Inside, there are various hands-on, step-by-step graphic and video tutorials for implementing AI technology, including practical tutorials for the entire process of AI-empowered testing (hand-holding, spoon-feeding tutorials; follow the steps, and even zero-basics can quickly get started. Currently contains 30+ Agent Skills for full-scenario AI testing).

Friendly reminder, "AI Testing" is just one of the eight major skill sections of the AI Evolution Club.

Final Words

Let's review the entire architecture:

Pain Point: From page parsing to script maintenance, every link in UI automation is time-consuming and labor-intensive, and highly dependent on manual experience.

Solution: Don't create a universal Skill; split into 4+1 specialized Skills by responsibility, each doing only one thing, forming a complete closed loop of Parse → Generate → Enhance → Adapt → Maintain.

Effect:

Traditional Model Agent Skill Model
Manually grab elements for 3-5 days One entry URL, full-site traversal in minutes
Handwrite POM + test cases + data, weeks One-step batch generation, in minutes
Demo-level scripts, fail on CI Automatically complete waits/exceptions/popups/captchas, production-level
DOM assertions can't detect style issues Visual assertions + multi-browser × multi-resolution automatic coverage
Frontend redesign → scripts all broken → manual fix Change detection + locator self-healing + baseline update

Boundary: AI is responsible for parsing, generation, enhancement, adaptation, and maintenance; humans are responsible for verification and decision-making.

This 4+1 Skill architecture is not a theoretical design, but a solution actually being used by members of "Kuangshi . AI Evolution Club." Many students have reported that the implementation efficiency of UI automation testing has significantly improved, and they are no longer tormented by element locating, script writing, and maintenance costs.

If you want to delve into the practical details of a specific Skill, you can look at the individual breakdowns in this series:

  1. ui-page-parser: Page Element Parsing — One entry URL, automatic full-site traversal
  2. ui-testscript-generator: Batch Script Generation — One step to generate POM + test cases + data
  3. ui-testscript-enhancer: Robustness Enhancement — Waits + Exceptions + Popups + Captchas + Screenshots
  4. ui-visual-assert: Visual Assertions and Multi-browser Adaptation — Visual-level verification + cross-browser compatibility