AI · Testing · Backend

api-scenario-runner Executes Full Business-Chain API Tests with Three-Level Assertions and CI-Ready Compilation

By 狂师 · Oct 29, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

End-to-end API testing now has an AI-driven path that treats a whole business flow as the unit under test, not isolated endpoints. That means fewer surprises in production where every endpoint works but the checkout still fails, and it turns scenario definitions into executable, CI-friendly pytest code.

Summary

All the single-interface tests can pass while the business flow still breaks. api-scenario-runner closes that gap by treating the whole chain from registration to order query as one test scenario. It reads a scenario YAML file, sends requests in sequence, auto-extracts and injects variables, and adds Bearer token headers where needed.

Assertions run at three levels: status code, business code, and JSONPath, plus a terminal DB state check that only allows SELECT with parameterized SQL. Failure stops execution and a snapshot file records the real request, response, and variables after injection, with sensitive values masked. Failures classify into five categories: missing variable, assertion failure, API error, connection error, and DB error.

Two execution modes exist: direct run and compile to a pytest project. Compiled scripts drop into the existing test hierarchy, schedule through api-test-executor, and feed a pipeline that includes cleanup and reporting. A dedicated idempotency round proves a second run goes green without changes, thanks to dynamic timestamp usernames and proper data cleanup, separating CI-viable tests from flaky ones.

Takeaways

— api-scenario-runner executes scenario YAML files, sending requests in order and auto-injecting variables and Bearer tokens.

— It uses three-level assertions: status code, business code, and JSONPath, followed by DB terminal-state assertions that only run SELECT and parameterized SQL.

— On failure, it stops immediately and saves a steps_snapshot.json with injected request, response, and variable views, with sensitive values masked.

— Failures are classified into five categories: VARIABLE_MISSING, ASSERTION_FAILED, API_ERROR, CONNECTION_ERROR, DB_ERROR.

— It supports compile mode that generates a pytest project (test_scenario_<id>.py and data/scenario_<id>.yaml) reusing the project's utils/config.

— The shopping flow scenario includes 14 API call steps, all passing in the demo, plus 4 DB terminal checks.

— Idempotency is ensured by dynamic timestamp-based usernames and cleanup through api-testdata-cleaner, so a second run stays green.

Conclusions

Moving from single-interface tests to whole business chains catches integration failures that unit-style API tests miss, like a token that doesn't propagate from login to order.

The idempotency round is the real gate for CI adoption; most test suites fail on repeated runs due to state pollution, and dynamic data plus cleanup is the solution.

Compiling scenario YAML to pytest bridges declarative AI-driven test design with existing test infrastructure, letting teams adopt the approach without abandoning their current runners and reporters.

Concepts & terms

scenario YAML file

A declarative definition of a business-flow test: the sequence of API calls, variable extraction and injection rules, assertion conditions, and optional DB terminal checks.

three-level assertion

Step-level checks that verify HTTP status code, business response code, and JSONPath values in that order, before a final database state assertion runs.

steps_snapshot.json

A diagnostic file saved on failure that captures the actual injected request, response, and variable values for each step, with sensitive fields masked, to help debug why a scenario failed.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗