跪拜 Guibai
← All articles
Interview · Testing · Artificial Intelligence

Six Interview Questions That Expose Whether Your AI Test-Case Agent Is Real

By 狂师 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Teams building AI test-generation agents will hit every one of these failure modes in production — input sensitivity, correctness blind spots, knowledge-base rot, and hallucination — and the interview's layered rule-first answer is a working blueprint for keeping LLMs out of the decision path.

Summary

An AI test-case generation agent looks great on a résumé until an interviewer asks how the efficiency numbers were measured. A recent interview debrief works through six such questions, each with the reasoning behind what the interviewer wants to hear. The first: the claim that case writing dropped from 68 to 23 hours. The answer isn't a formula; it's a controlled comparison using the same team, same complexity class, same end-to-end window (requirement review to case review), and excluding agent run time from human hours. Then the questions move to quality: how to guarantee executable and complete cases, and how to separate coverage (did we hit the branch?) from correctness (is the expected logic right?). The proposed architecture leans on structural metadata, rule engines, and a separate review agent that only flags doubts, with humans keeping final authority.

The harder questions come after: what are the agent's own limitations, how do you keep correctness at scale, and why use an LLM for validation when it hallucinates. The answers admit real costs: input quality is the ceiling, rule and knowledge bases become technical debt, multi-service and timing-dependent scenarios trip causal reasoning, and human reviewers still sign off on logic. At scale, the solution is tiered admission (low risk auto-pass with sampling, high risk forced dual review), diff-based review for iterative changes, and a feedback loop that captures execution-time case errors as negative few-shot samples. The hallucination question gets a clear stance: LLMs only possess the right to raise doubts, never to pass or reject; hard rules gate everything financial or permission-related.

Takeaways
— The 68-hour baseline and 23-hour post-agent figure are measured under a controlled comparison: same team, same complexity class, and the same end-to-end window from requirement review completion to test-case review approval, with agent run time excluded from human hours.
— Executability and coverage are handled separately; executability comes from fixed output specs, metadata references, and static checks, while coverage starts with a control-flow / branch decomposition checklist that the agent generates before writing any case.
— A built-in boundary-value knowledge base covers numeric, length, time, array, pagination, and ID types with normal, min, max, critical, out-of-bounds, empty, null, empty-string, special-character, and overlong strategies.
— Coverage reconciliation compares the branch checklist against generated cases to surface missing scenarios explicitly, rather than depending on the model's own judgment of what it covered.
— Correctness and coverage are two separate chains: four-layer logic validation uses structural metadata, a business rule engine, a review sub-agent that asks three fixed questions, and human tiered review.
— The agent's admitted limitations include input-quality dependency, maintenance burden of rule and knowledge bases, unstable reasoning in multi-service and timing-dependent scenarios, validation blind spots, paper-correct cases that fail in real environments, human skill degradation, and output instability.
— At scale, risk-tiered admission is used: L1 low-risk auto-passes with random spot-checks, L2 requires single review plus blind rechecks, and L3 mandates dual-person expert review for funds, permissions, settlement, and core transactions.
— Same-requirement diff comparison shows only added, modified, or deleted cases so reviewers focus on changes rather than re-reading stable historical cases.
— A post-hoc feedback loop classifies execution-time mismatches as either program bugs or case errors; case errors enter a few-shot sample library and trigger alerts, while online defects are reverse-traced to coverage gaps.
— The hallucination strategy gives the LLM only alert-raising power, not pass/reject authority. Hard rules remain the highest-priority gate, and funds, permissions, and core transactions never rely on LLM review.
— The closing advice from the interviewer's viewpoint: articulate the process and problems solved, admit limitations openly, and make the statistical definition and comparison clear when a number is challenged.
— A dedicated review sub-agent does not generate cases but only audits, asking whether preconditions, steps, and expected results contradict, whether upstream/downstream effects are reasonable, and whether implicit risk rules are violated.
Conclusions

The interviewer's approach treats an AI efficiency claim like any engineering metric: without a reproducible control group and an honest accounting of what the number excludes, the figure is worthless.

The answer architecture consistently assigns final authority to deterministic rules and humans, treating LLM output as drafts and suspicions, which mirrors the production reality for AI agents: the model proposes, the pipeline disposes.

The candid limitation list — knowledge base maintenance, human skill atrophy, and cases that pass all checks but can't run in a real environment — names the failure modes that rarely appear in vendor blogs but dominate operational cost.

At scale, the solution shifts from model capability to process design: tiered admission, diff-based review, and negative-sample feedback loops, suggesting sustained AI quality is an operations problem more than a model problem.

The red line that no financial or permission-sensitive check ever goes through an LLM is a defensible rule of thumb for any AI-assisted quality gate, and it quietly defines what the model is actually for.

Concepts & terms
Equivalence-class partitioning and orthogonal testing
A strategy to control combination explosion in test-case generation: group inputs into equivalence classes and use orthogonal test design to select representative combinations, prioritizing high-risk combinations and sampling low-risk ones instead of enumerating everything.
Coverage reconciliation
A mechanical step that compares a pre-generated branch and state checklist against the actual cases produced, so missing scenarios become explicit rather than being left to the model's own judgment of what it covered.
Risk-tiered admission (L1/L2/L3)
A classification scheme for test-case generation jobs by business risk and complexity, determining the level of human review required: low-risk auto-pass with random spot-checks, medium-risk single review plus blind recheck, and high-risk forced dual-person expert review.
Negative few-shot samples
Storing wrong-case examples discovered during execution or review back into the agent's prompt context or knowledge base, so the model learns to avoid similar errors on subsequent generations.
From the discussion
Featured comments
陈皮白茶

Awesome. But mainly, are there still testers nowadays? And this set isn't suitable for small companies, right? Small companies don't have testers as a safety net.

See top comments, translated →
Source: juejin.cn ↗ Google Translate ↗ Backup ↗