Six Interview Questions That Expose Whether Your AI Test-Case Agent Is Real
Sharing 6 AI Test Questions from a Real Interview, Explained One by One (Come Take a Look)
Author: 狂师
Tags: Interview, Testing, Artificial Intelligence

A while ago, a fan of mine went to an interview and felt he hadn't answered well. He came to me, hoping I could help him review it. When I looked at the questions, several were quite typical, so I picked a few out to share with everyone.
The interview was about his test-case Agent project. The interviewer asked in detail, from how the efficiency numbers were calculated to what shortcomings the solution itself had. For each question, I analyzed it from the interviewer's perspective: what the interviewer wanted to hear and how to answer it on point. You can think about your own answer first, then compare it with my thinking.
The candidate's answers are not shown, at the fan's request ~
Interview Question 1
Interviewer asks: For the test-case Agent, case writing time dropped from 68 hours to 23 hours (as mentioned in the résumé). How was this figure calculated?
What is the interviewer testing with this question?
Asking this question is not just about the calculation formula; this is a typical follow-up probing 'data authenticity / rigor'. What the interviewer cares about is not the number itself but whether you made it up on a whim, whether you have a clear, reproducible statistical definition, and whether you're honest about data limitations instead of blindly exaggerating gains.
Complete verbal answer
Hello interviewer, for this duration we did controlled statistics against requirements of similar complexity:
- Baseline 68 hours: This is the baseline before we introduced the Agent, taken from historical work hours for the same type of task in the project management platform (e.g., Jira/Zentao). We selected 2-3 iterations before the introduction, the same team, and requirements of similar complexity as the sample, e.g., 50 requirements with 500 test cases total. We measured the end-to-end time from requirement review completion to test-case review approval, not just the 'typing' time — it includes test-case design, writing, self-check, and review rework.
- 23 hours: After introducing the test-case Agent, we measured under the same definition: the same team, requirements of similar complexity, the same from requirement review completion to test-case review approval. The data comes from task work-hour records + test-case review records. What is counted is the actual human hours invested in this process; the time spent on automatic generation by the Agent is not included in the human work hours.
Moreover, to minimize interference, we intentionally did several things to ensure comparability:
- Compare same-class requirements: We fixed requirement scope and business module, graded requirements by complexity (e.g., number of interfaces, number of pages, number of scenarios), and only compared against requirements of the same grade.
- Same team: to eliminate differences in personnel proficiency.
- Multi-iteration observation: The 23 hours is a stable value from the most recent N iterations, not the best one.
When answering, you can also proactively break down where the time goes (showing depth)
For example, in the original 68 hours, design was about 20h, writing about 30h, and review rework about 18h. The Agent mainly compressed the writing stage (30h→8h), while design and review still rely mainly on people. So the essence of the improvement is human-machine division of labor, not AI replacing everything.
If a candidate can proactively break this down, it impresses the interviewer — it shows you understand the source of efficiency gains rather than just throwing out a number.
Possible follow-up
68h → 23h, you saved that much. Can it be compressed further? Where is the bottleneck?
Suggested answer: The bottleneck is not typing or structured listing, but implicit rules not written in the requirements, business coupling scenarios, and risk judgment. At this stage, the Agent finds it hard to identify these automatically; testers must understand the business and align with product. This portion of work hours is hard to compress significantly.
Interview Question 2
Interviewer asks: For AI-generated test cases, how do you ensure they are executable and have complete scenario coverage? How do you guarantee no missing boundary or branch scenarios?
What is the interviewer testing?
This is an extension of the previous question. The interviewer is testing your understanding of quality engineering for AI-generated content. The core answer approach: don't say 'we rely on human review.' An AI Agent itself cannot automatically guarantee full coverage; it's good at structured, explicit rules. Boundaries, branches, and implicit business constraints are its weaknesses. We mainly rely on a 'multi-layer constraints + validation mechanisms + human fallback loop' to ensure executability and scenario coverage, rather than trusting the model's one-time output.
Complete verbal answer
We mainly ensure this through four layers: input constraints, reasoning guidance during generation, automated validation, human review + continuous feedback loop, while separately addressing the two issues of [executability] and [coverage completeness].
First, solve executability.
- Give the Agent a fixed output specification: enforce alignment with our internal test-case template, including preconditions, operation steps, expected results, assertions, data preparation, and environment dependencies; free text is forbidden.
- Connect metadata: API documentation, field enums, parameter types, database dictionaries, API response structures. When the Agent generates assertions, it directly references fields instead of writing vague descriptions (e.g., cannot write 'return success' but must write code=0).
- Built-in validation rules: after generation, run a lightweight static check, such as whether parameters are valid, whether preconditions conflict, whether test data is missing, and whether assertions match API response fields. If a case violates rules, it goes back for regeneration.
- Define the executable standard: when this test case is handed to an automation script or a tester, it can run directly without guessing the business or doing extra mental work.
Second, solve no missing branch, boundary, or abnormal scenarios. This is the focus and the place where AI most easily fails:
- Do scenario decomposition before generation. Instead of directly throwing requirements to the LLM, the Agent first performs a control-flow / business-branch analysis: it maps out the main flow, branch decision points, state machines, enum values, and judgment conditions (if/else, switches, permissions, roles), outputs a branch checklist first, then generates test cases based on that checklist. For example, an order has payment state, refund switch, and user level; enumerate all combinations first, then generate cases, instead of letting the model hallucinate.
- Built-in boundary-value knowledge base: for common types like numbers, lengths, time, arrays, pagination, and IDs, include boundary strategies: normal values, min, max, critical values, out-of-bounds, empty, null, empty string, special characters, and overlong.
- Inject risk-scenario checklist: turn the historical defect library, high-frequency pitfalls, and business taboo rules into a prompt knowledge base, such as idempotency, concurrency, duplicate submission, cross-tenant, privilege escalation, and transaction rollback, forcing the Agent to prioritize covering them.
- Control combination explosion: when there are too many branches, do not blindly enumerate all; use an equivalence-class + orthogonal-testing strategy, prioritize high-risk combinations, and sample low-risk ones to avoid a flood of test cases.
Third, coverage validation to find missed scenarios: after generating cases, we do a coverage reconciliation in reverse:
- API layer: check whether input-parameter branches, enum values, and state transitions are all hit by test cases.
- Business layer: compare against the branch checklist produced earlier, mark which branches already have corresponding cases and which are missing. Hand the missing-gap list directly to testers, who decide whether coverage is unnecessary or cases need to be added.
Key point: it's not the AI judging whether coverage exists, but comparing the 'decomposition checklist' against the 'generated cases' to make omissions explicit.
Fourth, human fallback + continuous iteration to solve implicit rules. Many implicit logic, business unspoken rules, and historical compatibility logic in requirement documents are not written, and AI cannot see them. Therefore:
- The Agent outputs a draft, not the final version; testers focus on checking: implicit constraints, business risks, and cross-module interaction scenarios.
- All scenarios missed and incorrect cases found during review flow back into the Agent's example library / knowledge base as Few-shot samples, continuously optimizing prompts and decomposition logic to reduce missed scenarios for similar problems next time.
Finally, I would also proactively mention limitations: for strongly coupled, complex business, multi-system integration, and special rules verbally agreed with product, the Agent still finds it difficult to identify automatically; the fallback responsibility remains with test experts. The Agent assists decomposition and generation, not replaces coverage design.
Possible follow-up
If the requirements themselves are vague, will the Agent generate a pile of invalid scenarios?
Suggested answer: When the Agent detects ambiguity in requirements, we designed a mechanism: it first raises a list of questions rather than making assumptions to generate cases. Testers confirm with product before generation continues, avoiding a pile of wrong cases based on wrong assumptions.
Interview Question 3
Interviewer asks: Coverage checks only represent coverage along the requirement dimension, not logic correctness. How do you solve that?
Complete verbal answer
You are absolutely right. This is a pitfall we hit during implementation: coverage checks only solve 'whether this branch / scenario is covered,' i.e., breadth of coverage; being covered doesn't mean the case's logic, expectations, or business rules are correct. Coverage ≠ correctness.
Our approach separates these two things:
- [Coverage check]: handles breadth — whether branches, boundaries, and states are missed.
- [Logic correctness validation]: handles depth — whether the case's own judgment, expectations, transitions, and constraints conform to real business logic. This is a separate validation chain with four layers.
Layer 1: Hard validation with structured metadata (deterministic rules, machine-enforced). Use API schemas, enum dictionaries, database constraints, state-transition diagrams, and configuration rules as knowledge sources, not model imagination. For example:
- Compare field types, length constraints, required-field validation, and foreign-key constraints against the API documentation.
- For computable rules like idempotency, unique keys, and amount calculations, automatically validate whether expected values satisfy the calculation formula.
The feature of this layer: there is a standard answer, and rules can be codified, not relying on the LLM's subjective understanding. It intercepts low-level logic errors first.
Layer 2: Business rule validator (rule engine + knowledge base). For fixed business constraints, we maintain a business rule library written as rules, not left to the model's free play. For example: refund amount cannot exceed the original payment amount, unsettled orders cannot be closed. After the Agent generates expected results, they go through the rule engine: if the case's operation + expectation violates a business rule, mark it [logic conflict] and send an error back to the Agent for regeneration.
Layer 3: Case self-consistency validation + causal reasoning validation (LLM handles complex logic review). For complex business logic that can't be written as hard-coded rules, hand it to a dedicated 'review sub-Agent' that does not generate cases but only reviews, always asking three types of questions:
- Are there contradictions among preconditions, operation steps, and expected results? For example, if the precondition is order canceled, the operation is initiate payment, and the expectation is payment success, that is self-contradictory.
- Are the upstream/downstream interactions of this operation reasonable? For example, after a successful refund, do the coupon, points, and inventory interactions follow business logic?
- Does it violate implicit risk rules, such as over-deduction or duplicate posting in financial scenarios?
Note: the model's review is only a warning, not a basis for judging correctness; it only outputs a list of risk doubts for human recheck.
Layer 4: Human tiered review + regression loop, solving the most core 'implicit business logic'. Hard rules, rule libraries, and model review can only handle documented and accumulated logic. Many business logics are historical compatibility, verbal agreements, or special exception scenarios, not written in documents or in rule libraries, and machines cannot identify them. So we do tiered review:
- Low-risk, common basic scenarios: machine validation passes, then spot-check.
- Funds, permissions, settlement, and core chains: test experts must check business logic correctness one by one. Logic errors found during review flow back in two branches: (1) if it's a general rule, add it to the rule engine / metadata for automatic interception later; (2) if it's a special business exception, store it in the Few-shot sample library for the Agent to reference when generating similar scenarios next time.
Finally, add boundaries and proactively expose limitations (a plus): this system still cannot guarantee 100% automatic logic correctness. Its goal is to automatically intercept low-level, deterministic logic errors and shift human effort from checking whether branches are missed to validating complex business logic and special exception scenarios.
Interview Question 4
Interviewer asks: What problems does this test-case generation Agent solution itself have?
Complete verbal answer
We positioned this Agent mainly to improve efficiency for standardized, explicit-rule test cases. Of course it's not omnipotent. During implementation, we encountered several unavoidable shortcomings. The more complex, implicit, and business-meaning-heavy the scenario, the more obvious the shortcomings. I'll go through them one by one:
Dependence on input quality; garbage in, garbage out (the core pain point). The entire pipeline starts with requirement documents, API documents, and a knowledge base. If requirements are vague, contradictory, or missing preconditions; if API docs are not updated or out of sync with live code; then the decomposition checklist will split branches incorrectly or miss conditions, the baseline itself is wrong. Subsequent reconciliation and correctness validation are run on a wrong baseline, which amounts to 'full coverage of the wrong things.' Implicit verbal agreements and historical compatibility logic not written in documents are naturally invisible to the Agent and must be supplied by people.
High ongoing maintenance cost for the rule engine, knowledge base, and Few-shot samples; easily becomes technical debt. As mentioned earlier, schema validation, state-transition rules, business constraints, and defect sample libraries are the core of correctness validation. When business iterates frequently and product rules change often, the rule library, enums, and state machines must be updated in sync; lag leads to many false interceptions and missed validations. When there are many business lines and modules, rules explode; different lines have different rules, making a universal rule set hard. If sample libraries keep accumulating without cleanup or classification, the model receives conflicting samples and produces unstable output, sometimes right and sometimes wrong. In short: you save time writing cases but add governance work for metadata and rules. If business iterates fast and test staff is stretched, this maintenance easily falls behind.
Unstable reasoning in complex multi-condition combinations and cross-system integration scenarios. For a single API, single state transition, or simple branch, the effect is good. But in multi-service integration, distributed transactions, asynchronous callbacks, multiple roles and configurations stacking, and timing-dependent scenarios (where order of operations affects outcomes), the LLM easily gets causality wrong: it can decompose branch lists and cover scenarios, but it can't sort out upstream/downstream side effects, asynchronous delays, or eventual consistency, so expected results are often written incorrectly.
Blind spots in the validation chain; correctness validation cannot be fully automated. We use three layers: hard rules, rule engine, and review Agent. Hard rules only suit deterministic structural constraints; much business logic is 'soft constraints' that can't be written as code rules; the review Agent itself is an LLM that can only output doubts, and it both false-positives and false-negatives. In other words, the final gatekeeper for logic correctness is always a human; machines can only filter and denoise. When business complexity is high or newcomers lack experience, the review stage still carries missed-judgment risk; it doesn't automatically disappear just because an Agent exists.
Output tends to be 'formally correct but detached from practice'; executability still has pitfalls. Even though we force template and API schema binding, AI easily writes 'paper-usable' cases: it ignores real test environment dependencies, special data-creation costs, data isolation, permission preparation, and third-party mock dependencies; it only considers the ideal test path, not the blockers testers actually hit. Extreme case: the checklist covers everything, all syntax checks pass, but the case cannot run in the test environment and needs heavy rework.
Risk of human capability degradation, a soft problem on the team side. Long-term reliance on the Agent's drafts will gradually reduce testers' habit of actively doing equivalence-class, boundary, and state analysis. If newcomers directly depend on the Agent's output without training in test-case design, their ability to spot anomalies and judge business risk weakens. During review, it's easy to become 'take a glance at what AI wrote and approve it,' treating the Agent's output as the standard answer and relaxing scrutiny, which actually introduces hidden risk.
The model's own uncertainty causes unstable output. With the same requirement and same knowledge base, the quality of generated checklists and cases fluctuates over time; this time the decomposition is complete, next time a branch may be missing. It adds extra burden on reviewers: every time they have to re-verify carefully; they can't reliably trust the output.
Interview Question 5
Interviewer asks: Once usage scales up, how do you continuously ensure AI-generated cases are correct? What if human review also risks missing errors?
Complete verbal answer
Once scale grows, you can no longer rely on 'human line-by-line review' as the fallback. Roughly, it can be divided into four parts: pre-admission gating, mid-process multi-layer validation (reducing errors flowing to humans), review mechanism reform (reducing miss probability), and post-hoc observation with automatic feedback (capturing leaked errors and feeding them back).
1. Before: tiered admission, reducing error inflow from the source
At scale, the worst thing is throwing all requirements into the Agent indiscriminately.
- Grade admission by business risk and complexity.
- L1 (low risk: ordinary queries, configuration APIs, simple CRUD): Agent generates + automated validation passes, spot-check approval, no mandatory full review.
- L2 (medium risk: ordinary flows, simple state transitions): Agent generates + machine validation + sampled human recheck.
- L3 (high risk: funds, permissions, settlement, distributed async, core transactions): Agent is only a draft aid, mandatory expert review, even human-led design.
- Version-manage baselines: keep version history for decomposition checklists, state machines, business rules, and enums. When requirements change, update the baseline first, then regenerate cases. This avoids random output drift from parallel requirement edits and messy knowledge bases.
- Input quality gate: add pre-checks for whether requirements are ambiguous, API docs are in sync with live code, and state diagrams are missing. If input quality is poor, block it before it enters the generation flow, preventing 'wrong baseline producing wrong cases in batches.'
2. During: layer machine validation to intercept most errors before human review (reducing human pressure)
The hard validation, rule engine, and review Agent mentioned earlier need to become quantifiable, alarmable, and statable at scale, not just outputting doubts:
- Deterministic hard rules (Schema, state transitions, enums, field constraints): strong interception, not allowed into the review queue, directly returned for regeneration; zero human involvement in this layer.
- Business rule engine: distinguish blocking rules (refund amount cannot exceed original order) from warning rules (non-binding suggestions). Blocking rules reject directly; warning rules tag risk labels and push to reviewers for focused attention.
- Independent review Agent for doubt extraction, doesn't dump complete cases for people to read; only hands over high-risk doubt fragments. For example, out of 20 cases in a batch, the Agent flags only 3 as risky: 'this state transition is abnormal,' 'expected amount conflicts with rules.' People miss errors mainly because of boredom, high volume, line-by-line reading, and attention drift. This way, reviewers no longer read all cases; they check doubts first, and only after doubts are clear do they look at the rest. Focused review greatly reduces fatigue-driven misses.
- Same-requirement diff comparison (very suitable for scale). When requirements iterate, diff the new version's generated cases against the previous version and show only added, modified, deleted cases and change points. Reviewers only look at the diffs; stable historical cases are exempt. In version-iteration scenarios, this approach notably reduces missed review.
3. Review mechanism reform: use cross-check, spot-check, and multi-person recheck to reduce misses
Humans will always miss things, so you can't rely on a single person's one-pass approval; use mechanism to lower probability:
- Match review strategy to risk grade.
- L1 low risk: after machine validation passes, auto-approve with random background spot-checks. If a spot-check finds a problem, roll back and optimize Agent samples, without blocking the flow.
- L2 medium risk: single-person review + system random blind re-check (some randomly assigned to a second expert).
- L3 high risk: mandatory dual-person review; two people review independently, and disagreements go to review meeting.
- Standardize review checklists to reduce 'gut-feel review.' Give reviewers a fixed checklist: is the baseline list aligned, are state transitions compliant, upstream/downstream interactions, exception boundaries, expected results, and data-creation dependencies. The system forces checks to be ticked and does not allow one-click pass-through, avoiding 'eyes closed and approve.'
- Accumulate review traces and risk tags. Tag every doubt and every modification with [baseline error], [model reasoning error], or [implicit business rule] to later quantify which error types are frequent and target optimization of knowledge base / prompts.
Be honest here: this mechanism can only reduce miss probability, not eliminate it, so post-hoc fallback is required.
4. After: execution-side and online-side observation loop, capturing leaked errors and automatically feeding back
This is the most critical part of sustaining correctness at scale, and it's also what many solutions lack:
- Collect feedback from the test execution stage. When a case runs and [expected result differs from actual return], separate the cause:
- Program bug: file a normal defect.
- Case itself is wrong (Agent generated wrong, or missed review passed): mark it as [case error sample], automatically enter the sample library, and trigger an alert to the owner. Such error samples automatically serve as Few-shot examples to avoid the same scenario next time.
- Reverse attribution of online missed-test defects. When an online defect appears, trace back: does it belong to a branch that should have been covered, or was the case expectation wrong? If it was caused by Agent generation + missed review, add it to negative samples and check whether the baseline or rules are missing.
- Continuous indicator dashboard, monitoring Agent quality trends instead of single results. After scaling, watch trend indicators rather than one-off points:
- machine interception rate, review doubt detection rate
- error density found at review stage
- case error rate found at execution stage (core indicator, representing missed-review outflow rate)
- error distribution across business modules and requirement types. If error rate for a module keeps rising, auto-alert, suspend automatic approval for that module, switch to mandatory full human review, and first fix the knowledge base.
Interview Question 6
Interviewer asks: LLMs themselves hallucinate. Why use an LLM for test-case quality validation instead of pure rules? Have you considered this contradiction?
Complete verbal answer
We have seriously considered this contradiction.
Our strategy is layered: whatever can be fixed as deterministic rules is all validated by a rule engine, rules first, to avoid LLM hallucination; only scenarios that are hard to write as rules — complex interactions, business causality, and semantic self-consistency — are handed to the LLM review Agent.
In other words, we are not replacing rule validation with LLM; the two divide labor and complement each other, each doing what it is good at, with clear responsibility boundaries. Below I can expand layer by layer:
Layer 1: First distinguish two types of validation objects — which suits pure rules, which are hard to write as rules
- Suitable for pure rule engine (use first; if you can write a rule, never leave it to the LLM). These are deterministic, hard constraints: field types, enum ranges, amount cannot be negative, refund amount ≤ original payment amount, required-parameter validation, API schema constraints. These conditions have clear boundaries and rarely change. Written as code/DSL rules, this layer in our pipeline is a strong interception layer, runs first, and does not pass through the LLM. Any violation is directly rejected and does not flow into review.
- Scenarios hard to write as pure rules, or where writing rules is extremely costly (only then give to the LLM review Agent). These are the weaknesses of pure rules:
- multi-service integration, async callbacks, upstream/downstream chain effects;
- implicit business constraints, cross-condition business causality;
- semantic-level self-consistency checks: whether preconditions, steps, and expectations have logic conflicts;
- natural-language business-intent validation in requirements and risk-scenario identification (e.g., privilege escalation, capital-loss risk).
Layer 2: How to isolate hallucination risk
The core design has three items:
- The LLM only has 'doubt-raising power,' not 'approve/reject decision power.' The rule engine's result is a judgment: pass / fail, and it takes effect directly. The LLM review outputs a [list of risk doubts].
- Grade the LLM's doubts, and track its hallucination (false-positive) metrics to continuously converge:
- Model alert → after human recheck, determine: real error / hallucination false-positive. Accumulate false-positive samples, continuously optimize prompts, add RAG context, adjust its review scope. If a certain type of question has persistently high false-positive rate, directly take it offline and switch it to pure rules or manual check, no longer to the model.
- The model's review output is a supplementary signal, not the only signal. Whether a case can proceed ultimately comes from:
- ✅ hard-rule validation result (highest priority)
- 🟡 LLM review doubt list (reference signal)
- 👤 human review decision
Layer 3: Clarify boundaries and fallback, proactively expose limitations, showing depth of thinking
We have a clear red line: for all capital-loss, permission, and core-transaction strong validation, never rely on LLM review, all sink to rule engine. The LLM only handles cases where rules are hard to write, change frequently, and belong to business semantics and causal reasoning.
Of course, this solution does have shortcomings, e.g., the LLM has a probability of false negatives (another form of hallucination: not seeing obvious errors). So we don't assume a case is fine just because the model didn't raise a doubt. We pair it with: sampling recheck, change-diff review, and post-hoc execution feedback loop.
Possible follow-up
Since the LLM can miss errors, is its value low?
Suggested answer: Its value lies in early detection of many implicit logic problems that rules cannot cover. Many cross-module interactions and implicit business conventions are hard to anticipate and hard to write rules for in advance. The model can do low-cost batch scanning and help people focus on suspicious points. It cannot replace people, but it can reduce the probability of people missing complex causal problems.
Closing
Finally, from the interviewer's perspective, here's how to answer this kind of question and what to emphasize.
These follow-up questions around AI projects are all trying to confirm one thing: have you actually done this for real.
When a number is probed, making the definition and comparison clear matters more than the percentage improvement itself. In quality assurance, state clearly which layer the machine guards and which the human guards. When asked about weaknesses, admit them boldly. Candidates who can proactively state limitations reassure the interviewer instead. Don't just throw out conclusions; explain the process, the problems encountered, and how you solved them. These matter more than the conclusions themselves.
Top 1 from juejin.cn, machine-translated. The original thread is authoritative.
Awesome. But mainly, are there still testers nowadays? And this set isn't suitable for small companies, right? Small companies don't have testers as a safety net.