← Back to the summary

AI in Test Automation: What It Genuinely Can't Do and What You're Just Using Wrong

After spending a long time in the testing community, you'll hear the same few complaints about AI testing over and over again.

Some complaints are genuine. AI genuinely can't do certain things well at this stage, and forcing it won't help.

Others, to put it bluntly, stem from a lack of fundamental skills. People aren't aware of their own shortcomings and insist on blaming "AI." Honestly, I have zero sympathy for this.

But most of the time, it's a third scenario—most people are at a half-baked level. You can't say they don't know how to use it, because they can get AI to produce something. But you can't say they do know how, because their scripts break down every couple of days. They write a two-line prompt and call it done, can't articulate business rules clearly, and can't be bothered to build testability infrastructure. Naturally, AI works sometimes and fails at others—it runs today and is broken tomorrow. You really can't blame AI for this; it's because you haven't learned it properly.

Let's break this down today.

A Few Things AI Can Do, But Not Well Enough

This is the category I want to focus on. AI can indeed get started—it can produce code, run through processes, and deliver a demo that looks the part. But when you actually use it to test production business logic, you'll find it only reaches 60 points; the remaining 40 points still need a human to fill in. The following are typical examples of "can do, but can't do well."

1. Business Assertions

"Business assertions" sound like just an assert statement, but what really trips people up is never the syntax—it's "what to assert."

It's not about asserting that the interface returns 200; it's about whether the business result is correct—whether the order amount is calculated correctly, whether the risk control rules were triggered correctly, whether the loyalty points were issued correctly. At this level, even humans often can't articulate it clearly, let alone expecting AI to do it well.

Take a promotional example: spend 300 get 50 off, members get an additional 20% discount, and you can stack a 20-yuan coupon on top.

Getting AI to click through an order process is no problem. But asking it to judge "whether the final amount of this order is correct"—it starts guessing wildly.

AI can write this:

# AI's default output: generic interface assertion
def test_create_order():
    resp = client.post("/api/order", json=payload)
    assert resp.status_code == 200
    assert resp.json()["order_id"] is not None

But what you really need to verify is this:

# Real business assertion—AI can't write this
data = resp.json()
# Spend reduction → discount → coupon, the order cannot be wrong
expected = (300 - 50) * 0.8 - 20
assert data["final_price"] == round(expected, 2)
assert data["discount_sequence"] == "fullcut_discount_coupon"
# VIP stacked discounts trigger risk control rules
assert data["risk"]["triggered_rules"] == ["RULE_VIP_OVERLAP"]
# Points = actual payment × member point ratio
assert data["points_earned"] == int(expected * member_point_ratio)

Every line in the second code block requires knowing the promotion rules, risk control rules, and points rules. This isn't something you can grasp by reading a document once—it comes from soaking in the requirements pool, arguing with product managers, and learning from production incidents.

AI cannot replace this part. Don't waste time trying to force it.

2. Complex Asynchronous Chain Verification

"Asynchronous chain verification" sounds abstract, but the scenario is very common—a user clicks a button, and a whole chain of events is triggered behind the scenes. Place an order, deduct inventory, send a message after inventory is deducted, the message triggers a logistics push, logistics arrival updates the status. If any asynchronous link fails in the middle, the frontend still shows "Success."

What's AI's problem? It only sees the second the synchronous response returns, and has no idea what happened in that string of asynchronous events behind it.

Take a real scenario: Place order → Deduct inventory → Generate order → Trigger message → Push logistics → Update status. One chain crosses six systems, three tables, and two message queues.

AI will only write this by default:

# AI written: only verifies the synchronous response
def test_order_flow():
    resp = client.post("/api/order", json=payload)
    assert resp.status_code == 200
    print("✅ Order created successfully")

It looks like it passed, but the real problems all happen in the asynchronous stages:

# What really needs verification—AI won't proactively write this
assert resp.status_code == 200
order_id = resp.json()["order_id"]

# 1. The order creation message was actually sent
msg = consume_mq("order.created", timeout=5)
assert msg["order_id"] == order_id

# 2. Inventory was actually deducted
assert db.query("SELECT stock FROM sku WHERE id=%s", sku_id) == origin_stock - 1

# 3. The logistics queue received the push
assert mq_count("logistics.push") == 1

# 4. The order status transitions to PUSHED within 5 seconds
wait_until(lambda: order_status(order_id) == "PUSHED", timeout=5)

# 5. Idempotency: repeated calls do not deduct inventory again
client.post("/api/order", json=payload)
assert db.query("SELECT stock FROM sku WHERE id=%s", sku_id) == origin_stock - 1

If any asynchronous link fails in the middle, AI won't see it. You think it passed, but the inventory was already oversold.

For this kind of chain, there is currently no absolute silver bullet. It relies on humans to first sort out the rules, on monitoring, and on reconciliation. At best, AI can come in as a runner.

3. Performance Root Cause Analysis

The hard part of performance issues isn't "discovering slowness"—anyone can add a tracking point or configure a threshold alert. The hard part is "why it's slow."

Is it slow because of SQL? The connection pool? A downstream service? GC? Or is there a big key hidden in Redis? Behind this lies experience, investigation paths, and an understanding of the entire system. AI can report that "the interface is slow," but when it comes to digging deep into the root cause, it's at best a novice.

AI can tell you "the interface is 800ms slower" and can also tell you "the SQL isn't using an index."

# What AI can do: threshold alert
def test_api_perf():
    resp = client.get("/api/products")
    assert resp.elapsed.total_seconds() < 0.5

But real root cause location, it can't do:

# The following set, AI can assist but cannot complete independently
1. Look at the flame graph → 70% of time is stuck on redis_cmd
2. Check the slow query log → Find a KEYS * scan
3. Check the monitoring curve → GC time is rising simultaneously
4. Combine with experience → Determine it's caused by connection pool config + a big key
5. Provide a solution → Business side splits the key, ops side adjusts connection pool parameters

Reporting the news is its job, but cracking the case relies on a human.

A Few Things AI Actually Can Do, But You Don't Know How to Use It

Now let's talk about the second category—this one is the exact opposite of the previous. For the following things, AI is completely capable and can do them quite well. If you feel "AI is no good at this," nine times out of ten the problem isn't AI, it's you—either your application's testability isn't done well, or your prompt isn't clear enough, or your infrastructure isn't solidly built. Simply put, for this category, you mostly have to take the blame yourself.

1. Element Location

Let's state a fact first—mainstream AIs today (Claude, Cursor, Copilot) write Playwright code using get_by_role and get_by_label by default; for Appium, they default to accessibility id. Absolute XPath is no longer AI's default choice.

So why do your UI scripts still break? The problem isn't what location strategy AI chooses, it's that your application didn't leave any usable "anchors" for AI.

The code AI writes by default now looks like this:

# ✅ Playwright: AI defaults to smart locators, doesn't write XPath at all
page.get_by_role("button", name="Submit Order").click()
page.get_by_label("Phone Number").fill("13800138000")

# ✅ Appium: AI also defaults to accessibility id
driver.find_element(AppiumBy.ACCESSIBILITY_ID, "login_button").click()
driver.find_element(AppiumBy.ACCESSIBILITY_ID, "phone_input").send_keys("13800138000")

The code looks fine, but it breaks when run. Why? Because your application looks like this:

<!-- Webpage: button has no aria-label, input has no label -->
<button class="btn-primary">Submit</button>
<input type="text" placeholder="Phone Number" />

# App: All elements' ContentDescription is "button"
# AI wants to grab accessibility id, but grabs the same name for everything

AI wants to get_by_role("button", name="Submit Order"), but your page has 5 "Submit" buttons at the same time; it wants ACCESSIBILITY_ID("login_button"), but your app never set this label.

This isn't AI writing poorly; it's your application's testability that's poor.

The new generation of Agent tools (agent-device, mobilerun) follows the same logic—they assign semantic references to elements based on the accessibility tree:

$ agent-device snapshot
# @e1 [button] "Submit Order"
# @e2 [text-field] "Phone Number"

$ agent-device fill @e2 "13800138000"
$ agent-device click @e1

But the premise remains the same—your app must have a readable accessibility tree. Do the testability groundwork well, and AI's stability in finding elements immediately jumps a level. Skip this, and no model switch will help.

2. Test Data

"Test data" is a disaster zone for complaints—"The data AI generates is all test123, Zhang San, Li Si, not realistic at all."

But don't rush to curse. AI gives you test123 because that's the only amount of information it has in its head. You tell it "generate 10 users," it doesn't know your business, doesn't know your sandbox number segments, doesn't know your risk control rules—besides test1 to test10, what else can it give you?

❌ Your Prompt:

"Generate 10 test users"

AI honestly outputs:

test1, test2, test3 ... test123, password123

Make the rules clear, and it's immediately different:

✅ The Prompt you should write:

"""
Generate 10 test users, requirements:
- Users fall into three categories: 5 regular users, 3 VIPs, 2 merchant accounts
- Phone numbers start with 138/139/188 (sandbox number segments)
- VIP account level ranges 3-5, balance 1000-50000
- Merchant accounts must have shop_id and status='active'
- Avoid risk control: same phone number no more than 3 registrations within 24 hours
- Each user has a profile: registration times spread over the last 30 days
"""

Data generated according to this prompt can enter the sandbox, run through business processes, and trigger various boundary scenarios.

AI is an intern. If you don't give it a requirements document, it can only give you test123.

3. Script Generation

When you ask AI to write test code, its default output is linear scripts—stacked from start to finish, just enough to run.

# ❌ AI default output: linear script, runs but hard to maintain
def test_order():
    client.post("/login", json=cred)
    resp = client.post("/cart", json={"sku": "A1"})
    assert resp.status_code == 200
    resp2 = client.post("/order")
    assert resp2.status_code == 200
    resp3 = client.get("/order/" + order_id)
    # ... keeps stacking, no layering, no reuse

It runs, but has a pile of problems: strong coupling between steps, the login process rewritten for every test case, changing one interface triggers a full regression, and it leaves a mess of dirty data after running.

This isn't AI writing poorly; it's that you didn't tell it what framework to write according to.

Give it rules, and it's immediately different:

✅ The rules you should give AI:
"""
Generate according to the Page Object pattern, requirements:
- Pages/services encapsulated into classes (LoginPage, OrderService)
- Business actions encapsulated into methods (login(), create_order())
- Test data injected via fixtures, not hardcoded in scripts
- Each test function follows the Arrange-Act-Assert three-part structure
- Use the pytest framework, support parameterization
- Common steps extracted into fixtures for easy reuse
"""

Code generated according to these rules looks like this:

# ✅ AI output according to rules: layered, reusable, maintainable
class OrderService:
    def __init__(self, client):
        self.client = client

    def create(self, payload):
        return self.client.post("/order", json=payload)


@pytest.mark.parametrize("user,expected", [
    (UserData.vip(), "VIP_DISCOUNT"),
    (UserData.normal(), "NORMAL"),
])
def test_order_discount(auth_client, order_service):
    # Arrange
    order_data = OrderBuilder().with_user(user).build()
    # Act
    resp = order_service.create(order_data)
    # Assert
    assert resp.json()["discount_type"] == expected

AI's default output is "code that runs," not "code that can be maintained." The former relies on a single prompt; the latter relies on you providing rules, frameworks, and paradigms.

4. Waiting Strategies

There's a particularly torturous problem in automated testing—flaky tests. The same script passes this run, mysteriously breaks the next, passes again on a rerun, and you can't even investigate.

But open up those flaky scripts and take a look; nine times out of ten, it's the same problem—the waiting isn't done right.

When AI writes test scripts, the default output is this kind of code:

# ❌ AI default output—this is the root of all flaky evil
time.sleep(1)
driver.find_element(By.ID, "submit").click()

time.sleep(1)—so simple, so terrible.

Why is it terrible? Because it's waiting for time, not waiting for "the thing that should be waited for."

When the environment is fast, it loads in 0.3 seconds, you sleep 1 second, wasting 0.7 seconds; when the environment is slow, it takes 2 seconds to load, you sleep 1 second, the element hasn't appeared yet and you click—crash.

After crashing, you definitely think: "Then I'll just increase sleep to 3 seconds, right?"

# ❌ An even worse "fix"
time.sleep(3)
driver.find_element(By.ID, "submit").click()

Congratulations, now every test case wastes an extra 2 seconds, running 100 cases costs 3 more minutes, running 1000 cases costs half an hour more. What's worse—it will still crash when it's supposed to, because someone's network speed is always worse than you imagine.

The correct approach is the opposite: don't wait for time, wait for signals.

The page actually gives us many signals—the loading spinner disappears, the skeleton screen disappears, a certain DOM appears, the URL redirects, a window variable is ready. Waiting for these "business signals" is much more reliable than guessing 1 second or 3 seconds.

# ✅ Playwright: wait for business signals, not time
# Wait for loading animation to disappear
page.wait_for_selector("[data-loading]", state="hidden")
page.get_by_role("button", name="Submit").click()

# Wait for URL redirect
page.wait_for_url("**/dashboard")

# Wait for frontend semaphore to be ready
page.wait_for_function("() => window.appReady === true")

# ✅ Appium: WebDriverWait explicit wait
# Wait for element to appear
WebDriverWait(driver, 10).until(
    EC.presence_of_element_located((AppiumBy.ID, "content_loaded"))
)

# Can also wait for custom conditions—like cart count becoming non-zero
WebDriverWait(driver, 10).until(
    lambda d: d.find_element(AppiumBy.ID, "cart_count").text != "0"
)

See, it's waiting for "loading disappeared", "URL redirected", "count changed"—these are signals that the business is "truly ready," not a guessed 1 second or 3 seconds.

But the problem is—AI won't proactively write this way.

It doesn't know that your list is asynchronously loaded, doesn't know there's a loading spinner after clicking, doesn't know the popup has a 300ms animation, doesn't know a successful submission redirects to /dashboard. This "business timing" information needs to be told to it in your prompt:

✅ You should add these lines in your prompt:
"""
Timing conventions:
- The list is asynchronously loaded, wait for the skeleton screen to disappear before operating
- After clicking "Submit," a loading spinner appears, wait for it to disappear before asserting
- The popup has a 300ms entry animation, don't click buttons during the animation
- A successful submission redirects to /dashboard, wait for the URL to change before continuing
"""

The code AI writes according to this prompt immediately becomes stable.

The matter of waiting strategies is essentially a fundamental testing skill. AI won't proactively judge "when to wait, what to wait for"; you must clearly explain the "business timing" in your prompt. If this isn't explained clearly, no model switch can save your flaky tests.

A Final Few Words

Writing up to here, back to the opening question—can AI do automated testing or not?

My answer is: Yes, but it depends on how you use it.

Business assertions, asynchronous chains, performance root cause analysis—for these things, AI genuinely can't do them well at this stage. Making it the main force here is tough on it and tough on yourself. Let it run errands, improve efficiency, and assist humans; that's enough.

Element location, test data, script generation, waiting strategies—for these things, AI really can do them. If you feel "it's no good," don't rush to curse; first turn back and ask yourself three questions:

Does your application leave usable "anchors" for AI (accessibility labels, testid, aria-label)?
Does your prompt clearly articulate the business rules, timing, and boundaries?
Is your testing infrastructure solidly built (data factories, environment isolation, code frameworks)?

If these three things aren't done well, no model switch will help.

As for those shouting "AI will fully automate and replace test engineers"—wake up, that's sales talk.

Ultimately, the test of AI doing automated testing has never been about how strong AI is, but about whether the person using AI has solid fundamental testing skills.

A tester who can't even articulate testability, assertions, and waiting strategies clearly can't be saved by any model;

A tester who knows how to use AI will sooner or later make those who don't unemployed. I'm saying this.

Next time you want to curse "AI is no good," first look back at your own prompts and code infrastructure. That's the root of the problem.