Flaky E2E Tests Are Worse With AI-Generated Code
AI-generated code doesn't invent new categories of test flakiness. It raises the odds on the categories that already existed, and it does that through two separate mechanisms that call for two different fixes — one of which an environment can solve, and one of which it can't. Getting that distinction wrong is expensive in a specific way: teams that assume ephemeral environments alone fix flakiness get disappointed when a third of their flaky tests keep failing after the migration, and mistakenly conclude the infrastructure investment didn't work — when in fact it did exactly what it was supposed to, on exactly the categories it was ever going to touch.
Three kinds of flaky, and where they come from
A widely cited study of flaky tests (Luo et al., FSE 2014) breaks the causes down by mechanism, not by symptom:
| Category | Share of documented causes | What it looks like |
|---|---|---|
| Async wait issues | ~45% | Test doesn't wait properly for an operation to finish before asserting on its result |
| Concurrency | ~20% | Race conditions and deadlocks in code under real concurrent load |
| Test order dependency | ~12% | A test passes in isolation, fails when run after another test that left behind state |
| Environmental | Remainder | CI runner differences, resource limits, network latency, timezone, drifted shared environments |
The first two — async waits and concurrency — are bugs in the application or the test itself. The second two — order dependency and environmental drift — are properties of where the test ran, not what it checked. That distinction is the whole argument of this piece.
Each category has a recognizable shape once you've seen it:
Async wait issues look like a test that clicks a "Save" button and immediately asserts the success message is visible, without waiting for the network request to actually complete. It passes most of the time, because the request is usually fast enough to beat the assertion. It fails whenever the network is slow enough that it isn't — nothing about the application changed, only the timing.
Concurrency issues look like two requests writing to the same record at nearly the same time, where the order they're processed in determines the result. In a test suite, this shows up as a test that passes when run alone and fails intermittently when run alongside others hitting the same resource — the classic "works on my machine, fails in CI" pattern, except it's not a machine difference, it's a timing difference.
Order dependency looks like a test that creates a user with a specific email address, and a second test — written independently, weeks apart — that also creates a user with that same email address. Run in one order, both pass. Run in the other, the second collides with data the first left behind. Neither test is wrong in isolation; the dependency only exists between them.
Environmental flakiness looks like a test that's rock-solid in local development and flaky only in CI, or flaky only on Tuesdays when a scheduled job on the shared staging server happens to run mid-test. Nothing about the test or the code changed — the ground it was standing on moved.
The cost, and why estimates vary so much
Industry estimates for how much flaky tests cost range widely — from around 2% of engineering time on direct investigation up to 15-30% of total CI time once reruns and triage are included, depending on whose methodology you use and what counts as "cost." That range isn't sloppiness; it reflects a real measurement problem: most organizations don't tag failures as flaky versus real at the point of failure, so the true cost is reconstructed after the fact, if at all. Mabl's 2024 State of Testing in DevOps report, surveying 500+ development and QA leaders, found a 138% increase in teams ranking test maintenance as their top testing challenge year-over-year — not a percentage of time, but a clean signal that the problem is getting worse, not stabilizing, industry-wide.
The practical takeaway isn't "flaky tests cost exactly X%." It's that almost no team measures this number for themselves, which means almost no team can tell whether a given quarter's engineering investment in test reliability is paying off. That's the same measurement gap test environment management runs into more broadly — you can't manage what you've never actually measured.
The cost that's hardest to put a number on is the least visible one: trust. A suite where a red X sometimes means a real bug and sometimes means "just re-run it" trains engineers to treat every failure as probably-noise, which is fine right up until the failure that wasn't. Once a team develops the habit of clicking re-run without reading why something failed, a genuine regression has to survive that habit to get caught — and the data above suggests it frequently doesn't, since 84% of the time the habit is statistically justified, which is exactly what makes the other 16% dangerous.
Why AI-generated code raises the odds on both halves
On the code side: the same categories of bugs that make E2E testing of AI-generated code necessary in the first place — async and ordering issues, cross-service assumptions that were never true — are exactly the async-wait and concurrency mechanisms behind the largest share of documented flakiness. An agent that doesn't fully reason about timing or execution order doesn't just produce one bug; it produces the specific kind of bug most likely to show up as an intermittent, hard-to-reproduce test failure rather than a clean, repeatable one.
On the infrastructure side: more PRs means more concurrent demand on whatever test infrastructure exists. AI coding agents measurably raise PR volume — that's not in dispute, only the multiplier is. On a shared staging environment, that directly increases order-dependent and environmental flakiness: more concurrent deploys, more leftover state from the PR that ran ten minutes ago, more resource contention. We've covered why shared staging breaks down under this exact pressure — flakiness is one more symptom of the same root cause, not a separate problem.
The two mechanisms compound rather than just add. A codebase producing more timing-sensitive bugs, tested on infrastructure with more concurrent contention, doesn't just get twice the flaky tests — it gets flaky tests that are harder to diagnose, because a failure could plausibly be either mechanism until someone actually checks. That ambiguity is exactly what erodes trust in a test suite fastest: not the flaky tests themselves, but the growing uncertainty about which failures are worth investigating at all.
What an environment fixes, and what it doesn't
This is the part worth being honest about, because it's tempting to oversell it: isolated, freshly-provisioned-per-PR environments structurally remove order-dependent and environmental flakiness. There's no leftover state from the previous test, because the environment didn't exist before this run. There's no resource contention from three other PRs, because nothing else is using it. That's the "environmental" and "order dependency" categories — a meaningful chunk of documented causes, gone by construction rather than by discipline.
What isolation does not touch is the roughly two-thirds of flakiness rooted in async waits and concurrency bugs in the code itself. A race condition an AI agent introduced is exactly as flaky in a pristine, isolated environment as in a shared one — arguably more visible, since there's no environmental noise left to blame it on. That's a code-quality and test-design problem: proper waits instead of fixed sleeps, deterministic test data, assertions that wait for state rather than assuming timing.
The async-wait fix, concretely, is usually smaller than it sounds. Compare asserting immediately after an action to waiting for the actual state the action produces:
1// Flaky: asserts before the async save has necessarily completed
2await page.click('#save-button');
3await expect(page.locator('.success-message')).toBeVisible();
4
5// Not flaky: waits for the specific network response, then asserts
6const responsePromise = page.waitForResponse('**/api/save');
7await page.click('#save-button');
8await responsePromise;
9await expect(page.locator('.success-message')).toBeVisible();Neither version is more code, meaningfully. The second one just waits for the thing that actually determines whether the assertion should be true yet, instead of hoping enough time has passed. Most async-wait flakiness in AI-generated tests looks exactly like the first pattern, because it's the more obvious way to write the test and nothing forces the more precise version unless someone — human or a Playwright Healer with the right prompt — specifically checks for it.
Worth naming directly: this is a case where treating AI-generated test code with the same scrutiny as AI-generated application code matters just as much. A generated test that happens to pass in CI ninety-plus percent of the time looks like a working test in every dashboard that only tracks pass/fail — the fixed-sleep pattern above doesn't announce itself as a problem until someone's specifically looking for the category of bug it produces.
Quarantine: the practical middle ground
Not every known-flaky test gets fixed immediately, and pretending otherwise leads to either ignoring CI failures wholesale (worse than not testing) or blocking every merge on tests nobody's had time to fix yet (worse for velocity than the flakiness itself). Quarantine is the standard middle ground: a known-flaky test gets tagged and moved out of the merge-blocking suite into a separately tracked, non-blocking run, with an owner and a deadline attached — not deleted, not ignored, not silently retried into passing.
The discipline that makes quarantine work instead of becoming a graveyard: a quarantined test has to be visible somewhere a human actually looks (a dashboard, a weekly report, not just a tag in the test runner's config) and it has to have an owner, not just a category. Without both, quarantine is where flaky tests go to be forgotten, and the suite's real coverage quietly shrinks while its reported coverage stays the same — a slow, invisible version of the same trust erosion described further down, just moved one step earlier in the process.
A triage checklist before you "fix" a flaky test
Before reaching for a retry decorator — which hides the signal rather than fixing anything — work through this in order:
- Re-run it against a fresh, isolated environment. If it passes reliably there but was flaky on shared staging, you've found environmental flakiness. The fix is infrastructure, not code.
- Run it back-to-back with other tests in a fixed order, then in a randomized order. If the result depends on order, you've found test order dependency — usually a shared database, cache, or global state that one test doesn't clean up.
- Check whether the test asserts immediately after an async action (a navigation, an API call, a queued job) instead of waiting for a specific state. If so, that's an async wait issue — the most common single cause, and the easiest to fix once identified.
- If none of the above reproduce it, and it's genuinely inconsistent under load, you're looking at a real concurrency bug in the application. That one goes to engineering, not QA — an environment or a better wait strategy won't make it go away, because the bug is in the code's actual behavior under concurrent access.
Skipping straight to retries treats all four as the same problem. They aren't, and only one of the four gets fixed by infrastructure alone.
Two failures that look identical and aren't
Here's why the triage order above matters more than it might seem: two genuinely different bugs can produce the exact same symptom, and guessing wrong wastes real time.
Case one. An E2E test for an order-confirmation flow fails intermittently, roughly one run in eight. The failure is always the same assertion — the confirmation email's content doesn't match what was expected. Investigation shows the test is running against a shared staging environment where a scheduled cleanup job occasionally deletes the test's order record between the order being placed and the email content being checked, because the job's "abandoned order" heuristic doesn't account for a test order that's mid-flow. Move the same test to a fresh, isolated environment with no cleanup job running against it, and the failure disappears completely across fifty consecutive runs. This was environmental flakiness — a property of where the test ran, not a bug in the checkout code.
Case two. A different test, same symptom — intermittent failure, same assertion pattern, roughly the same failure rate. Move it to a fresh, isolated environment and it still fails intermittently, at almost the same rate. Investigation this time shows the order-confirmation service and the inventory-decrement service both write to the same order record, and under specific timing, the confirmation service reads the record before the inventory write completes, picking up stale data. This is a real race condition in application code that an AI agent introduced while adding an inventory check to an existing flow — no environment, however clean, makes it go away, because the bug reproduces given the right timing regardless of where the code runs.
From the outside, before investigation, these two cases are indistinguishable — same intermittent pattern, same assertion failing. That's exactly why re-running against a fresh environment first, before anything else, is the highest-value single step in the triage checklist: it splits these two cases apart immediately, for the cost of one extra test run, instead of a QA engineer and an application engineer both independently investigating the same symptom for two different reasons, arriving at two different conclusions, and losing a half-day each figuring out which of them was chasing the real issue.
Detecting flakiness before it's a recurring complaint
Most teams find out a test is flaky by accident — someone notices the same test failing intermittently over a few weeks and starts to suspect a pattern. A more systematic approach is cheap to run and catches it earlier: run new or modified E2E tests several times in a row (most CI systems and frameworks support this natively — Playwright's --repeat-each, a simple loop in a CI job) against a fresh, isolated environment each time, before merging. A test that fails even once in ten fresh-environment runs is either flaky or genuinely non-deterministic, and either way, it's worth knowing before it's merged and blocking someone else's PR three weeks from now.
This is cheap specifically because ephemeral, per-PR environments make it cheap — running a test ten times against ten fresh environments costs ten environments' worth of compute, not ten times the coordination overhead of booking a shared staging slot ten times. On shared staging, the same detection approach is prohibitively expensive in practice, which is part of why so few teams do it and why so much flaky-test detection stays reactive instead of proactive — the tooling to be proactive has existed for years, but the environment cost of using it routinely didn't make sense until per-PR environments made repeated fresh runs close to free.
FAQ
What percentage of test failures are actually flaky tests, not real bugs? At Google, published research found 84% of pass-to-fail transitions in their test suite involve a flaky test rather than a genuine regression, with roughly 1.5% of all test runs exhibiting flaky behavior and about 16% of tests affected at some point. Most organizations don't measure this, so the true rate is usually unknown rather than low.
Does moving to ephemeral environments fix flaky tests? It fixes environmental and order-dependent flakiness — roughly a third of documented causes — by giving every test run a fresh, isolated environment instead of a shared, drifting one. It does not fix flakiness caused by race conditions or async timing bugs in the application code itself, which is a code-quality problem, not an infrastructure one.
Why does AI-generated code produce more flaky tests? Two separate mechanisms. First, AI coding agents write more async and ordering bugs than average, which shows up as genuine race-condition flakiness. Second, higher PR volume increases contention on shared test infrastructure, which shows up as environmental flakiness — a test that's fine in isolation but flaky under concurrent load from other PRs.
How do you tell the difference between a flaky test and a real bug? Re-run the exact same test against the exact same environment and code, several times, with no changes. If the result is inconsistent, it's flaky by definition — a real bug reproduces every time given the same inputs and environment. The harder question is diagnosing which of the three flaky categories (random, environmental, order-dependent) is responsible, which determines the fix.
What is test quarantine, and when should a flaky test be quarantined instead of fixed immediately? Quarantine means moving a known-flaky test out of the merge-blocking suite into a separately tracked, non-blocking run, with an explicit owner and deadline — not deleting it or ignoring its failures. It's appropriate when the fix isn't trivial and blocking every merge on it would cost more in velocity than the risk of temporarily reduced coverage in that one area.
How can we catch flaky tests before they're merged instead of after? Run new or modified E2E tests several times in a row against a fresh, isolated environment before merging — most frameworks support repeated runs natively. A test that fails even once across several fresh-environment runs is flaky or non-deterministic. This is only cheap to do routinely when environments are ephemeral and disposable; it's prohibitively expensive to repeat against a shared, booked staging slot.
Remove environmental flakiness by construction.
Every PR gets a fresh, isolated, production-parity environment — no leftover state, no contention from other branches. It won't fix a race condition, but it stops blaming the wrong thing.
