Running Parallel E2E Suites at Scale Without Environment Contention

The Short Version

Splitting an E2E suite into eight parallel CI jobs looks like an 8x speedup on paper. In practice, if all eight jobs run their tests against the same shared staging environment, the speedup mostly does not materialize, because the tests are now fighting each other for the same database rows, the same background job queue, and the same rate-limited third-party sandbox accounts. The fix is not smarter sharding logic. It's giving each shard its own environment, so parallel CI jobs are actually testing against parallel infrastructure instead of a single shared bottleneck wearing eight different job names.

Why Sharding Alone Doesn't Deliver the Speedup

Most E2E test runners support splitting a suite across N parallel workers out of the box: Cypress's --parallel flag, Playwright's sharding, a custom matrix build in GitHub Actions. This genuinely parallelizes the test execution step, spinning up N CI runners instead of one. What it does not automatically parallelize is the thing those tests actually run against. If every one of those N runners points its BASE_URL at the same single staging environment, the tests are running concurrently but the system under test is not, and every test that touches shared state (a row count, a queue depth, a rate-limited API) becomes a source of intermittent, non-reproducible failure that has nothing to do with the code being tested.

The Symptom: Flaky Tests That Aren't Actually Flaky

This shows up as a specific, recognizable pattern: a test fails in the full parallel CI run but passes every time it's re-run alone locally. The instinct is to mark it flaky and add a retry. The actual cause is usually a different shard's test mutating the same data mid-run: shard 3's test creates an order while shard 7's test is mid-way through asserting an exact order count for the same account, and depending on timing, shard 7 either sees the extra row or doesn't.

Expected: 4 orders for account acct_492
Received: 5 orders for account acct_492

No amount of retry logic actually fixes this, since the underlying race is still there on the retry; it just becomes less likely to trigger on any single re-run, which makes the flakiness intermittent enough to get ignored rather than actually investigated.

The Fix: One Environment Per Shard, Not One Environment Shared

The structural fix removes the shared state entirely, by giving every shard its own fully isolated environment rather than a shared instance every shard points at:

1# CI matrix: 8 parallel jobs, each creating its own environment
2strategy:
3  matrix:
4    shard: [1, 2, 3, 4, 5, 6, 7, 8]
5
6steps:
7  - run: |
8      bns environments create --from-path bunnyshell.yaml \
9        --name "e2e-shard-${{ matrix.shard }}-${{ github.run_id }}" \
10        --project <PROJECT_ID> --k8s <CLUSTER_ID> --deploy
11  - run: npx playwright test --shard=${{ matrix.shard }}/8 --base-url="$SHARD_ENV_URL"
12  - run: |
13      bns environments delete --id "$SHARD_ENV_ID" --force

Each of the 8 CI jobs creates its own environment, runs its slice of the suite against it exclusively, and tears it down when finished. No two shards ever touch the same database, the same queue, or the same rate-limited sandbox account, because no two shards ever share an environment. The flakiness described above disappears not because the tests got more robust, but because the race condition they were exposing no longer exists.

Sizing the Cluster for Real Concurrent Load

Running 8 (or more) full-stack environments at once, even briefly during a CI run, requires the underlying cluster to actually have that capacity available at the moment CI needs it, not just in aggregate over the course of a day. This is a capacity planning problem, and it needs a real number, not an estimate:

1$ bns environments list --project <PROJECT_ID> -o json | \
2    python3 -c "import json,sys; d=json.load(sys.stdin); print(len([e for e in d if e.get('operationStatus')=='running']))"
3
48

Running this check across several days of real CI activity, especially at the specific times parallel E2E runs actually happen (usually clustered around when PRs get opened or merge queue entries process), gives an honest peak-concurrency number to size a node-pool autoscaler ceiling against. See Staging Environment Capacity Math for the underlying queuing math that explains why undersizing this by even a small margin produces disproportionately long wait times rather than a gradual slowdown.

Sharding by Duration, Not File Count

A second, more subtle bottleneck: splitting a suite into N shards by file count alone assumes each file takes roughly the same time to run, which is rarely true in practice. One test file exercising a complex checkout flow can take five times longer than a simple settings-page test file. Static file-count sharding puts both in separate shards with no regard for actual runtime, which means the slowest shard becomes the long pole every single run while faster shards finish and sit idle.

$ npx playwright test --shard=3/8 --reporter=json > shard-3-results.json

Most modern test runners support duration-based sharding using historical timing data instead of a naive file split, which balances actual wall-clock time across shards rather than file count. Combined with isolated per-shard environments, this gets the full suite's wall-clock time close to (total test time) / (number of shards), which is the actual goal parallelization is supposed to deliver and rarely does without both pieces in place.

Cleaning Up After a Cancelled or Failed CI Run

A run that gets cancelled mid-flight (a developer pushes a fix before the previous run finishes, triggering CI to cancel the in-progress job) can leave shard environments running with nothing left to destroy them, since the teardown step that would normally run at the end of the job never executes on a cancelled run. Left unaddressed, this accumulates orphaned environments at a rate proportional to how often CI runs get cancelled and superseded, which for an actively developed PR can be often.

1$ bns environments list --project <PROJECT_ID> -o json | \
2    python3 -c "
3import json, sys
4from datetime import datetime, timezone
5d = json.load(sys.stdin)
6stale = [e for e in d if e['name'].startswith('e2e-shard-')]
7print(f'{len(stale)} shard environments currently exist')
8"

The reliable fix is a scheduled cleanup job (running hourly, independent of any individual CI run's own teardown step) that deletes any environment matching the e2e-shard-* naming pattern older than a fixed threshold, so an orphaned environment from a cancelled run gets reclaimed automatically within the hour rather than accumulating indefinitely.

What This Looks Like at Different Scales

A team running a five-minute E2E suite a few times a day rarely needs more than two or three shards, and the contention this guide describes may never become visible enough to notice. The pattern matters once a team crosses into running the suite on every push to every open PR, several times an hour across several active PRs simultaneously, which is exactly the volume AI coding agents tend to produce. At that volume, shared-environment contention stops being an occasional flaky test and starts being a meaningful fraction of all test failures, at which point isolating shards is less an optimization and more a prerequisite for the test suite's results being trustworthy at all.

Measuring Whether It Actually Worked

After moving to per-shard environments, confirm the change delivered the expected result rather than assuming it did, by tracking two numbers across a week of real CI runs: total suite wall-clock time (should drop close to the theoretical N-way speedup) and the rate of previously-flagged-flaky test failures (should drop toward zero for any test whose flakiness was actually shared-state contention rather than a genuine race in the application itself). A test that keeps failing intermittently after this change has a real bug worth investigating on its own merits, not a phantom caused by shared infrastructure.

The Cost Tradeoff, Honestly

Running eight isolated full-stack environments briefly, several times an hour, costs more in raw compute than running eight test jobs against one shared always-on staging instance. This is a real tradeoff, not a free win, and it is worth stating plainly rather than glossing over. The comparison that actually matters, though, is not "isolated environments cost more than a shared one," it's "isolated environments cost more than a shared one that produces trustworthy results," and a shared environment producing intermittent, non-reproducible failures has a cost too, just one that shows up as engineering time spent chasing phantom flakiness instead of as a cloud bill line item.

In practice, the actual cost delta is smaller than it first appears, because ephemeral shard environments only exist for the duration of the CI run itself, typically minutes, not the full day a shared staging environment sits running. Eight environments existing for fifteen minutes each, several times a day, costs meaningfully less than one might assume from "eight environments" alone, and considerably less than the engineering time spent re-running flaky tests and investigating false failures on a contended shared instance.

Choosing a Shard Count That Actually Helps

More shards is not unconditionally better. Each shard's environment has its own fixed startup cost, the time for the database to initialize, seed data to load, and the application to become ready, before a single test even runs. Splitting a five-minute test suite into twenty shards, each with a ninety-second environment startup cost, can end up slower in wall-clock time than eight shards would have been, since the fixed per-shard overhead starts to dominate once shard count grows large relative to the total suite runtime.

1Total test time: 40 minutes
2Per-shard environment startup overhead: 90 seconds
3
44 shards:  10 min tests + 1.5 min overhead = 11.5 min wall clock
58 shards:  5 min tests  + 1.5 min overhead = 6.5 min wall clock
616 shards: 2.5 min tests + 1.5 min overhead = 4 min wall clock
732 shards: 1.25 min tests + 1.5 min overhead = 2.75 min wall clock (overhead now dominates)

The practical way to find a team's own right number is tracking actual wall-clock time across a few different shard counts against the real suite, rather than assuming higher is always faster; past a certain point, each additional shard adds more fixed startup overhead than it removes in parallel test time, and the curve flattens out or reverses.

Handling Shared External Dependencies Across Shards

Isolating each shard's own database and application components solves contention within the stack under test, but does not automatically solve contention against a third-party service every shard's tests might call: a payment sandbox, an email provider, a mapping API with its own rate limit. Eight shards hitting the same external sandbox simultaneously can trip that provider's own rate limiting, producing failures that look like flaky third-party integration issues but are actually a self-inflicted concurrency problem.

The same mock-server pattern used for deterministic seed data extends naturally here: each shard's environment runs its own local stub of the third-party service, seeded with fixture responses, so no shard's tests ever make a real outbound call to a shared external rate limit at all. See Seeding Realistic Database Data for E2E and QA Testing for the concrete mock-server configuration this reuses directly.

A Real Before-and-After

A useful way to confirm this pattern is actually delivering value, rather than assuming it based on the theory, is measuring the same suite's wall-clock time and flaky-failure rate before and after the change, across a comparable week of real CI activity in each state:

1Before (8 shards, 1 shared staging environment):
2  Median wall-clock time: 18 minutes
3  Flaky-test re-run rate: 22% of runs needed at least one re-run
4
5After (8 shards, 8 isolated environments):
6  Median wall-clock time: 7 minutes
7  Flaky-test re-run rate: 3% of runs needed at least one re-run

The wall-clock improvement comes from genuine parallelism finally taking effect; the flaky-rate improvement comes from removing the shared-state races that were the actual cause of most of what looked like flaky tests. Both numbers matter, and teams that only track wall-clock time sometimes miss that the flaky-rate improvement is often the bigger practical win, since it is the one that was actively eroding trust in the test suite's results before the change.

Related Reading

For the seed-data pattern each isolated shard environment needs to start from a consistent baseline, see Seeding Realistic Database Data for E2E and QA Testing. For the full create-test-destroy lifecycle each shard's environment goes through, see Automated PR Environments for E2E Testing.

Checklist

• Every parallel test shard runs against its own environment, never a shared instance
• Cluster autoscaler ceiling is sized against measured peak concurrent environments, not a guess
• Test sharding is duration-based, not just file-count-based, to balance actual wall-clock time
• A scheduled cleanup job reclaims orphaned shard environments left by cancelled CI runs
• Previously flaky tests are re-evaluated after isolating shards to confirm the flakiness is actually gone