Regression Testing in PR Preview Environments
What Regression Testing Actually Checks
Regression testing isn't about validating new functionality — that's what E2E and acceptance tests are for. It's about re-confirming old functionality still works after something else changed, run against a test environment that's actually representative of production.
A regression suite accumulates over time. Every bug that's been fixed once earns a test that keeps checking it stays fixed. Every critical user flow gets a test that keeps confirming it still works, PR after PR, release after release. The suite grows; its job doesn't change.
Fatima maintained the regression suite for a 16-person healthtech team. Over two years it grew to 340 tests covering appointment booking, insurance verification, and patient records — each one added after a real bug had shipped and been fixed. When a routing library upgrade quietly broke insurance verification for out-of-network providers, the regression suite caught it in nine minutes. Without it, that bug would have reached production and affected real patients.
Why Regression Tests Get a Reputation for Being Flaky
Ask most engineering teams about their regression suite and you'll hear some version of "it's noisy, half the failures aren't real." That reputation is usually earned by the environment, not the tests.
Shared staging causes two specific failure patterns:
Version mismatch. The regression suite runs against whatever happens to be deployed to staging at that moment — which might be three commits behind the PR actually being tested, because someone else's deploy is queued ahead of it.
Contaminated state. A regression test that checks "user can complete checkout" can fail because a previous test run (from a different PR, hours earlier) left the cart in an unexpected state. The test isn't wrong. The environment it ran against wasn't clean.
Neither failure mode is a problem with the test suite. Both disappear the moment each run gets a genuinely isolated, freshly provisioned environment instead of a shared one that's accumulated hours or days of other people's test runs.
Give your regression suite a clean environment every time.
Full-stack, isolated environments per pull request — automatic, no leftover state from someone else's test run.
Running Regression Tests Against Per-PR Environments
The setup mirrors E2E testing, because for most teams the regression suite and the E2E suite share infrastructure, if not the exact same tests.
Environment per PR, not per team. When a pull request opens, a full-stack environment provisions from the existing Docker Compose, Helm, or manifest definition (see Environment as Code for how that definition is structured) — not a shared box everyone's PRs compete for.
Tiered execution. Running all 340 of Fatima's tests on every commit would slow the team down. A tiered approach fixes this:
| Tier | When it runs | What it covers |
|---|---|---|
| Smoke (10-15 tests) | Every push | Critical paths only — login, checkout, core workflow |
| Full regression | Every PR before merge, or nightly | The complete accumulated suite |
| Extended/manual | Pre-release | Edge cases, exploratory QA on the same environment |
Same environment, escalating coverage. All three tiers can run against the same per-PR environment — no need to provision separately for "smoke" versus "full" versus "manual." The environment doesn't change; how much of the suite runs against it does.
Distinguishing a Real Regression from a Flaky Test
Not every failure is a real bug, and not every failure is flakiness. Telling them apart matters, because treating a real regression as flaky lets a shipped bug through, and treating flakiness as a real regression wastes a debugging cycle on nothing.
The check: re-run the exact same test against the exact same fresh environment, several times, with no code changes in between. A result that's consistent every time is either a real regression (if it fails) or genuinely passing (if it doesn't). A result that flips between runs against an identical environment is flakiness — likely a race condition or timing issue in the test or the application code, not something the environment caused.
Jonas led backend for an 11-person logistics platform. A regression test for shipment-status updates started failing intermittently — passing on retry maybe 60% of the time. Rather than mark it flaky and move on, his team ran it 20 times against a fresh environment each time. It failed exactly when a specific webhook retry happened to land during the assertion window — a genuine race condition, not test flakiness. Fixing the underlying timing bug (not the test) eliminated the failures completely.
A SaaS platform serving institutional finance clients hit a related bottleneck from the other direction: environment setup itself took 2 days, which meant regression testing against a real environment barely happened before release. Cutting that to 7 minutes made it routine instead of exceptional.
Regression Testing With AI-Generated Code
Teams using Cursor, Copilot, or Claude Code accumulate regression risk faster, for a specific reason: AI-generated changes are more likely to touch code paths the author didn't fully trace through, because the agent optimized for the immediate task rather than the system's full history of prior bugs.
A regression suite is exactly the safety net for this — it doesn't need to understand why a fix mattered two years ago, it just needs to keep checking that the fix still holds. The volume problem is the same one E2E testing faces with AI-assisted development: more PRs per week means the regression suite runs more often, against more environments, in parallel. See QA automation without a DevOps engineer for how small teams handle that scaling without a dedicated hire.
Getting Started Without a Dedicated QA Engineer
Building a regression suite doesn't require a dedicated QA role, especially at the scale most small teams need. Start narrow: pick the 5-10 flows that would actually hurt the business if they broke silently, write tests for those, and add one more test every time a real bug slips through and gets fixed.
The infrastructure side — giving each run a clean environment instead of a shared, contaminated one — is the part that used to require platform engineering time. That's the piece a managed per-PR environment platform now handles automatically.
✅ Key Takeaways
- Regression testing re-confirms old functionality still works — it's not the same job as E2E testing new functionality, though the suites often share tests
- Most "flaky regression tests" are actually environment problems: version mismatch or contaminated state from shared staging
- A tiered approach (smoke on every push, full suite nightly or pre-merge, extended pre-release) keeps a growing suite from slowing the team down
- Distinguish real regressions from flakiness by re-running against an identical fresh environment several times — consistent results are real, flipping results are flaky
- The suite itself doesn't need a dedicated QA engineer to build at small-team scale; the environment layer underneath it used to be the hard part, and that's now a rented service, not a build
FAQ
What is regression testing?
Regression testing re-runs existing tests after a code change to confirm that something which used to work still works. It's not about testing new functionality — it's about catching cases where a change broke something unrelated that used to be fine.
What's the difference between regression testing and E2E testing?
They overlap but answer different questions. E2E testing validates a complete user workflow, often for new or changed functionality. Regression testing specifically re-checks previously working behavior after a change, using a suite that accumulates over time. Many regression suites are made of E2E and integration tests run repeatedly, not a separate test type.
Why do regression tests fail on shared staging but pass locally?
Usually because the shared environment has state from other in-flight PRs, a different deployed version than what's actually being tested, or leftover data from a previous test run. The regression suite isn't wrong — the environment it ran against wasn't a clean, isolated copy of the code under test.
How often should regression tests run?
On every pull request for a fast, high-value smoke subset, and on a schedule (nightly or pre-release) for the full suite. Running the entire regression suite on every commit is usually too slow; running none of it until release is too risky. Most teams land on a tiered approach.
Can regression testing be automated for a small team without a dedicated QA engineer?
Yes. The test suite itself doesn't require a dedicated QA role to write or maintain, especially at the size most small teams need. The harder part historically was giving each run a clean environment to test against, which a managed per-PR environment platform now handles without a dedicated hire.
Stop chasing false regression failures.
Run your suite against a fresh, isolated environment on every pull request.
