Running Playwright and Cypress Suites in Ephemeral Environments

Running an E2E suite against localhost proves the app works on your machine. Running it against a shared staging server proves the app works on a server three other branches have also touched recently. Neither is what "the PR works" actually means. This guide covers wiring Playwright or Cypress — without touching the test files themselves — to run against a real, isolated, per-PR environment instead, plus the specific configuration, parallelization, authentication, and debugging patterns that come up once dynamic, per-PR URLs replace a fixed staging domain.

The case for why this matters is covered separately, including why AI coding agents specifically make this migration more urgent rather than optional. This is the how — framework-agnostic where the two frameworks agree, and explicit about where they diverge.

The one thing that actually changes: the base URL

Both frameworks resolve every relative URL in a test against a single configured base. Point that base at the ephemeral environment for the current PR, and every existing test — written against /login, /api/checkout, whatever relative paths your suite already uses — runs against the real thing with zero changes to the test files. Everything else in this guide is a consequence of that one change, not a separate migration step.

Playwright

1// playwright.config.ts
2import { defineConfig } from '@playwright/test';
3
4export default defineConfig({
5  use: {
6    baseURL: process.env.PREVIEW_URL || 'http://localhost:3000',
7    trace: 'retain-on-failure',
8  },
9  // No webServer block — the app is already running remotely.
10  // webServer is for spinning up a local dev server; omit it entirely
11  // when testing against a deployed environment.
12  retries: process.env.CI ? 2 : 0,
13  workers: process.env.CI ? 4 : undefined,
14});

Cypress

1// cypress.config.js
2const { defineConfig } = require('cypress');
3
4module.exports = defineConfig({
5  e2e: {
6    baseUrl: process.env.PREVIEW_URL || 'http://localhost:3000',
7    retries: { runMode: 2, openMode: 0 },
8  },
9});

Cypress resolves baseUrl the same way — every cy.visit('/dashboard') call becomes a request against the environment, not localhost. retries is set explicitly rather than left at Cypress's default in both examples above, since a suite migrating from a stable localhost target to a freshly provisioned remote one benefits from a small retry buffer while the team confirms which failures are real and which are provisioning-timing noise from the migration itself.

Wiring it into CI

The base URL has to come from somewhere at run time. In a GitHub Actions workflow where Bunnyshell provisions the environment as a prior step, that's a single output variable:

1- name: Deploy PR environment
2  uses: bunnyshell/deploy-action@v2
3  id: get-url
4  with:
5    bunnyshell-token: ${{ secrets.BUNNYSHELL_TOKEN }}
6    bunnyshell-organization: ${{ secrets.BUNNYSHELL_ORGANIZATION }}
7    environment-id: ${{ env.ENV_ID }}
8    wait: true
9    timeout: 600
10
11- name: Run Playwright tests
12  run: npx playwright test --reporter=github
13  env:
14    PREVIEW_URL: ${{ steps.get-url.outputs.preview_url }}
15
16- name: Run Cypress tests
17  uses: cypress-io/github-action@v6
18  with:
19    config: baseUrl=${{ steps.get-url.outputs.preview_url }}
20  env:
21    CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}

The wait: true on the deploy step matters as much as the URL itself — it blocks until Bunnyshell reports the environment healthy, so the test step never starts against a target that's still booting. The full deployment workflow, including teardown on merge, is covered in the GitHub Actions guide; this is the piece specific to wiring E2E frameworks into it.

GitLab CI and other pipelines

The pattern is identical outside GitHub Actions — provision, capture the URL, run tests against it — only the syntax for passing the URL between jobs changes. In GitLab CI, using dotenv artifacts to pass the URL from a deploy job to a test job:

1deploy_preview:
2  stage: deploy
3  script:
4    - bns environments deploy --id "$ENV_ID" --wait
5    - echo "PREVIEW_URL=$(bns environments show --id $ENV_ID --output json | jq -r '._embedded.components[0].endpoints[0]')" >> deploy.env
6  artifacts:
7    reports:
8      dotenv: deploy.env
9
10e2e_tests:
11  stage: test
12  needs: [deploy_preview]
13  script:
14    - npx playwright test --reporter=github

The full GitLab CI integration, including MR-triggered provisioning and cleanup, is covered separately. The E2E wiring itself — reading PREVIEW_URL into baseURL — is identical to the GitHub Actions version above regardless of which CI system provisions the environment.

Parallelization: sharding vs. one environment per shard

Both frameworks split a suite across parallel workers to keep CI time down. Against an ephemeral environment, there are two ways to structure that:

Shard against one environment. All workers point at the same environment's URL, and Playwright's --shard or Cypress's parallelization splits the test files between them. Fast to set up, no extra provisioning cost, and correct as long as your tests don't share mutable state (the same database rows, the same user account) across workers.

One environment per shard. Each worker gets its own environment, fully isolated from the others. This removes cross-worker interference entirely, at the cost of provisioning N environments instead of one for a single PR's test run.

Start with the first approach. Move to the second only once you can point at actual test flakiness caused by parallel workers stepping on shared state — usually visible as tests that pass reliably in isolation but fail intermittently only when run in parallel, which is order-dependent or environmental flakiness specifically, not a framework problem.

Sharding against one environment, concretely, uses Playwright's built-in --shard flag with a matrix strategy in CI:

1jobs:
2  e2e:
3    strategy:
4      matrix:
5        shard: [1/4, 2/4, 3/4, 4/4]
6    steps:
7      - name: Run Playwright shard
8        run: npx playwright test --shard=${{ matrix.shard }}
9        env:
10          PREVIEW_URL: ${{ needs.deploy.outputs.preview_url }}

Each matrix job runs a quarter of the suite in parallel, all four pointed at the same PREVIEW_URL. Cypress achieves the same result via its own --parallel flag combined with the Cypress Cloud (or a self-hosted equivalent) to coordinate work distribution across runners.

One environment per shard instead means each matrix entry provisions and tears down its own environment — the deploy step moves inside the matrix rather than running once beforehand. This is meaningfully more infrastructure per PR, and it's worth confirming the flakiness is actually cross-worker interference (re-run the failing shard alone, see if it passes) before paying that cost, rather than assuming isolation is the fix without checking.

Debugging a failing test against a live environment

When a test fails in CI against an ephemeral environment and the failure isn't obvious from the log, both frameworks support pointing local tooling at the same remote target rather than only being able to debug against localhost:

1# Playwright: run a single test file against the PR environment, headed, with the trace viewer
2PREVIEW_URL="https://pr-482.preview.yourapp.com" npx playwright test checkout.spec.ts --headed --trace on
3
4# Then inspect the trace with full DOM snapshots, network calls, and console logs
5npx playwright show-trace trace.zip

# Cypress: open the interactive runner against the same remote environment
CYPRESS_BASE_URL="https://pr-482.preview.yourapp.com" npx cypress open

Both approaches point the exact same test file at the exact same environment the CI failure occurred against — not a local rebuild that may not reproduce the issue. Since ephemeral environments are destroyed on merge, this only works while the PR is still open; for a failure discovered after merge, the environment is gone and reproducing it means recreating the same PR state manually.

API-level checks alongside E2E

Not everything worth verifying against a live environment needs a browser. Both frameworks support making direct API requests within the same test run, useful for asserting on backend state a UI interaction triggered without paying the cost of driving a browser through every check:

1// Playwright: request context reuses the same baseURL config
2test('order confirmation triggers webhook', async ({ page, request }) => {
3  await page.goto('/checkout');
4  // ...complete checkout through the UI...
5  const response = await request.get('/api/orders/latest/webhook-status');
6  expect(response.ok()).toBeTruthy();
7});

// Cypress: cy.request() resolves against the same baseUrl
cy.request('/api/orders/latest/webhook-status').its('status').should('eq', 200);

This matters more than it might seem for AI-generated code specifically: the failure patterns most common in AI-generated PRs — a webhook handler that silently drops an unrecognized payload, a background job that never actually enqueues — often have no visible UI symptom at all. The checkout button still shows a success message; the confirmation email never sends. A UI-only E2E suite can miss this class of bug entirely unless it also asserts on backend state directly, which is a strong argument for writing at least one API-level assertion into any test that exercises a flow with an asynchronous side effect behind it.

Screenshots, videos, and trace artifacts

Both frameworks capture failure evidence automatically, and it's worth wiring the capture settings deliberately rather than leaving defaults, since default settings are usually tuned for local development, not CI triage:

1// playwright.config.ts
2use: {
3  baseURL: process.env.PREVIEW_URL,
4  trace: 'retain-on-failure',
5  screenshot: 'only-on-failure',
6  video: 'retain-on-failure',
7},

1// cypress.config.js
2module.exports = defineConfig({
3  e2e: {
4    baseUrl: process.env.PREVIEW_URL,
5    video: true,
6    screenshotOnRunFailure: true,
7  },
8});

Upload these as CI artifacts attached to the specific run, and — since the environment that produced the failure is torn down on merge — the artifact becomes the only record of what the environment actually looked like at failure time. Tag artifacts with the PR number and environment ID, not just a generic build number, so a failure investigated days later can still be traced back to the exact environment it happened against.

Both frameworks also support accessibility scanning within the same run, worth mentioning since it's the same pattern as every other check covered here — run against the real environment, not a special case: Playwright integrates with axe-core via @axe-core/playwright, and Cypress via cypress-axe, both asserting on the actual rendered page in the actual environment rather than a static analysis of the source. Wiring this in is a few lines added to an existing test, not a separate testing pipeline.

Reusing authenticated sessions across tests

Logging in through the UI before every single test is slow, and against a freshly provisioned environment — where no session exists yet from a previous run — it's also the default most teams start with before optimizing. Both frameworks support authenticating once and reusing that session across a test file or an entire run:

1// Playwright: authenticate once in a setup project, reuse the storage state everywhere else
2// auth.setup.ts
3import { test as setup } from '@playwright/test';
4
5setup('authenticate', async ({ page }) => {
6  await page.goto('/login');
7  await page.fill('#email', 'test@example.com');
8  await page.fill('#password', 'test-password');
9  await page.click('#login-button');
10  await page.context().storageState({ path: 'storage-state.json' });
11});

1// playwright.config.ts references it for every other project
2projects: [
3  { name: 'setup', testMatch: /auth\.setup\.ts/ },
4  { name: 'tests', use: { storageState: 'storage-state.json' }, dependencies: ['setup'] },
5],

1// Cypress: cache the login session by key, restore it across tests
2Cypress.Commands.add('loginByApi', () => {
3  cy.session('test-user', () => {
4    cy.request('POST', '/api/login', { email: 'test@example.com', password: 'test-password' })
5      .then((res) => window.localStorage.setItem('token', res.body.token));
6  });
7});

The environment-specific detail worth flagging: this session data is scoped to the environment it was created against. A storage-state.json captured against one PR's environment is meaningless against a different PR's environment — session tokens, cookies, and auth state don't carry across environments the way they might across a stable, long-lived staging domain. Regenerate it as part of each test run against the current environment, not as a cached artifact reused across PRs. Caching the setup step's output within a single CI run (so multiple sharded workers share one login instead of each logging in separately) is still worth doing — the constraint is specifically against reusing it across environments, not against reusing it within one.

Pitfalls specific to dynamic, per-PR URLs

A handful of issues show up reliably the first time a team points E2E tests at dynamically generated subdomains instead of a fixed staging URL, and none of them are framework bugs:

CORS configured for a fixed origin. If your API's CORS policy allow-lists staging.yourapp.com explicitly, it will reject requests from pr-482.yourapp.com the moment the domain changes per PR. Configure a pattern match for your preview subdomain format instead of a fixed list — a one-time change, not a per-environment one.

Cookies scoped too narrowly. A cookie set with Domain=staging.yourapp.com won't be sent on pr-482.yourapp.com. If your frontend and API need to share a session within the same environment, scope cookies to the parent domain rather than a specific subdomain.

Self-signed or wildcard TLS certificates. Automated per-PR environments typically provision certificates dynamically. Test runners that pin a specific certificate fingerprint, or that fail on certificate hostname mismatches for wildcard certs, need that check relaxed for preview domains specifically — not disabled globally.

WebSocket connections hardcoded to a host. Real-time features often have a ws:// or wss:// URL configured separately from the main API base. It needs the same environment-variable treatment as baseURL — easy to miss because it's a second place the same information has to live.

Cypress environment variables not reaching the browser context. Cypress's Node process and the browser it drives are separate contexts. A plain process.env.PREVIEW_URL read in cypress.config.js works fine for baseUrl, but a test that needs to reference the same value from within browser-executed code needs it exposed explicitly via the env config block (env: { previewUrl: process.env.PREVIEW_URL }) and accessed with Cypress.env('previewUrl') — a plain environment variable read inside a test file silently returns undefined in the browser context, not an error, which makes this specific mistake easy to miss until a test behaves unexpectedly.

Applications served from a subpath. If the app isn't served from the domain root (preview.yourapp.com/app/ rather than preview.yourapp.com/), baseURL needs the subpath included, and any test using absolute paths (page.goto('/dashboard') instead of a relative dashboard) will silently 404 against the environment while working fine locally if local development happens to serve from root. Worth checking explicitly rather than assuming path handling that worked on localhost transfers unchanged.

Responsive and device-specific testing

Neither framework needs anything environment-specific to test multiple viewports against the same ephemeral environment — device emulation is orthogonal to where the app under test is running, which is worth confirming explicitly since it's a common point of confusion the first time a team wires this up:

1// Playwright: run the same spec against multiple device profiles
2import { devices } from '@playwright/test';
3
4export default defineConfig({
5  use: { baseURL: process.env.PREVIEW_URL },
6  projects: [
7    { name: 'Desktop Chrome', use: { ...devices['Desktop Chrome'] } },
8    { name: 'Mobile Safari', use: { ...devices['iPhone 14'] } },
9  ],
10});

1// Cypress: set viewport per test or globally
2describe('checkout flow', () => {
3  it('works on mobile', () => {
4    cy.viewport('iphone-x');
5    cy.visit('/checkout'); // resolves against baseUrl, same as desktop run
6  });
7});

Both run the identical test logic against the identical PREVIEW_URL — only the emulated viewport and user agent change. This is useful specifically for AI-generated frontend code, where a layout change verified only at desktop width is a common gap: an agent asked to add a component to a page has no reason to check mobile breakpoints unless the test suite does.

The same principle extends to cross-browser coverage. Playwright ships WebKit and Firefox engines alongside Chromium, so a projects array can run the same spec across all three without a separate test file per browser — { name: 'WebKit', use: { ...devices['Desktop Safari'] } } alongside the Chrome and mobile Safari entries above. Cypress supports Chromium-family browsers natively and Firefox via a separate launch flag. Neither requires anything environment-specific beyond what's already configured — the browser matrix and the environment target are independent axes, and testing across both is a matter of combining configuration, not building separate infrastructure for each.

Troubleshooting checklist

If tests pass locally and against shared staging but fail against a fresh ephemeral environment, check these in order:

1. Is the environment actually healthy when the test step starts, or did the test run start before provisioning finished? A missing or insufficient health check before the test step is the single most common cause.
2. Is baseURL actually being read from the environment variable, or is a hardcoded fallback silently winning? Log the resolved URL at the start of the test run.
3. Does the environment have working test data? A healthy but empty environment fails every test that assumes seeded data exists — see our guide on test data management for parallel, per-PR environments.
4. Is a WebSocket, CORS, or cookie configuration hardcoded to the shared staging domain specifically, per the pitfalls above?
5. Is the app served from a subpath, and does baseURL include it? A test using absolute root-relative paths will 404 silently against a subpath deployment.
6. For Cypress specifically: is the environment variable actually reaching the browser context, or only the Node process that launched the test runner?
7. Did this pass reliably in isolation but fail only when run in parallel? That's cross-worker interference, not an environment or configuration problem — see the sharding section above before assuming it's a broken test.

Most failures that look like "the test is broken" at this stage are actually one of these seven, not the test itself. Work through them in order rather than guessing — the fastest path to a real fix is ruling out the cheap, common causes (health check, base URL, test data) before investigating anything specific to the test logic itself.