Securing E2E Test Environments: IP Allowlisting and Secrets in CI Pipelines

The Blind Spot: "It's Just a Test Environment"

Once PR environments are automated (see Automated PR Environments for E2E Testing), a specific complacency creeps in: because these environments are temporary and clearly labeled as previews, they get treated as lower-stakes than production — and their security gets correspondingly less attention.

That complacency is usually wrong on two counts. First, ephemeral environments frequently hold real secrets: third-party sandbox API keys, database credentials, signing keys needed for an auth flow under test. Second, they get a real public URL by design — that's what makes them useful for E2E testing and review — which means "public URL + real secrets + less scrutiny than production" is a genuinely worse combination than production's "public URL + real secrets + lots of scrutiny."

This guide covers the two concrete levers that close that gap: restricting who can reach the environment (IP allowlisting), and making sure secrets never sit in plaintext anywhere they persist.

Lever One: IP Allowlisting

Bunnyshell supports restricting access to an environment's hosts at the ingress level using the standard NGINX ingress annotation for source-range allowlisting:

1hosts:
2  - hostname: 'api-{{ env.base_domain }}'
3    path: /
4    servicePort: 8080
5    k8s:
6      ingress:
7        annotations:
8          nginx.ingress.kubernetes.io/whitelist-source-range: '203.0.113.0/24,198.51.100.14/32'

Anything outside the listed CIDR ranges gets rejected at the ingress before it reaches your application — the restriction is enforced at the network edge, not inside your app code, so there is no way for an application bug to accidentally bypass it.

There's also an environment-wide restriction mode for cases where you want everything private by default and only specific hosts exposed publicly:

1kind: Environment
2name: my-app
3environmentVariables:
4  RESTRICT_ALL: true   # illustrative — see full IP restriction docs for the exact global toggle
5components:
6  - kind: Application
7    name: public-docs
8    hosts:
9      - hostname: 'docs-{{ env.base_domain }}'
10        public: true   # explicitly opts this host out of the environment-wide restriction

The pattern to take away: restriction defaults to "closed," and individual hosts opt into being public, rather than the reverse. That ordering matters — it is much harder to accidentally leave something exposed when exposure is the opt-in case.

The CI IP Problem, and How to Actually Solve It

The most common practical objection to IP allowlisting on a test environment is that CI runners (GitHub Actions, GitLab CI, CircleCI) run on ephemeral infrastructure with IPs that change between runs — so a naive "allowlist my current IP" approach breaks the next time the pipeline runs from a different address.

Two workable solutions, in order of preference:

1. Allowlist your CI provider's published IP ranges. GitHub Actions, for example, publishes its runner IP ranges via a metadata API. These ranges change occasionally but are stable enough to maintain as a scheduled allowlist update rather than a per-run one.
2. Route E2E traffic through a fixed egress IP. A NAT gateway or a dedicated runner with a static IP in front of your CI jobs turns "changing CI IP" into "one known IP," which is simpler to allowlist and also gives you a single point to audit or revoke.

Reviewer and stakeholder access (people clicking the preview link from Slack, not CI) is a separate allowlisting decision — often those get a broader but still-restricted range (company VPN egress IP, for instance) rather than the CI-specific range.

Lever Two: Secrets That Never Sit in Plaintext

The second half of securing a test environment is making sure nothing sensitive is ever written to disk, git history, or a CI log in plaintext — including in the environment's own configuration file.

Bunnyshell supports two equivalent authoring patterns for this:

Option A — write plaintext once, let Bunnyshell encrypt on save:

1environmentVariablesGroups:
2  app:
3    DB_PASSWORD: SECRET["actual-plaintext-password"]

The plaintext only exists in this form for the moment before the environment is created or updated — Bunnyshell encrypts it server-side and stores the encrypted form from then on. This is confirmed directly: after creating an environment with a SECRET["..."] value, exporting its live definition back shows the stored, encrypted form, not the original plaintext:

1$ bns environments definition --id LPDqgKKEzk
2
3environmentVariablesGroups:
4    app:
5        DEMO_SECRET: 'ENCRYPTED[Q2dncFYIpW629swUrDuO0RJLADm2thdcUDcwttbm0mORfluCtcEgQ6t9PqkWKlFgWucJu5XvQJGnhMr3AMPilZEybndubHQ=]'

Option B — pre-encrypt before it ever touches a file, using the CLI directly:

$ echo -n "hunter2-db-password" | bns secrets encrypt --organization NA1Q2kyalg
Expression | ENCRYPTED[ckXzb+vOS9ebl6M+uYcYJnjrCWg6ZSUEjxgELT6/SRI3d18a4dGCBLXIypJ+UjnilZEybndubHQ=]

That ENCRYPTED[...] string is what you commit directly to bunnyshell.yaml — the plaintext never touches a file at all, since it went straight from stdin to the encrypt call. This is the safer of the two patterns for anything that will live in a git-tracked file reviewed by other people, since there is no intermediate commit (even a since-amended one) that ever contained the plaintext.

Decryption follows the same CLI, for cases where you need to verify a stored value (debugging a broken credential, auditing what's actually configured) rather than take it on faith:

$ bns secrets decrypt "ENCRYPTED[qO8fTvKr/E5+4nHMa/0OOz6/fXed0554ihdj+FLe2KWO5jvAHWAQtN3C5GqqTpHmL3efNeKVkTJud25sdA==]" --organization NA1Q2kyalg
Value | another-plaintext-secret

For the full mechanics of this workflow, see Encrypting and Decrypting Secrets via CLI.

The Third Leak Path People Forget: CI Logs

Even with secrets properly encrypted at rest, a very common leak path is a CI job printing an environment variable to its own log — either directly (echo $DB_PASSWORD left in from debugging) or indirectly (a verbose HTTP client logging request headers that include an auth token). This is not a Bunnyshell-specific problem, but it is the leak path most likely to undo all the encryption work above, because most CI providers treat job logs as far less sensitive than they actually are.

Practical mitigations:

• Mark CI-level secret variables as masked in your CI provider's settings, so accidental echoes are redacted in the rendered log even if the command itself wasn't careful.
• Avoid -v/verbose flags on HTTP clients and package managers in CI steps that handle authenticated requests, unless you've confirmed the tool redacts auth headers in verbose output.
• Treat "this ran in CI" as no different from "this ran on a shared machine" for logging discipline — CI logs are frequently more widely readable within an org than people assume.

The Compliance Angle: Why Auditors Care About "Just a Test Environment" Too

Teams operating under SOC 2, ISO 27001, or similar frameworks sometimes discover during an audit that their compliance scope implicitly extends to non-production environments they hadn't been thinking about — if a test environment holds anything derived from real customer data, or has credentials that could reach a production system (a shared third-party API key, for instance), an auditor will reasonably ask about its access controls, not just production's. Getting IP allowlisting and secret encryption right on test environments from the start, as described above, is meaningfully cheaper than retrofitting it under audit pressure, when someone is asking "show me the control that prevents unauthorized access to this environment" and the honest answer is "there isn't one yet." Treating test-environment security as a checklist item alongside production — not a separate, lower-stakes category — is the posture that holds up under that kind of scrutiny.

Secret Rotation: The Part Encryption Alone Doesn't Solve

Encrypting a secret (see Encrypting and Decrypting Secrets via CLI) protects it from being readable in plaintext at rest. It does not address a different, equally real risk: a credential that was valid and is now stale, because the person who provisioned it left the team, or the third-party service's security team recommends periodic rotation regardless of any known compromise. A practical rotation cadence for test-environment secrets:

• Third-party sandbox API keys: rotate on the vendor's recommended schedule, typically quarterly or on staff offboarding
• Database credentials for ephemeral environments: since these environments are destroyed and recreated per-PR anyway, they naturally rotate on every new PR if the primary environment's bunnyshell.yaml generates a fresh credential per environment rather than reusing one static value
• Any secret shared across long-lived environments (a primary staging environment, a persistent demo environment): rotate manually on a calendar reminder, since these don't get the automatic churn ephemeral environments do

The practical mechanism for rotating a stored secret is straightforward — re-encrypt the new value and update the environment's configuration:

echo -n "new-rotated-password" | bns secrets encrypt --organization <ORG_ID>
# then update bunnyshell.yaml with the new ENCRYPTED[...] value and redeploy

Incident Response: If a Test-Environment Secret Leaks Anyway

Despite the controls above, plan for the case where something still leaks — a CI log that wasn't properly masked, a screenshot shared in a support ticket that included a visible token. The response sequence that matters is the same regardless of whether the leaked credential belonged to a "real" production system or a test-environment sandbox account: rotate the credential immediately at the source (the third-party provider's dashboard, not just in Bunnyshell), audit whether the leaked credential was used for anything unexpected in the window it was exposed, and re-encrypt and redeploy the rotated value. Treating a test-environment credential leak as lower-urgency than a production leak is exactly the complacency this guide opened by warning against — the blast radius may genuinely be smaller, but "smaller" is not "zero," and the response should still be prompt.

Threat-Modeling a Fork-PR Preview Environment Specifically

The single highest-risk scenario this guide's controls address is a repository that automatically creates preview environments for pull requests from external forks — a common and valuable pattern for open-source projects or any codebase accepting outside contributions, and also the scenario where "public URL plus real secrets" is genuinely adversarial rather than merely careless. A fork PR's code is, by definition, code you have not reviewed yet running inside your infrastructure. The combination of controls in this guide — network isolation preventing lateral movement to other environments, IP restriction limiting who can even reach the environment's public endpoint, and secrets that are encrypted at rest and never logged in plaintext — is what makes it reasonable to run untrusted fork-PR code in an automated preview environment at all. Teams enabling this pattern should treat it as the highest-scrutiny case for these controls specifically, not assume the same defaults used for trusted-team PRs are automatically sufficient once external contributors are in scope.

A Note on Third-Party Sandbox Credentials Specifically

A specific category worth calling out because it's easy to underweight: third-party sandbox credentials (a payment processor's test-mode API key, an email provider's sandbox key) often feel lower-stakes than a production database password, since a leak "only" grants access to a sandbox, not real customer data. This reasoning has a gap — sandbox accounts frequently share infrastructure quotas or rate limits with the vendor relationship as a whole, and abuse of a leaked sandbox key (spamming a sandbox email API from a botnet, for instance) can affect your account standing with that vendor regardless of whether any real data was ever at risk. Treat sandbox credentials with the same SECRET[...]/ENCRYPTED[...] discipline as anything else in this guide, rather than reasoning that "it's just a sandbox key" exempts it.

Related Reading

For the network-level controls (IP allowlisting, isolation) that complement the secrets-handling practices in this guide, see IP-Restricting Staging Access and Custom Domains and Network Isolation. For the exact CLI mechanics behind encryption referenced above, see Encrypting and Decrypting Secrets via CLI.

A Minimal Security Checklist for E2E Test Environments

• Every host that doesn't need to be public is not public (allowlist restricts to CI + reviewer IPs, or is restricted entirely with explicit per-host opt-in)
• Every secret in bunnyshell.yaml is SECRET[...] or ENCRYPTED[...], never a bare plaintext string
• CI secret variables are marked masked at the provider level
• No verbose/debug logging flags are left enabled on steps handling authenticated requests
• Test environment credentials are distinct from production credentials, even for third-party sandboxes — never point a test environment at a production API key "just this once"

None of this is exotic — it is the same discipline production deployments already require, applied to environments that are easy to mentally categorize as "temporary, so it doesn't matter." The temporariness of the environment does not make a leaked credential temporary.