Auditing and Cleaning Up Stale Non-Production Environments

The Short Version

Auto-destroy is a real and necessary control, and it only fires on the specific trigger it's wired to, usually a PR closing or merging. Every environment that never goes through that trigger, an abandoned branch whose PR sat open and stale until someone eventually deleted it without ever properly closing it through the normal flow, a manually created test environment someone spun up for a one-off investigation and forgot about, an environment created directly via the API by a script that never got a matching cleanup step, quietly accumulates outside auto-destroy's reach. None of these show up as a single dramatic incident. They show up gradually, as a cluster that's mysteriously running more concurrent environments than anyone can account for and a monthly bill that's crept up without a corresponding increase in actual team activity.

Why Auto-Destroy Alone Isn't Enough

Auto-destroy solves the common case extremely well: the normal PR lifecycle, opened, reviewed, merged or closed, environment destroyed. The gap is everything outside that normal path. A PR that gets abandoned rather than formally closed (a developer just stops working on a branch and it sits there for months) may never trigger the close event auto-destroy listens for, depending on exactly how the team's workflow and auto-destroy trigger are configured. An environment created manually via bns environments create for a one-off debugging session has no PR lifecycle attached to it at all, so there is no merge or close event to ever fire. A scripted or API-driven workflow that creates environments programmatically for some automated purpose can, if its own cleanup step has a bug or gets skipped on a failure path, leave environments running indefinitely with nothing watching for them specifically.

Finding What's Actually Running

The starting point for any cleanup effort is an honest, current inventory, not an assumption based on what the team remembers creating:

1$ bns environments list --project <PROJECT_ID> -o json | \
2    python3 -c "
3import json, sys
4data = json.load(sys.stdin)
5running = [e for e in data if e.get('operationStatus') == 'running']
6print(f'{len(running)} environments currently running')
7for e in running:
8    print(f\"  {e['id']}  {e['name']}  ({e['type']})\")
9"
10
1147 environments currently running
12  8AEJQ1oXW7  staging (primary)
13  LPDqgKKEzk  pr-482-clone (ephemeral)
14  x9FkR2mNop  test-migration-debug (ephemeral)
15  ...

Running this across every project in the organization, not just the ones a team happens to remember actively using, is what surfaces the genuinely surprising cases: a project nobody has touched in months that still has environments quietly running against it.

Determining Actual Last Activity

The environment listing itself does not carry an explicit "last activity" timestamp, but the most recent pipeline or event associated with an environment is a reliable proxy for it, since any real activity (a redeploy, a stop, a start) generates a corresponding pipeline or event record:

1$ for env_id in $(bns environments list --project <PROJECT_ID> -o json | python3 -c "import json,sys; print('\n'.join(e['id'] for e in json.load(sys.stdin)))"); do
2    last=$(bns pipeline list --environment "$env_id" --sort createdAt:desc -o json | \
3      python3 -c "import json,sys; d=json.load(sys.stdin); print(d[0]['createdAt'] if d else 'never')")
4    echo "$env_id: last activity $last"
5  done

Sorting this output by last-activity date, oldest first, immediately surfaces the environments that have gone silent longest, which is a much better prioritization signal than guessing based on which environment names sound unfamiliar.

Setting a Reasonable Staleness Threshold

There's no universal number here, and picking one requires balancing false positives (flagging an environment that's genuinely still in active, if infrequent, use) against the cost of doing nothing. A reasonable starting default for ephemeral and manually-created test environments is 14 to 30 days without any deploy or pipeline activity. Long-lived primary or staging environments, and anything explicitly tagged as a demo environment, should be exempted from this specific check entirely, or given a much longer threshold, since infrequent activity on a long-lived environment is often expected and not actually a sign of abandonment.

1from datetime import datetime, timedelta, timezone
2
3STALE_THRESHOLD_DAYS = 21
4cutoff = datetime.now(timezone.utc) - timedelta(days=STALE_THRESHOLD_DAYS)
5
6stale = [
7    env for env in environments
8    if env['type'] == 'ephemeral'
9    and datetime.fromisoformat(env['last_activity']) < cutoff
10]

Staging the Rollout: Report Before You Delete

Moving straight to automatic deletion the first time this audit runs is a good way to delete something someone actually needed, even with a reasonable threshold, since any threshold has edge cases. The safer rollout runs the audit as a reporting-only step first, for several weeks, giving the team a chance to notice any false positives in the criteria before anything actually gets deleted:

1$ python3 audit_stale_environments.py --project <PROJECT_ID> --days 21 --report-only
2
3Stale environments (no activity in 21+ days):
4  x9FkR2mNop  test-migration-debug   (28 days idle)
5  q7VtNz3XcW  demo-feature-flag-poc  (45 days idle)
6
7Report posted to #platform-alerts. No environments deleted (--report-only).

After a few cycles of this report matching what the team actually expects (nothing flagged that turned out to still be needed), moving to automatic deletion with a short grace-period notification is a reasonable next step: flag the environment, notify whoever created it or last deployed to it, wait a fixed number of days (48 to 72 hours is common), and only then delete if nobody objects.

$ bns environments delete --id x9FkR2mNop --force

Turning This Into a Scheduled Job

Like the other recurring checks described elsewhere in this cluster, this doesn't need a continuously running service, a weekly scheduled job is enough to catch accumulation well before it becomes a real cost problem:

1on:
2  schedule:
3    - cron: '0 8 * * MON'
4
5jobs:
6  audit-stale-environments:
7    runs-on: ubuntu-latest
8    steps:
9      - run: python3 scripts/audit_stale_environments.py --project <PROJECT_ID> --days 21

Preventing Accumulation at the Source, Not Just Cleaning Up After It

The audit-and-cleanup process described here is a necessary safety net, not a substitute for closing the gaps that let stale environments accumulate in the first place. Every environment created through an automated process (a CI pipeline, a script) should have a corresponding, tested teardown path for every failure mode, not just the happy path, since a script that creates an environment and then errors out before its own cleanup step runs is exactly the kind of gap that produces the orphaned environments this audit exists to catch. See Auto-Destroy After Merge for closing the most common specific gap, PR environments that don't get destroyed reliably on merge or close.

Distinguishing Stale From Simply Stopped

It's worth being careful not to conflate a genuinely abandoned environment with one that's deliberately stopped to save cost between uses, since these look similar (no recent activity) but require opposite treatment. An environment stopped via Stop and Start Environments On-Demand is intentionally idle and preserving state for a planned future restart; deleting it destroys work someone is actively planning to resume. The audit criteria should check the environment's actual status, not just its activity timestamp, and exclude anything already in a stopped state from the stale-and-should-be-deleted bucket, treating it instead as correctly following a different, already-intentional pattern.

Attaching a Cost Estimate to Each Stale Environment

A stale-environment report that just lists names and idle durations is useful; one that attaches an approximate cost figure to each entry is considerably more persuasive when the goal is getting a team to actually act on the cleanup rather than deprioritizing it indefinitely. Estimating this doesn't require a precise billing integration, a rough figure based on the environment's own component resource requests, multiplied by days idle, is enough to make the report concrete rather than abstract:

1def estimate_monthly_cost(env, cost_per_cpu_hour=0.03, cost_per_gb_hour=0.006):
2    cpu_cores = sum(c['resources']['requests']['cpu_millicores'] for c in env['components']) / 1000
3    memory_gb = sum(c['resources']['requests']['memory_mb'] for c in env['components']) / 1024
4    hourly = (cpu_cores * cost_per_cpu_hour) + (memory_gb * cost_per_gb_hour)
5    return round(hourly * 24 * 30, 2)

1Stale environments (no activity in 21+ days), estimated monthly cost:
2  x9FkR2mNop  test-migration-debug   (28 days idle)   ~$34/month
3  q7VtNz3XcW  demo-feature-flag-poc  (45 days idle)   ~$61/month
4
5Total reclaimable: ~$95/month from 2 environments

A weekly report showing a running reclaimable total, rather than isolated per-environment figures, tends to get acted on faster, since it reframes cleanup from "tidying up" to "a specific, recoverable number on the monthly bill," which is a much easier case to make in a planning meeting.

Environments Created by People Who've Since Left

A specific and easy-to-miss case: an environment created by someone who has since left the team or the company, which nobody currently on the team feels ownership over or confidence deleting, since nobody knows what it was actually for. These accumulate specifically because the natural social check ("let me ask whoever made this") is no longer available, and the safer-feeling default becomes leaving it alone indefinitely rather than risking deleting something important.

The practical fix is treating "creator no longer with the team" as an explicit flag in the audit, surfaced separately from ordinary staleness, and routing it to whoever currently owns that project or team for an active decision (keep, document and hand off, or delete) rather than letting the absence of an obvious owner default to permanent inaction. An environment nobody can explain the purpose of, after a reasonable attempt to find out, is a stronger candidate for deletion than one that's merely idle, not a weaker one, since an idle environment with a known owner and purpose might still be intentional, while an unexplainable one almost never has a good reason to keep running.

Building This Into Onboarding and Offboarding, Not Just a Recurring Audit

The most durable fix to this whole class of problem is upstream of any audit script: making environment ownership explicit at creation time, rather than trying to reconstruct it later from activity logs. Requiring a --tag owner:<team-or-person> (or an equivalent required field) on every environment created outside the automated PR flow means the audit script never has to guess whose it was in the first place.

1$ bns environments create --from-path bunnyshell.yaml \
2    --name "migration-debug" --project <PROJECT_ID> --k8s <CLUSTER_ID> \
3    --tag "owner:jane.doe" --tag "purpose:investigate-slow-query" --deploy

Pairing this with an offboarding checklist item, "reassign or delete any environments tagged to the departing person," closes the loop entirely: instead of an audit discovering an orphaned environment months after someone left, the departure itself triggers an explicit decision about what happens to anything they were the last known owner of, at the moment the information is freshest and easiest to act on.

Related Reading

For closing the specific gap of PR environments not being destroyed reliably on merge, see Auto-Destroy After Merge. For the API surface this audit script's cross-project inventory pulls from, see Automate Everything: Using the Bunnyshell Public API.

Setting Expectations Before the First Report Goes Out

Introducing this audit for the first time in a team that's never had one tends to land better with a short heads-up beforehand than as a surprise report naming specific environments (and implicitly, specific people) out of nowhere. A brief note explaining the threshold being used, the report-only phase it's starting in, and an invitation to flag any known exceptions before the first real report goes out, heads off the defensive reaction that can otherwise follow a first cleanup report that reads as an accusation rather than routine hygiene.

Checklist

• A full inventory of running environments is pulled across every project, not just the ones a team remembers actively using
• Last activity is determined from actual pipeline or event history, not assumed
• Long-lived and demo environments are exempted from the ephemeral-environment staleness threshold
• The audit runs in report-only mode for several cycles before moving to automatic deletion
• Stopped-but-intentional environments are excluded from the stale bucket, not conflated with abandoned ones