Debugging Failed Deploy Pipelines Before They Block a Merge

The Short Version

A pull request's environment fails to deploy, the merge queue blocks it, and the reflex is to assume the PR's code broke something. Sometimes it did. Just as often, the actual cause is one layer removed from the code entirely: an image that never finished pushing, a database component that was not actually ready when a dependent component tried to connect, or a password baked into a persistent volume from three deploys ago that no longer matches what the current config expects. Diagnosing which layer actually failed, fast, is the difference between a five-minute fix and an hour spent staring at application code that was never the problem.

Start With the Pipeline, Not the Application

The instinct when a deploy fails is to open the application's own logs first. That is usually the wrong starting point, because if the pipeline never got the application running at all, there are no meaningful application logs yet, only a component that never started. The actual first move is checking pipeline and job status directly:

1$ bns pipeline list --environment <ENV_ID> --status failed
2
3ID           STATUS   CREATED
4OG7nYbkwM3   failed   2026-09-29T14:02:11Z
5
6$ bns pipeline jobs --id OG7nYbkwM3 --jobStatus failed
7
8JOB          COMPONENT   STATUS   STEP
9j-4471       web         failed   deploy

This immediately narrows a multi-component deploy failure down to the one component whose job actually failed, rather than requiring a manual scan through every component's output looking for the failure. In a stack with five or six components, this alone usually cuts diagnosis time by more than half.

Filtering Logs to Just the Failure

Once the failing job is identified, the next command pulls logs scoped to only the failed step, not the entire pipeline's output:

1$ bns pipeline logs --id OG7nYbkwM3 --jobStatus failed --stepStatus failed
2
3[web] Step: deploy
4[web] Error: container_error_pull_image
5[web] Failed to pull image "registry.example.com/web:pr-482-a1b2c3": 
6      rpc error: code = Unknown desc = failed to resolve reference: 
7      manifest unknown

This particular error, container_error_pull_image, is common enough to be worth recognizing on sight. It almost never means the application code is broken. It means the image tag the deploy step is trying to pull does not actually exist in the registry yet, which is usually a build-step ordering problem, not a runtime problem.

The Image-Pull Failure, Diagnosed

ImagePullBackOff and its close relatives account for a large share of deploy failures that get initially misdiagnosed as application bugs. The actual causes are almost always one of three things: the image tag genuinely does not exist yet (the build step that should have pushed it either failed silently or has not finished), the registry requires credentials the deploy step is not providing, or an anonymous pull hit a registry rate limit.

1$ curl -s -o /dev/null -w "%{http_code}" \
2  "https://registry.example.com/v2/web/manifests/pr-482-a1b2c3"
3404

A 404 here confirms the image tag genuinely does not exist yet at the registry, which means the fix is upstream in the pipeline (the build step needs to complete and push successfully before the deploy step runs), not anywhere in the application. Chasing this as a code bug wastes exactly the time this one curl command saves.

The Database-Not-Ready Failure, Diagnosed

A second common failure shape looks like an application crash on startup, but traces back to a dependency ordering problem: a component tries to connect to its database before the database component has actually finished initializing, even though both report as "running" from a pod-status perspective.

1$ bns logs --component <API_COMPONENT_ID> --tail 50
2
3Error: connect ECONNREFUSED postgres:5432
4    at TCPConnectWrap.afterConnect

The pod being in a running state and the database inside it being ready to accept connections are two different moments, and the gap between them can be several seconds during first initialization. The fix is an explicit dependsOn relationship plus a readiness check the dependent component's deploy script waits on, not just a dependsOn declaration alone, since dependsOn controls deploy ordering, not readiness:

1components:
2  - kind: Database
3    name: postgres
4
5  - kind: Application
6    name: api
7    dependsOn: [postgres]
8    deploy:
9      - 'until pg_isready -h postgres -p 5432; do sleep 1; done'
10      - 'npm run start'

The pg_isready loop is what actually closes the gap. dependsOn alone only guarantees the database component's deploy step starts first, not that it has finished becoming ready to accept connections by the time the API's own deploy step runs.

The Stale Password Failure, Diagnosed

This one is subtle enough to be worth naming explicitly, because it produces a genuinely confusing symptom: a password that was correct in the config, on a component that redeployed successfully, suddenly fails authentication. PostgreSQL (and most databases with a similar persistence model) only reads its password environment variable once, on first initialization of an empty data directory. Once the volume is populated, changing the variable has no further effect, since the actual credential lives in the volume's already-initialized state, not in the environment variable read at every restart.

Error: password authentication failed for user "app_user"

This surfaces most often after a team rotates a secret and expects every environment to pick it up on next redeploy, when in fact only environments created fresh after the rotation will actually use the new value; any environment with a pre-existing volume keeps whatever password was baked in at its own first initialization. The fix, when this is genuinely the cause, is destroying and recreating the affected environment (which discards the stale volume along with everything else), not repeatedly redeploying the same environment expecting the new password to take effect.

Reading Events for the Bigger Picture

When a failure is not obviously one of the above, bns events list gives a chronological view of everything that happened to the environment, which is useful for catching a failure that happened several steps before the one that actually surfaced as an error:

1$ bns events list --environment <ENV_ID> --sort createdAt:desc
2
3TYPE      STATUS       CREATED
4deploy    failed       2026-09-29T14:02:11Z
5deploy    in_progress  2026-09-29T14:01:40Z
6webhook   received     2026-09-29T14:01:38Z

This is particularly useful for catching a race: a webhook triggering a redeploy while a previous deploy for the same environment was still in progress, which can produce a failure that looks like it's about the current commit but is actually a collision with the deploy that preceded it.

Monitoring a Deploy Live Instead of Waiting for the Failure

For a deploy that is actively hanging rather than one that already failed, bns pipeline monitor gives a live view instead of requiring repeated polling:

1$ bns pipeline monitor --id OG7nYbkwM3 --interval 5s
2
3[14:02:05] web: pulling image...
4[14:02:10] web: pulling image...
5[14:02:15] web: pulling image... (still pulling after 10s)

A component visibly stuck on "pulling image" for well past the time a normal pull takes is a strong early signal of the image-pull failure described above, and catching it live saves waiting for the pipeline to time out and fail formally before starting to diagnose it.

Falling Back to kubectl for the Cases the CLI Doesn't Cover

For failures the platform's own commands don't surface cleanly (a pod stuck in CrashLoopBackOff with an exit code that needs interpreting, or a scheduling failure from insufficient node capacity), direct kubectl access against the environment's namespace is the fallback:

1$ kubectl get pods -n env-otmwdf
2
3NAME                    READY   STATUS             RESTARTS
4web-5d9498ddf4-m2rfl    0/1     ImagePullBackOff    0
5
6$ kubectl describe pod web-5d9498ddf4-m2rfl -n env-otmwdf

kubectl describe surfaces the underlying Kubernetes event stream for the pod, which includes scheduling failures, resource limit rejections, and pull errors with more raw detail than the platform's own summarized status, at the cost of needing direct cluster credentials. See SSH Into a Running Environment for the equivalent access pattern for a container that is running but behaving incorrectly, rather than failing to start at all.

Building This Into the Merge Queue's Own Triage

Once a team recognizes these as the handful of recurring failure shapes they actually are, it's worth wiring the most common ones into an automated first-pass triage step that runs before a human looks at the failure at all: check for container_error_pull_image specifically and surface "image not pushed yet, re-run once build completes" as the comment, rather than a generic "deploy failed" that sends a developer down a full manual investigation for a failure mode that has one well-known cause and one well-known fix. See Merge Queue Chaos From AI-Generated PRs for the broader context of why this kind of fast, cheap triage matters more as PR volume grows.

Resource and Scheduling Failures

A different failure class shows up not as an error in the pipeline logs at all, but as a pod that never starts: it sits in Pending indefinitely, with no crash, no error, just no progress. This is almost always a scheduling failure, and kubectl describe is the right tool for it specifically, since the platform's own pipeline logs generally have nothing to report yet, there is no container to log from.

1$ kubectl describe pod web-5d9498ddf4-m2rfl -n env-otmwdf
2
3Events:
4  Warning  FailedScheduling  Insufficient cpu, Insufficient memory

Insufficient cpu or Insufficient memory in the event list means the cluster's current node pool has no node with enough free capacity to satisfy the pod's resource requests, which is a cluster capacity problem, not a bug in the component's own configuration, unless the requested resources themselves are set unrealistically high for what the workload actually needs. A related but distinct message, OOMKilled, appears after a pod started successfully and then got killed for exceeding its memory limit during actual runtime, which points at the workload genuinely needing more memory than currently allocated, rather than a scheduling problem.

1components:
2  - kind: Application
3    name: web
4    k8s:
5      resources:
6        requests:
7          cpu: '250m'
8          memory: '256Mi'
9        limits:
10          memory: '512Mi'

Distinguishing these two, a request too high for the cluster to schedule at all versus a limit too low for the workload once it's running, matters because the fixes are opposite: the first is fixed by lowering the request or growing the node pool, the second by raising the limit, and applying the wrong fix to either just produces the same failure again in a slightly different shape.

When a Failure Really Is Just Flaky

Not every pipeline failure has a deterministic cause worth root-causing individually. A transient network blip during an image pull, a brief registry rate-limit window, a cluster autoscaler momentarily lagging behind a spike in concurrent deploys, these produce failures that a simple retry genuinely resolves, and treating every single failure as requiring full investigation wastes time on the subset that was never going to recur.

The practical distinction: a failure worth investigating in full recurs on retry, or recurs across multiple different PRs in the same shape. A failure that clears on a single retry and does not recur for days is reasonably treated as transient infrastructure noise, logged but not chased further. Wiring an automatic single retry into the deploy pipeline itself, specifically for the small set of failure signatures known to be commonly transient (a specific registry timeout error, for instance), catches this category before it ever reaches a human, while still surfacing anything that fails a second time for real investigation.

1deploy:
2  - |
3    for i in 1 2; do
4      docker pull "$IMAGE" && break
5      echo "Pull attempt $i failed, retrying..."
6      sleep 5
7    done

The key discipline is keeping this retry narrow and specific (a known-transient step, a small fixed retry count) rather than wrapping the entire deploy pipeline in a blanket retry, which would silently mask genuine, recurring failures behind a "it passed on retry" result that hides a real problem instead of surfacing it.

Turning Recurring Failures Into a Team Runbook

Once a team has diagnosed the same failure shape (image-pull errors, database-not-ready races, stale passwords, scheduling failures) more than once, the fastest path for the next occurrence is not re-diagnosing from scratch, it's checking a short, living runbook the team maintains of exactly the failure signatures already seen and their known fixes.

1## Known Deploy Failures
2
3**container_error_pull_image**
4→ Image tag not pushed yet. Check the build job finished before the deploy job ran.
5
6**ECONNREFUSED postgres:5432**
7→ dependsOn without a readiness wait. Check the component's deploy script for a pg_isready loop.
8
9**password authentication failed**
10→ Stale volume from before a secret rotation. Destroy and recreate the environment.
11
12**FailedScheduling: Insufficient cpu**
13→ Cluster capacity. Check node pool autoscaler ceiling against current concurrent environment count.

This is a small artifact, usually just a markdown file in the same repository as the bunnyshell.yaml template, but it compounds in value every time a new team member hits one of these failures for the first time and finds the answer in thirty seconds instead of rediscovering it the hard way, which is exactly the kind of tribal knowledge that otherwise lives only in one senior engineer's memory.

Related Reading

For the deploy pipeline this debugging process is diagnosing failures inside, see Automated PR Environments for E2E Testing. For debugging a component that deployed successfully but is misbehaving at runtime rather than failing to start, see SSH Into a Running Environment.

Checklist

• Check bns pipeline jobs --jobStatus failed before opening application logs
• Filter logs with --stepStatus failed to skip past successful step output
• Recognize container_error_pull_image as a build-step ordering problem, not an application bug
• Confirm dependsOn is paired with an actual readiness check, not assumed to guarantee readiness alone
• Rule out a stale password baked into a persistent volume before assuming a config value is wrong