Connecting Your Own Kubernetes Cluster (BYOC) to Bunnyshell

The Short Version

Bring-your-own-cluster (BYOC) means every environment deploys into a Kubernetes cluster you already own, already pay for, and already control, rather than a managed black box abstracted away behind an API. The actual connection process is a one-time step: create a scoped service account inside the cluster, generate a kubeconfig for it, and register that kubeconfig with a single CLI command. After that, every environment created against this cluster deploys the same way, regardless of whether the underlying infrastructure is EKS, GKE, AKS, or a self-managed cluster running on bare metal.

Why BYOC Instead of a Fully Managed Cluster

A fully managed cluster is simpler to get started with, since there's no existing infrastructure to connect, one gets provisioned automatically. BYOC trades a small amount of that initial simplicity for control that matters a great deal at scale: environments run on infrastructure already covered by existing cost commitments (reserved instances, committed-use discounts), already subject to existing network policies and compliance controls, and already monitored by whatever observability stack a team has already invested in. For a team that already runs production on a specific cloud provider with specific security requirements, connecting that same provider's Kubernetes offering, rather than trusting environments to an entirely separate managed cluster, is usually the more defensible choice from both a cost and a compliance standpoint.

Prerequisites Before Connecting

The cluster needs a small set of things in place before the connection step, none of which requires rebuilding an existing cluster from scratch:

An ingress controller. Environments need a way to expose components with public URLs, which requires an ingress controller already installed in the cluster. bns-nginx (Bunnyshell's own recommended nginx ingress class) is the default expected by templates using k8s.ingress.className: bns-nginx, though an existing ingress-nginx installation can generally be adapted with the corresponding class name change in the template.

Sufficient node capacity. The cluster's node pool needs headroom to run environment workloads on top of whatever else it's already hosting. This doesn't need to be dedicated capacity, but it does need honest accounting, since environments (especially several concurrent ephemeral ones) can consume meaningful CPU and memory during active development periods.

Network access from wherever deploy pipelines run. The connection relies on standard kubeconfig-based API access, so whatever network path deploy pipelines use to reach the cluster's API server needs to actually be open, which matters specifically for a cluster sitting behind a private network or VPN rather than a public-facing API endpoint.

Creating a Scoped Service Account

Rather than connecting with a cluster-admin credential, which works but grants far more access than environments actually need, the safer and recommended path creates a dedicated service account scoped to what environment management genuinely requires:

1kubectl create serviceaccount preview-deployer -n default
2kubectl create clusterrolebinding preview-deployer-binding \
3  --clusterrole=edit --serviceaccount=default:preview-deployer

The edit clusterrole grants the permissions needed to create, update, and delete namespaces, deployments, services, and ingress resources, the actual operations environment lifecycle management performs, without granting the broader administrative permissions (managing cluster-level RBAC, nodes, or other tenants' resources) a full cluster-admin binding would include. For an organization with stricter isolation requirements, this can be scoped further to a specific namespace prefix rather than cluster-wide, at the cost of some additional per-namespace RBAC setup.

Generating the Kubeconfig for the Service Account

The service account itself needs a kubeconfig file that authenticates as it, which is what actually gets registered with the connection command:

1SECRET_NAME=$(kubectl get serviceaccount preview-deployer -n default -o jsonpath='{.secrets[0].name}')
2TOKEN=$(kubectl get secret "$SECRET_NAME" -n default -o jsonpath='{.data.token}' | base64 -d)
3CA_CERT=$(kubectl get secret "$SECRET_NAME" -n default -o jsonpath='{.data.ca\.crt}')
4CLUSTER_ENDPOINT=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
5
6cat <<EOF > kubeconfig.yaml
7apiVersion: v1
8kind: Config
9clusters:
10  - name: target-cluster
11    cluster:
12      server: ${CLUSTER_ENDPOINT}
13      certificate-authority-data: ${CA_CERT}
14contexts:
15  - name: preview-deployer-context
16    context:
17      cluster: target-cluster
18      user: preview-deployer
19current-context: preview-deployer-context
20users:
21  - name: preview-deployer
22    user:
23      token: ${TOKEN}
24EOF

Newer Kubernetes versions (1.24+) don't auto-generate a long-lived secret for a service account the way older versions did, so on those clusters this step needs an explicit token request instead:

kubectl create token preview-deployer -n default --duration=87600h > token.txt

An 87600-hour duration is roughly ten years, chosen deliberately long since this token authenticates the ongoing connection between the platform and the cluster, not a short-lived session; rotating it is a deliberate, occasional operation, not something that should happen unexpectedly from a token expiring mid-use.

Registering the Connection

With kubeconfig.yaml generated, the actual connection is a single CLI command:

1$ bns k8s-clusters create --name my-cluster --kubeconfig ./kubeconfig.yaml
2
3Cluster connected successfully.
4ID: x0MvvlyMJa

This is a one-time step per cluster. Every environment created afterward against this cluster ID reuses the same connection, with no per-environment reconnection or re-authentication needed. Confirming the connection succeeded and checking the cluster's registered details afterward is worth doing immediately, rather than assuming success from the command's exit code alone:

1$ bns k8s-clusters list --organization <ORG_ID>
2
3ID           NAME          STATUS      CLOUD
4x0MvvlyMJa   my-cluster    connected   aws
5
6$ bns k8s-clusters show --id x0MvvlyMJa
7
8Name: my-cluster
9Status: connected
10Cloud: aws
11Kubernetes version: 1.29
12Nodes: 6

Creating an Environment Against the Connected Cluster

Once connected, the cluster ID is what any subsequent environment creation references, exactly the same way it would for a fully managed cluster, since from the deploy pipeline's perspective the two are identical after connection:

$ bns environments create --from-path bunnyshell.yaml \
    --name "pr-482" --project <PROJECT_ID> --k8s x0MvvlyMJa --deploy

This is the detail that makes BYOC genuinely low-friction in practice once the one-time connection is done: templates, deploy pipelines, and every day-to-day command work identically regardless of whether the underlying cluster is managed or bring-your-own. The only difference is the --k8s value pointing at a cluster ID the team's own infrastructure produced, rather than one automatically provisioned.

Multi-Tenancy: One Cluster, Multiple Teams or Projects

A single connected cluster can host environments from multiple projects simultaneously, isolated by standard Kubernetes namespace boundaries, which is a common and cost-effective pattern once an organization has several teams that don't individually need dedicated cluster capacity.

1$ bns k8s-clusters list --organization <ORG_ID>
2
3ID           NAME                       CLOUD
40DGBrJEGl1   bns-tools                  gcp
5OY13OP4Gnr   managed-demo               aws
6x0MvvlyMJa   stage-neo-k8s-1-htz        hetzner

Sizing shared cluster capacity for multiple teams' concurrent environment load follows the same peak-concurrency measurement approach described in Running Parallel E2E Suites at Scale, just aggregated across every team sharing the cluster rather than a single team's own CI activity.

Disconnecting or Rotating Cluster Access

Revoking access, whether to rotate a compromised credential or to fully disconnect a cluster that's being decommissioned, starts at the Kubernetes side, not the platform side, since the actual authority lives in the service account and its token:

kubectl delete clusterrolebinding preview-deployer-binding
kubectl delete serviceaccount preview-deployer -n default

Deleting the service account immediately invalidates any token issued for it, which is the fastest way to cut off access in an emergency, faster than waiting on any platform-side disconnection step, since the underlying credential simply stops being valid the moment the service account backing it is gone.

Verifying the Connection End to End Before Relying on It

A connection that reports success is worth confirming with an actual deploy, not just trusting the confirmation message, since a subtle RBAC or ingress misconfiguration sometimes only surfaces once a real component tries to schedule and expose itself:

1$ bns environments create --from-path bunnyshell.yaml \
2    --name "connection-test" --project <PROJECT_ID> --k8s x0MvvlyMJa --deploy
3
4$ bns environments show --id <TEST_ENV_ID>
5
6Status: running
7Components: 1/1 healthy
8
9$ curl -s -o /dev/null -w "%{http_code}\n" https://web-abc123.example-domain.com/
10200

A test environment that deploys, reports healthy, and actually responds on its generated URL confirms all three layers at once: the service account has enough permission to schedule workloads, the ingress controller is correctly installed and reachable, and DNS for the base domain resolves to it. Tearing this test environment down immediately afterward keeps it from lingering as an unexplained entry in the environment list.

Clusters Behind a Private Network

A cluster whose API server isn't reachable from the public internet, common for organizations running Kubernetes inside a VPC with no public endpoint, needs a network path from wherever deploy pipelines actually run to that private API server, which the connection step alone doesn't create. The two common solutions are running deploy pipelines on infrastructure already inside the same private network (a self-hosted CI runner in the same VPC) or exposing the API server through a bastion host or a VPN the CI environment can reach. Neither is configured by the bns k8s-clusters create command itself; both need to be solved at the network layer first, with the connection step only succeeding once that path already exists and the kubeconfig's server address is actually reachable from wherever the command runs.

Capacity Planning for a Shared BYOC Cluster

A connected cluster that also runs other workloads, an existing production service, another team's environments, needs capacity planning that accounts for all of it together, not just the environment workloads in isolation. Tracking actual node utilization across everything the cluster hosts, not just environment-attributed usage, avoids the specific failure mode of an environment failing to schedule because a completely unrelated workload on the same cluster happened to be under unusually heavy load that day.

1$ kubectl top nodes
2
3NAME       CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
4node-1     2100m        70%    5800Mi          72%
5node-2     1950m        65%    6100Mi          76%

Reserving explicit headroom, rather than assuming whatever's left over after other workloads is automatically enough, is the practical way to keep a shared BYOC cluster from becoming a source of intermittent, hard-to-diagnose scheduling failures for environments specifically.

Related Reading

For the fuller context of setting up preview environments end to end on a Kubernetes cluster, including the ingress and DNS pieces this connection step assumes are in place, see How to Set Up Preview Environments on Kubernetes. For wiring the connected cluster's environments to actually build from your repositories, see Connecting Your Git Provider.

BYOC and Compliance Requirements

For organizations operating under specific data residency or compliance requirements (keeping all workloads within a specific cloud region, a specific provider, or under a specific set of network controls), BYOC is often the deciding factor in whether a managed environment platform is usable at all. Since the cluster itself never leaves the organization's own cloud account, every existing control already applied to that account (network policies, audit logging, IAM boundaries) applies to environment workloads exactly the same way it applies to any other workload in the same cluster, with no separate compliance review needed for a completely separate, externally-managed piece of infrastructure. This is frequently the specific reason a team chooses BYOC over a managed cluster in the first place, well before cost or existing-commitment considerations enter the discussion at all.

Checklist

• An ingress controller is installed and reachable before attempting to connect the cluster
• The connecting service account is scoped to edit, not cluster-admin, unless a specific reason requires broader access
• The generated kubeconfig authenticates as the scoped service account, not a personal or admin kubeconfig
• The connection is confirmed via bns k8s-clusters show immediately after connecting, not assumed from a successful exit code
• A documented process exists for rotating or revoking the service account's access if the credential is ever compromised