cloud-byoc

# Redpanda Cloud API: BYOC Clusters BYOC (Bring Your Own Cloud) lets you run Redpanda in your own AWS, GCP, or Azure account: Redpanda manages the control plane and agent lifecycle, while your VPC, IAM roles, and storage buckets stay under your cloud account. You pay cloud infrastructure costs directly. This is the primary alternative to Serverless, which runs entirely in Redpanda's account. The provisioning workflow has three phases: (1) create a **Network** resource that registers your VPC/VNet with Redpanda, (2) create a **Cluster** resource of `TYPE_BYOC` referencing that network, and (3) run `rpk cloud byoc apply --redpanda-id <id>` to execute the Terraform that installs the agent in your cloud account. Both the Network and Cluster create calls return an `Operation` object — poll `GET /v1/operations/{id}` until `state` is `STATE_COMPLETED` before proceeding. ## Quickstart ```bash # 1. Get an OAuth2 bearer token (client credentials flow) TOKEN=$(curl -s -X POST "https://auth.prd.cloud.redpanda.com/oauth/token" \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=${CLIENT_ID}" \ -d "client_secret=${CLIENT_SECRET}" \ -d "audience=cloudv2-production.redpanda.cloud" \ | jq -r '.access_token') BASE="https://api.redpanda.com" # 2. Create (or identify) a Resource Group curl -s -X POST "${BASE}/v1/resource-groups" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{"resource_group": {"name": "my-byoc-rg"}}' | jq . RG_ID="<resource_group_id from response>" # 3. Create a Network (AWS example — Redpanda-managed VPC, CIDR-based) NET_OP=$(curl -s -X POST "${BASE}/v1/networks" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d "{ \"network\": { \"name\": \"my-byoc-network\", \"resource_group_id\": \"${RG_ID}\", \"cloud_provider\": \"CLOUD_PROVIDER_AWS\", \"region\": \"us-east-1\", \"cidr_block\": \"10.0.0.0/20\", \"cluster_type\": \"TYPE_BYOC\" } }" | jq .) NET_OP_ID=$(echo "${NET_OP}" | jq -r '.operation.id') # 4. Poll the network operation until STATE_COMPLETED until [ "$(curl -s "${BASE}/v1/operations/${NET_OP_ID}" \ -H "Authorization: Bearer ${TOKEN}" | jq -r '.operation.state')" = "STATE_COMPLETED" ]; do echo "Waiting for network…"; sleep 10 done NET_ID=$(curl -s "${BASE}/v1/networks?filter.name_contains=my-byoc-network" \ -H "Authorization: Bearer ${TOKEN}" | jq -r '.networks[0].id') # 5. Create the BYOC cluster CLUSTER_OP=$(curl -s -X POST "${BASE}/v1/clusters" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d "{ \"cluster\": { \"name\": \"my-byoc-cluster\", \"resource_group_id\": \"${RG_ID}\", \"type\": \"TYPE_BYOC\", \"cloud_provider\": \"CLOUD_PROVIDER_AWS\", \"region\": \"us-east-1\", \"zones\": [\"use1-az1\", \"use1-az2\", \"use1-az3\"], \"throughput_tier\": \"tier-1-aws-v2-arm\", \"network_id\": \"${NET_ID}\", \"connection_type\": \"CONNECTION_TYPE_PUBLIC\" } }" | jq .) # connection_type is the legacy single-topology selector and is deprecated. For a cluster that # serves a public AND a private listener per service, send per-service "connections" instead — # see "Cluster Connectivity" below. # Note: tier names are version-dependent. Authoritative list: # GET /v1/regions/CLOUD_PROVIDER_AWS (or CLOUD_PROVIDER_GCP/CLOUD_PROVIDER_AZURE; cloud_provider is a path segment) # or see https://docs.redpanda.com/redpanda-cloud/reference/tiers/byoc-tiers/ # Real AWS examples: tier-1-aws-v2-arm, tier-1-aws-v2-x86, tier-1-aws-v3-arm # Real GCP examples: tier-1-gcp-v2-x86, tier-1-gcp-um4g CLUSTER_OP_ID=$(echo "${CLUSTER_OP}" | jq -r '.operation.id') # Poll until cluster_id is available in operation metadata (may not be populated on first poll) until CLUSTER_ID=$(curl -s "${BASE}/v1/operations/${CLUSTER_OP_ID}" \ -H "Authorization: Bearer ${TOKEN}" \ | jq -r '.operation.metadata.cluster_id // .operation.resource_id // empty'); \ [ -n "${CLUSTER_ID}" ]; do echo "Waiting for cluster ID…"; sleep 5 done # 6. Poll until CREATING_AGENT state — the control plane is waiting for the agent until [ "$(curl -s "${BASE}/v1/clusters/${CLUSTER_ID}" \ -H "Authorization: Bearer ${TOKEN}" | jq -r '.cluster.state')" = "STATE_CREATING_AGENT" ]; do echo "Waiting for cluster to reach CREATING_AGENT…"; sleep 15 done # 7. Run the rpk byoc agent (install + apply) rpk cloud byoc install --redpanda-id "${CLUSTER_ID}" \ -X cloud.client_id="${CLIENT_ID}" \ -X cloud.client_secret="${CLIENT_SECRET}" # AWS provider — runs Terraform in your account rpk cloud byoc aws apply --redpanda-id "${CLUSTER_ID}" \ -X cloud.client_id="${CLIENT_ID}" \ -X cloud.client_secret="${CLIENT_SECRET}" # 8. Poll until STATE_READY until [ "$(curl -s "${BASE}/v1/clusters/${CLUSTER_ID}" \ -H "Authorization: Bearer ${TOKEN}" | jq -r '.cluster.state')" = "STATE_READY" ]; do echo "Waiting for cluster to be ready…"; sleep 30 done echo "Cluster is READY" ``` ## BYOC vs Serverless | Dimension | BYOC | Serverless | |---|---|---| | Infra ownership | Customer VPC / account | Redpanda's account | | Cloud bill | Customer pays AWS/GCP/Azure directly | Redpanda charges per usage | | Network isolation | Fully isolated VPC | Shared multi-tenant | | Agent required | Yes — `rpk cloud byoc apply` | No | | Cluster type | `TYPE_BYOC` | N/A (separate `ServerlessCluster` resource) | | Network resource | Required | Not required | | Customer-managed IAM | Optional (`customer_managed_resources`) | Not supported | | Throughput tiers | Dedicated tiers | Serverless tiers | ## Control Plane API Overview **Base URL:** `https://api.redpanda.com` **Auth:** Bearer token obtained via OAuth2 client credentials. Set `Authorization: Bearer <token>` on every request. **Protocol:** ConnectRPC (also accepts standard HTTP/JSON via the REST gateway). All mutating operations (CreateNetwork, CreateCluster, DeleteCluster, DeleteNetwork) return an `Operation` object with a 20-character ID. Poll `GET /v1/operations/{id}` to track progress. | Resource | Endpoints | |---|---| | Networks | `POST /v1/networks`, `GET /v1/networks/{id}`, `GET /v1/networks`, `PATCH /v1/networks/{id}?update_mask=...`, `DELETE /v1/networks/{id}` | | Clusters | `POST /v1/clusters`, `GET /v1/clusters/{id}`, `GET /v1/clusters`, `PATCH /v1/clusters/{id}?update_mask=...`, `DELETE /v1/clusters/{id}` | | Network Peerings | `POST /v1/network/{network_id}/network-peerings`, `GET`/`DELETE /v1/network/{network_id}/network-peerings/{id}`, `GET /v1/network/{network_id}/network-peerings` | | Shadow Links | `POST /v1/shadow-links`, `GET`/`DELETE /v1/shadow-links/{id}`, `GET /v1/shadow-links`, `PATCH /v1/shadow-links/{id}?update_mask=...` | | Operations | `GET /v1/operations/{id}`, `GET /v1/operations` | | Resource Groups | `POST /v1/resource-groups`, `GET /v1/resource-groups/{id}`, `GET /v1/resource-groups` | | Regions | `GET /v1/regions/{cloud_provider}`, `GET /v1/regions/{cloud_provider}/{name}` | ## Cluster Connectivity: `connection_type` vs `connections` `connection_type` (`CONNECTION_TYPE_PUBLIC` / `CONNECTION_TYPE_PRIVATE`) makes the whole cluster public or private and is **deprecated**. It is superseded by a per-service `connections` list on `kafka_api`, `http_proxy`, and `schema_registry` — **dual listener mode**, which serves a public and a private listener at the same time, each with its own server-assigned endpoint and its own SASL/SCRAM or mTLS auth: ```json "kafka_api": {"connections": [ {"type": "CONNECTION_TYPE_PUBLIC", "auth": {"mode": "AUTH_MODE_SASL"}}, {"type": "CONNECTION_TYPE_PRIVATE", "auth": {"mode": "AUTH_MODE_SASL"}} ]} ``` Decision rules: - Want one connectivity mode for the whole cluster and nothing else? `connection_type` still works. - Need in-VPC clients on a private listener while external clients stay public? Use `connections` — set them on **all three** services with the **same** topology, and omit `connection_type` and the per-service `sasl` block (the API rejects either alongside `connections`). - Dual listener mode is **beta, AWS only, and enabled per organization**; Azure is rejected outright, and the fields are not in the published API reference yet. It cannot be configured in the Cloud UI. - Read endpoints from `connections[].endpoint` in `GET /v1/clusters/{id}`, not from the deprecated `seed_brokers`/`url` fields. - Migrating public-only ↔ dual is self-service (with the `controlplane_cluster_migrate_connectivity` permission) — **except on BYOVPC clusters**, which always keep a private connection: every BYOVPC migration to or from dual listener mode goes through Redpanda Support. Anything that moves any cluster between private-only and publicly reachable also goes through Support. - Once a service uses `connections`, its legacy `sasl`/`mtls` update path is closed — the API rejects such a PATCH and tells you to use `connections`. Change auth via `connections[].auth.mode`. Full rules, examples, and migration semantics: [Clusters and Agent](references/clusters-and-agent.md#dual-listener-mode-public--private-listeners-per-service-beta-aws). ## Cluster State Machine BYOC clusters move through these states: ``` STATE_CREATING_AGENT → STATE_CREATING → STATE_READY ↗ STATE_UPGRADING ────────────────────── STATE_DELETING_AGENT → STATE_DELETING → (deleted) STATE_FAILED STATE_SUSPENDED ``` The cluster enters `STATE_CREATING_AGENT` after the API accepts the create request. This is when you must run `rpk cloud byoc apply` to install the agent Terraform. Once the agent completes provisioning, the cluster transitions to `STATE_CREATING`, then `STATE_READY`. ## rpk cloud byoc Commands ```bash # Install the byoc plugin (pinned to the cluster's required version) rpk cloud byoc install --redpanda-id <cluster-id> # Apply (provision) agent infra — cloud-provider subcommand is required. # GCP also requires --project-id; Azure also requires --subscription-id. rpk cloud byoc aws apply --redpanda-id <cluster-id> rpk cloud byoc gcp apply --redpanda-id <cluster-id> --project-id <gcp-project-id> rpk cloud byoc azure apply --redpanda-id <cluster-id> --subscription-id <azure-sub-id> # Destroy agent infra (same per-provider account flags as apply) rpk cloud byoc aws destroy --redpanda-id <cluster-id> rpk cloud byoc gcp destroy --redpanda-id <cluster-id> --project-id <gcp-project-id> rpk cloud byoc azure destroy --redpanda-id <cluster-id> --subscription-id <azure-sub-id> # Validate prerequisites without a cluster ID (uses latest plugin version) rpk cloud byoc aws validate rpk cloud byoc gcp validate # Note: only aws/gcp validate are confirmed; azure validate is not separately attested. # Uninstall the local plugin binary rpk cloud byoc uninstall ``` The `--redpanda-id` flag is required for `apply` and `destroy`. The plugin is automatically pinned to the version the control plane specifies for the given cluster ID. Set `RPK_CLOUD_SKIP_VERSION_CHECK=1` to use the currently installed plugin binary as-is (dev/CI use only). Authentication for `rpk cloud byoc` uses the same client credentials as `rpk cloud login`: ```bash # Via -X flags (not persisted) rpk cloud byoc aws apply --redpanda-id <id> \ -X cloud.client_id=<id> \ -X cloud.client_secret=<secret> # Via environment variables export RPK_CLOUD_CLIENT_ID=<id> export RPK_CLOUD_CLIENT_SECRET=<secret> rpk cloud byoc aws apply --redpanda-id <id> ``` ## Enterprise Features on BYOC