cloud-serverless

# Redpanda Cloud API: Serverless Clusters Redpanda Cloud Serverless clusters are fully managed Kafka-compatible clusters provisioned through the public Control Plane API at `https://api.redpanda.com`. You authenticate once with OAuth2 client credentials, then create a `ResourceGroup`, pick a `ServerlessRegion`, and call `POST /v1/serverless/clusters`. The API returns an async `Operation`; when the Operation reaches `STATE_COMPLETED`, the cluster has reached `STATE_READY` and its `dataplane_api.url` is live for Kafka workloads and the Data Plane REST API. (`STATE_COMPLETED` is the Operation state; `STATE_READY` is the cluster state — they are distinct state machines.) This skill does **not** cover BYOC (Bring Your Own Cloud) clusters — see the `cloud-byoc` skill for those. For the CLI equivalent, see the `rpk-cloud` skill. ## Quickstart Copy-paste these commands. Replace `CLIENT_ID` and `CLIENT_SECRET` with the credentials from your Redpanda Cloud service account. ```bash # ── 1. Get a bearer token ───────────────────────────────────────────────────── TOKEN=$(curl -s -X POST https://auth.prd.cloud.redpanda.com/oauth/token \ -H "Content-Type: application/x-www-form-urlencoded" \ -d "grant_type=client_credentials" \ -d "client_id=${CLIENT_ID}" \ -d "client_secret=${CLIENT_SECRET}" \ -d "audience=cloudv2-production.redpanda.cloud" \ | jq -r .access_token) # ── 2. List available serverless regions ────────────────────────────────────── curl -s -X GET "https://api.redpanda.com/v1/serverless/regions?cloud_provider=CLOUD_PROVIDER_AWS" \ -H "Authorization: Bearer ${TOKEN}" | jq . # ── 3. Create a resource group (billing/org container) ─────────────────────── RG_ID=$(curl -s -X POST https://api.redpanda.com/v1/resource-groups \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{"resource_group": {"name": "my-dev-group"}}' \ | jq -r .resource_group.id) # ── 4. Create a serverless cluster ──────────────────────────────────────────── OP_ID=$(curl -s -X POST https://api.redpanda.com/v1/serverless/clusters \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d "{ \"serverless_cluster\": { \"name\": \"my-cluster\", \"resource_group_id\": \"${RG_ID}\", \"serverless_region\": \"us-east-1\" } }" | jq -r .operation.id) # ── 5. Poll the operation until STATE_COMPLETED ─────────────────────────────── watch -n 5 "curl -s https://api.redpanda.com/v1/operations/${OP_ID} \ -H 'Authorization: Bearer ${TOKEN}' | jq '{state: .operation.state}'" # ── 6. Get the cluster and its data-plane URL ───────────────────────────────── # Read the cluster ID directly from the completed operation (resource_id or # metadata.serverless_cluster_id) — more reliable than listing by name. OP_RESP=$(curl -s "https://api.redpanda.com/v1/operations/${OP_ID}" \ -H "Authorization: Bearer ${TOKEN}") CLUSTER_ID=$(echo "${OP_RESP}" | jq -r '.operation.resource_id') CLUSTER=$(curl -s "https://api.redpanda.com/v1/serverless/clusters/${CLUSTER_ID}" \ -H "Authorization: Bearer ${TOKEN}" | jq '.serverless_cluster') echo "${CLUSTER}" | jq '{id: .id, state: .state, dataplane_url: .dataplane_api.url}' DP_URL=$(echo "${CLUSTER}" | jq -r '.dataplane_api.url') # ── 7. Create a topic via the data-plane URL ────────────────────────────────── curl -s -X POST "${DP_URL}/v1/topics" \ -H "Authorization: Bearer ${TOKEN}" \ -H "Content-Type: application/json" \ -d '{"topic": {"name": "events", "partition_count": 3, "replication_factor": 3}}' ``` ## Authentication The Control Plane API and Data Plane API both use the same OAuth2 bearer token. See [Authentication Reference](references/auth.md) for full details. | Field | Value | |---|---| | Token endpoint | `https://auth.prd.cloud.redpanda.com/oauth/token` | | Audience | `cloudv2-production.redpanda.cloud` | | Grant type | `client_credentials` | | Control Plane base URL | `https://api.redpanda.com` | Credentials come from a **ServiceAccount** created in the Redpanda Cloud console or via `POST /v1/service-accounts`. The response includes a `client_id` and `client_secret` (the secret is only shown once on creation; save it securely). `GET /v1/service-accounts/{id}/credentials` returns the client ID only, so a lost secret must be rotated with `GET /v1/service-accounts/{id}/rotate-secret` — it cannot be read back. ## Control Plane Resources ### ResourceGroup A ResourceGroup is the billing and organizational container for clusters. It must exist before you can create a ServerlessCluster. ``` POST /v1/resource-groups GET /v1/resource-groups/{id} GET /v1/resource-groups # list with optional filter.name_contains PATCH /v1/resource-groups/{resource_group.id} DELETE /v1/resource-groups/{id} ``` Name constraints: 3–253 alphanumeric characters and hyphens (`^[a-zA-Z0-9-]+$`). ### ServerlessRegion Regions represent the cloud-provider geographic areas where Serverless clusters can be placed. List them before creating a cluster to confirm `placement.enabled`. ``` GET /v1/serverless/regions?cloud_provider=CLOUD_PROVIDER_AWS GET /v1/serverless/regions?cloud_provider=CLOUD_PROVIDER_GCP GET /v1/serverless/region?cloud_provider=CLOUD_PROVIDER_AWS&name=us-east-1 ``` `cloud_provider` accepts `CLOUD_PROVIDER_AWS`, `CLOUD_PROVIDER_GCP`, or `CLOUD_PROVIDER_AZURE`. Use `filter.placement_enabled_only=true` to skip regions that cannot currently accept new clusters. ### ServerlessCluster Lifecycle Full CRUD plus the Operation pattern: ``` POST /v1/serverless/clusters → 202 CreateServerlessClusterOperation GET /v1/serverless/clusters/{id} → ServerlessCluster GET /v1/serverless/clusters → ListServerlessClustersResponse (paginated) PATCH /v1/serverless/clusters/{id} → 202 UpdateServerlessClusterOperation DELETE /v1/serverless/clusters/{id} → 202 DeleteServerlessClusterOperation GET /v1/serverless/clusters/{id}/prometheus/credentials ``` **State machine** (from the proto): | State | Meaning | |---|---| | `STATE_PLACING` | Finding a cell with sufficient resources | | `STATE_CREATING` | Creating control-plane state | | `STATE_READY` | Running and accepting requests | | `STATE_SUSPENDED` | Running but blocking external requests | | `STATE_DELETING` | Removal in progress | | `STATE_FAILED` | Could not reach READY from PLACING or CREATING | **Create request fields**: | Field | Required | Notes | |---|---|---| | `name` | yes | 3–128 chars, `^[A-Za-z0-9-_:]+$` | | `resource_group_id` | yes | UUID of an existing ResourceGroup | | `serverless_region` | yes | Region name string, e.g. `"us-east-1"` | | `tags` | no | `map<string,string>`, max 50 pairs | | `networking_config` | no | Optional `public` / `private`, each `STATE_ENABLED` or `STATE_DISABLED`; omitted resolves as public enabled, private disabled | **Response fields on GET** (output-only): | Field | Notes | |---|---| | `id` | 20-char opaque ID | | `state` | One of the states above | | `kafka_api.seed_brokers[]` | Public bootstrap servers | | `kafka_api.private_seed_brokers[]` | Private bootstrap (when private networking enabled) | | `dataplane_api.url` | Base URL for the per-cluster Data Plane REST API | | `schema_registry.url` | Schema Registry public endpoint | | `console_url` | Redpanda Console web UI URL | | `prometheus.url` | Prometheus scrape endpoint | ### Operations Create, Update, and Delete each return an `Operation` whose `type` is `TYPE_CREATE_SERVERLESS_CLUSTER`, `TYPE_UPDATE_SERVERLESS_CLUSTER`, or `TYPE_DELETE_SERVERLESS_CLUSTER` respectively, so you can filter operations by any of the three. Poll at `GET /v1/operations/{id}`. ``` GET /v1/operations/{id} GET /v1/operations # list with filter.type_in / filter.state / filter.resource_id ``` Operation states: `STATE_IN_PROGRESS`, `STATE_COMPLETED`, `STATE_FAILED`. When `state == STATE_COMPLETED`, the `response` field contains the final resource. When `state == STATE_FAILED`, the `error` field contains a `google.rpc.Status`. ### ServerlessPrivateLink (AWS PrivateLink) The `private_link_id` field on a cluster refers to a **ServerlessPrivateLink** resource, managed by its own control-plane service (**AWS only**). Create the private link first, then pass its 20-char `id` as `private_link_id` when you create a cluster with private networking enabled. Serverless on AWS went GA in Feb 2026 with PrivateLink support. ``` POST /v1/serverless/private-links → 202 Operation (TYPE_CREATE_SERVERLESS_PRIVATE_LINK) GET /v1/serverless/private-links/{id} GET /v1/serverless/private-links # list (paginated, filterable) PATCH /v1/serverless/private-links/{id} → 202 Operation DELETE /v1/serverless/private-links/{id} → 202 Operation ``` Required create fields: `name`, `resource_group_id`, `cloudprovider` (`CLOUD_PROVIDER_AWS` only, set together with `aws_config`), `aws_config.allowed_principals` (min 1 AWS principal ARN), and `serverless_region`. See [Control Plane: Serverless](references/control-plane-serverless.md#serverlessprivatelink) for the full field-level reference and the private-networking workflow. ## Data Plane API Once your cluster is `STATE_READY`, use `dataplane_api.url` as the base URL. The same bearer token is valid. Base path is `/v1`. Always read the URL from the API response — do not construct it manually. The real DNS pattern (from the Control Plane API reference examples) is `https://<cluster-id>.any.<region>.mpx.prd.cloud.redpanda.com`. Available services: | Service | Endpoint prefix | |---|---| | Topic | `/v1/topics` | | ACL | `/v1/acls` | | User | `/v1/users` | | Secret | `/v1/secrets` | | Pipeline (Redpanda Connect) | `/v1/redpanda-connect/pipelines` | | Security (RBAC roles) | `/v1/roles` | | Quota | `/v1/quotas` | | KafkaConnect | `/v1/kafka-connect/clusters/{cluster_name}/connectors` | See [Data Plane Reference](references/data-plane.md) for examples and availability notes on Serverless clusters. ## Enterprise Features on Serverless Redpanda Cloud is a managed deployment of **Redpanda Enterprise Edition** — the license is included, so there is no `rpk cluster license` step for tenants. The enterprise differentiators you configure on Serverless are exposed as **topic configuration keys** (via `${DP_URL}/v1/topics`) and **RBAC roles** (via `${DP_URL}/v1/roles`), not cluster/node config: | Feature (Enterprise) | How to set on Serverless | Key(s) | |---|---|---| | Iceberg Topics | Topic config | `redpanda.iceberg.mode` (`disabled`/`key_value`/`value_schema_id_prefix`/`value_schema_latest`), `redpanda.iceberg.target.lag.ms`, `redpanda.iceberg.partition.spec`, `redpanda.iceberg.invalid.record.action` (`drop`/`dlq_table`), `redpanda.iceberg.delete` | | Server-Side Schema ID Validation | Topic config | `redpanda.key.schema.id.validation`, `redpanda.value.schema.id.validation`, `redpanda.key.subject.name.strategy`, `redpanda.value.subject.name.strategy` (`TopicNameStrategy`/`RecordNameStrategy`/`TopicRecordNameStrategy`) | | Leadership Pinning | Topic config | `redpanda.leaders.preference` (inherits cluster `default_leaders_preference`) | | RBAC | Data Plane SecurityService | `${DP_URL}/v1/roles` | Cluster/node-level enterprise features (Tiered Storage internals, FIPS, Continuous Data Balancing thresholds, Audit Logging, Remote Read Replicas, Cloud Topics, Shadow Linking / cross-cluster DR, Whole Cluster Restore) are **managed by Redpanda** on Serverless and are not tenant-configurable — use a BYOC/Dedicated or self-managed cluster for those. See [Enterprise Features Reference](references/enterprise-features.md) for the full key tables, accepted values, defaults, mode semantics, and self-managed mappings. ## Serverless vs BYOC | | Serverless | BYOC | |---|---|---| | Infra ownership | Redpanda manages everything | Customer's cloud account | | Network resources | Not required | Requires creating a `Network` resource | | Cluster type | Serverless (separate `ServerlessCluster` resource; no `Cluster.Type`) | BYOC (`Cluster.Type` = `TYPE_BYOC`) | | Provisioning agent | None | `rpk cloud byoc apply` runs Terraform | | Billing | Redpanda bills per-use | Customer pays cloud provider directly | ## Reference Directory - [Authentication](references/auth.md): OAuth2 client-credentials flow, token endpoint, audience, Authorization header, service-account creation, and token refresh. - [Control Plane: Serverless](references/control-plane-serverless.md): ResourceGroup, ServerlessRegion, and ServerlessCluster field-level reference, the async Operation pattern, state machine, and list filtering.