setup-k8s-backend

name

setup-k8s-backend

description

Create the Observe backend resources required for Kubernetes Explorer: datastreams, content packages, and an ingest token. Use this skill whenever the user needs to provision Observe backend objects for K8s monitoring — whether they're starting a fresh setup, recreating resources after manual removal, adding a missing datastream, or generating an ingest token for the helm chart — and even when they don't explicitly say "datastream" or "backend." Also invoked by deploy-k8s-explorer during full setup.

Set Up Kubernetes Explorer Backend

Focused workflow for creating the Observe backend resources needed to run Kubernetes Explorer: datastreams, content packages, and an ingest token. This skill does not touch the Kubernetes cluster or the helm chart.

Reference Files

Read this before executing to understand the datastream-to-dataset mapping:

• datastream-reference.md — Datastream-to-dataset mapping (useful for understanding CLI output)

---

Prerequisites to Gather

Before starting, confirm you have the following. If called from deploy-k8s-explorer, these will already be known — do not re-ask.

1. Customer URL — the full Observe tenant URL (e.g., https://105611059680.observeinc.com)
2. Cluster name — human-readable identifier used for the ingest token name
3. Data types — whether traces are included (determines which datastreams to create)

---

Phase 1: Authenticate via CLI

Auth state is stored in ~/.observe/config.json.

Step 1a: Check Existing Auth

observe auth status

If auth is already configured and valid, skip to Phase 2. If not authenticated or the token is expired, proceed to Step 1b.

Step 1b: Log In

observe auth login --url <CUSTOMER_URL>

This opens a browser for the user to authenticate. For headless environments (no browser), use:

observe auth login --url <CUSTOMER_URL> --use-device-code

After login completes, verify with:

observe auth status

If auth status reports success, proceed. If it fails, inform the user their login did not complete and ask them to retry.

---

Phase 2: Detect Existing State

Step 2a: Query existing datastreams

observe datastream list

Parse the JSON output and check for:

1. Legacy monolithic "Kubernetes Explorer" datastream (name without slash) — if found, warn the user: "A legacy Kubernetes Explorer datastream was found. The current setup uses multiple datastreams. Would you like to migrate to the new multi-stream setup, or keep the existing one?" Ask user to choose.
2. New-style datastreams (names with / like Kubernetes Explorer/Prometheus) — note which already exist.
3. All datastreams present — inform user everything is already set up. Offer to regenerate helm instructions only.
4. Some datastreams present — show what exists and what's missing. Ask: "Some datastreams already exist. Should I create the missing ones, or start fresh?"

You can also filter by name to narrow results:

observe datastream list --name "Kubernetes Explorer"
observe datastream list --name "Tracing"
observe datastream list --name "Metrics/OpenTelemetry"
observe datastream list --name "Observe Agent"

Step 2b: Query existing content

observe content kubernetes view

And if traces are relevant:

observe content tracing view

Report what is already installed. If content commands return empty/null results, nothing is installed yet.

---

Phase 3: Create Backend Resources

Step 3a: Show Summary

Present a summary of what will be created:

Backend resources to create:
  Datastreams:
    - Kubernetes Explorer/OpenTelemetry Logs (directWrite: otelLogs)
    - Kubernetes Explorer/Prometheus (directWrite: prometheus)
    - Kubernetes Explorer/Kubernetes Entity (directWrite: k8sEntity)
    - Observe Agent/Events (directWrite: k8sEntity)
    [If traces selected:]
    - Tracing/Span (directWrite: otelTrace)
    - Metrics/OpenTelemetry (directWrite: otelMetrics)

  Content:
    - Kubernetes Explorer Content (links datasets to explorer)
    [If traces selected:]
    - Tracing Content (links span/metric datasets to service explorer)

  Token:
    - Ingest token: "K8s Explorer - <CLUSTER_NAME>"

Proceed? (y/n)

Step 3b: Create Datastreams

Create each datastream sequentially using the CLI. For each, parse the JSON response and extract the dataset IDs from the directWrite fields (see datastream-reference.md for the mapping).

Core K8s datastreams (always created):

observe datastream create --name "Kubernetes Explorer/OpenTelemetry Logs" --direct-write-otel-logs

Capture: directWrite.otelLogs.datasetId → used as otel-logs-dataset-id for content install.

observe datastream create --name "Kubernetes Explorer/Prometheus" --direct-write-prometheus

Capture: directWrite.prometheus.datasetId → used as prometheus-dataset-id for content install.

observe datastream create --name "Kubernetes Explorer/Kubernetes Entity" --direct-write-k8s-entity

Capture: directWrite.k8sEntity.datasetId → used as entity-dataset-id for content install.

observe datastream create --name "Observe Agent/Events" --direct-write-k8s-entity

Tracing datastreams (if traces selected):

observe datastream create --name "Tracing/Span" --direct-write-otel-trace

Capture: directWrite.otelTrace.spanDatasetId, spanEventDatasetId, spanLinkDatasetId → used for tracing content install.

observe datastream create --name "Metrics/OpenTelemetry" --direct-write-otel-metrics

Capture: directWrite.otelMetrics.datasetId → used as otel-metrics-dataset-id for tracing content install.

If any creation fails: stop immediately, show the error, and ask the user how to proceed.

Step 3c: Install Content

Install Kubernetes Explorer content using the dataset IDs captured from datastream creation:

observe content kubernetes install \
  --otel-logs-dataset-id <OTEL_LOGS_DATASET_ID> \
  --prometheus-dataset-id <PROMETHEUS_DATASET_ID> \
  --entity-dataset-id <K8S_ENTITY_DATASET_ID>

If traces were selected, also install tracing content. The tracing install can auto-discover datasets if you omit the flags, but for explicit control pass the IDs:

observe content tracing install \
  --span-raw-dataset-id <SPAN_DATASET_ID> \
  --span-event-dataset-id <SPAN_EVENT_DATASET_ID> \
  --span-link-dataset-id <SPAN_LINK_DATASET_ID> \
  --otel-metrics-dataset-id <OTEL_METRICS_DATASET_ID>

Step 3d: Create Ingest Token

Create an ingest token with no datastream associations. Routing to the correct datastreams happens automatically at ingest time via target-package prefix matching (datastream names like Kubernetes Explorer/..., Observe Agent/..., Tracing/... are matched by prefix). Do not pass --datastream-ids.

observe ingest-token create \
  --name "K8s Explorer - <CLUSTER_NAME>" \
  --description "Ingest token for Kubernetes Explorer agent on <CLUSTER_NAME>"

IMPORTANT — display the token to the user immediately, in the same shape the Observe UI uses when minting a token: a monospaced block they can copy from, paired with a clear "save this now" warning. Wait for the user to confirm they have copied it before continuing to Phase 4.

Use roughly this format:

Your ingest token (copy now — Observe will not show it to you again):

    <SECRET_VALUE>

Save this somewhere safe (a password manager, a secrets vault, a trusted
note). The next phase uses it to create a Kubernetes secret for the helm
chart. There is no way to retrieve this token later via the CLI or the UI;
if you lose it, you'll need to mint a new one and update the secret.

Have you copied it? (yes/no)

Do not proceed until the user confirms. If they say no, repeat the token. If they explicitly skip the confirmation ("just go ahead"), proceed but flag that they're on their own to retrieve it before the helm install needs it.

---

Phase 4: Summary

After all resources are created, print a complete summary:

Backend Setup Complete
======================

Tenant: <CUSTOMER_URL>
Cluster: <CLUSTER_NAME>

Resources Created:
  Datastreams:
    - Kubernetes Explorer/OpenTelemetry Logs  (ID: <ID>)
    - Kubernetes Explorer/Prometheus          (ID: <ID>)
    - Kubernetes Explorer/Kubernetes Entity   (ID: <ID>)
    - Observe Agent/Events                    (ID: <ID>)
    [If traces:]
    - Tracing/Span                            (ID: <ID>)
    - Metrics/OpenTelemetry                   (ID: <ID>)

  Content:
    - Kubernetes Explorer Content: installed
    [If traces:]
    - Tracing Content: installed

  Ingest Token:
    - Name: "K8s Explorer - <CLUSTER_NAME>"
    - Secret: <TOKEN_SECRET>  ← save this; it cannot be retrieved again

---

Error Handling

At any phase, if an operation fails:

1. Show the full error message
2. Show what has been created so far
3. Ask the user how to proceed:

AskQuestion:
  id: error-recovery
  prompt: "An error occurred. How would you like to proceed?"
  options:
    - id: retry
      label: "Retry the failed step"
    - id: skip
      label: "Skip this step and continue"
    - id: abort
      label: "Stop here (resources created so far are kept)"

---

Backend Resource Removal

The Observe CLI does not expose delete commands for datastreams, content packages, or ingest tokens. This is intentional — these resources can hold or feed historical data, and removing them via an automated agent risks irreversible data loss.

If the user wants to remove backend resources, direct them to do it manually in the Observe UI. Do not attempt to delete via the CLI or any other automated path.