# lightspeed-agentic-operator
Kubernetes controller for the agentic proposal workflow (`agentic.openshift.io/v1alpha1`): AgenticRun, AgenticRunApproval, Agent, LLMProvider, ApprovalPolicy, and related resources.
- **How to work (agents)**: [`agent.md`](agent.md)
- **Two Go modules (`go.mod` / `go.sum` at root and under `api/`)**: below; **directory map, phases, conventions**: [`CLAUDE.md`](CLAUDE.md)
- **Tests, manifests, Makefile, cluster workflow, `make api-lint`, CEL / `XValidation`**: this file
## CLI Plugin (`oc-agentic`)
The `oc-agentic` binary is an `oc` CLI plugin for managing proposals. Place it in `$PATH` and `oc` auto-discovers it as `oc agentic <subcommand>`. It also works standalone.
### Install
```bash
# Linux amd64
curl -L https://github.com/openshift/lightspeed-agentic-operator/releases/latest/download/oc-agentic_linux_amd64.tar.gz | tar xz
sudo mv oc-agentic /usr/local/bin/
# macOS Apple Silicon
curl -L https://github.com/openshift/lightspeed-agentic-operator/releases/latest/download/oc-agentic_darwin_arm64.tar.gz | tar xz
sudo mv oc-agentic /usr/local/bin/
```
### Usage
```bash
# Create a proposal
oc agentic run create --request="Fix crashloop in my-app namespace"
# List proposals
oc agentic run list
# Inspect a proposal
oc agentic run get ag-abc12
# Approve steps (analysis → execution → verification)
oc agentic run approve ag-abc12 --stage=analysis
oc agentic run approve ag-abc12 --stage=execution --option=0
oc agentic run approve ag-abc12 --stage=verification
# Or approve all pending steps at once
oc agentic run approve ag-abc12 --all --wait
# Deny a step
oc agentic run deny ag-abc12 --stage=execution
# Watch phase transitions
oc agentic run watch ag-abc12
# Stream sandbox logs
oc agentic run logs ag-abc12 --step=Execution -f
# Delete a proposal
oc agentic run delete ag-abc12
# System operations
oc agentic status # check if system is active or suspended
oc agentic suspend --yes # halt all agentic operations
oc agentic resume # resume operations
oc agentic version
```
### Command Reference
| Command | Description |
| --------- | ------------- |
| `proposal create` | Create a new proposal (`--request`, `--agent`) |
| `proposal list` (`ls`) | List proposals (`-A`, `--phase`, `-o wide\|json\|yaml`) |
| `proposal get` | Show proposal details (`-o json\|yaml`) |
| `proposal approve` | Approve a step (`--stage`, `--option`, `--agent`, `--all`, `--wait`) |
| `proposal deny` | Deny a step (`--stage`, defaults to next pending) |
| `proposal watch` | Stream phase transitions until terminal |
| `proposal logs` | Stream sandbox pod logs (`--step`, `-f`) |
| `proposal delete` | Delete a proposal |
| `status` | Show system suspension state |
| `suspend` | Suspend all operations (`--yes` to skip prompt) |
| `resume` | Resume operations |
| `version` | Print plugin version |
Default namespace is `openshift-lightspeed` unless overridden with `-n` or kubeconfig context.
Suspension requests zero-grace deletion of sandbox Pods in both bare-pod and sandbox-claim modes, bypassing graceful shutdown. Ordinary cleanup keeps its existing termination behavior. This API request does not guarantee immediate process termination on an unreachable node.
Global suspension separates its terminal outcome from permission to resume. The
current-generation `SafeToResume=True` condition is required alongside current
`Suspended=True/AdminActivated` or `AdminActivatedWithErrors` before resuming or
deleting a suspended `AgenticOLSConfig`. The initial cleanup deadline is ten
seconds; unresolved work produces terminal error completion but does not itself
authorize either action. `oc agentic status` shows
the safety state and diagnostics; `oc agentic resume` remains an API-authorized
request and reports the admission denial when safety is unconfirmed. Control-plane
confirmation does not prove process death on an unreachable node.
### Preserve a Failed Sandbox for Debugging
To retain the sandbox Pod and associated diagnostic resources after a failed run, add this annotation to the `AgenticRun`:
```yaml
metadata:
annotations:
agentic.openshift.io/preserve-sandbox: "true"
```
This is an opt-in debugging aid. Terminal TTL processing is disabled for the failed run while the annotation is present. Delete the `AgenticRun` when investigation is complete; normal cleanup then removes the preserved sandbox resources.
## Development
Run the manager against a cluster using the usual controller-runtime kubeconfig rules (`KUBECONFIG`, default kubeconfig path, or in-cluster as a pod). Auth plugins (OIDC, GCP, Azure, …) are registered via `k8s.io/client-go/plugin/pkg/client/auth`.
### Go modules (two `go.mod` trees)
This repo has **two Go modules**:
| Location | Module path | Role |
|----------|-------------|------|
| Repo root **`go.mod`** | `github.com/openshift/lightspeed-agentic-operator` | Controller, CLI, **`cmd/`**, **`controller/`**, etc. |
| **`api/go.mod`** | `github.com/openshift/lightspeed-agentic-operator/api` | CRD types and API helpers only—downstreams can **`require`** this module without pulling the full operator graph. |
The root module **`replace`s** `…/api` **`=> ./api`** for local builds. The **`Dockerfile`** copies **both** pairs of **`go.mod` / `go.sum`** before **`go mod download`**.
**When changing Kubernetes / controller-runtime versions**, keep the two **`go.mod`** files in sync (same PR / same intent) and run **`go mod tidy`** at the root and under **`api/`** as needed.
**`make test`** and **`make api-lint`** run tests and the kube API linter under **`api/`** with **`GOWORK=off`** so a repo-root **`go.work`** does not force the wrong module boundary.
### Agent Sandbox
Reconciling proposals uses [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) (`SandboxClaim`, `Sandbox`, `SandboxTemplate`). **`make run`** and **`make deploy`** both run **`install-agent-sandbox`** first: it only hits the network if the three Sandbox CRDs are missing (otherwise three cheap `kubectl get crd` calls). You can run **`make install-agent-sandbox`** alone to set **`AGENT_SANDBOX_VERSION`** without a full **`make run`**.
```bash
make install-agent-sandbox # optional; also run automatically before make run / make deploy
# Override release: make install-agent-sandbox AGENT_SANDBOX_VERSION=v0.4.5
```
Upstream install details: [Agent Sandbox installation](https://agent-sandbox.sigs.k8s.io/docs/getting_started/install_prerequisites/). The Makefile target applies **`manifest.yaml`** and **`extensions.yaml`** for **`AGENT_SANDBOX_VERSION`** (default pinned in the Makefile). You still need a **`SandboxTemplate`** named like **`TEMPLATE_NAME`** (default `lightspeed-agent`) in **`OPERATOR_NAMESPACE`** — usually from Lightspeed / your platform, not from this target.
Startup checks all required Sandbox CRDs within one shared 30-second deadline.
A missing CRD selects bare-Pod mode; other discovery failures stop startup.
In forced bare-Pod mode, suspension cleanup verifies each unavailable optional API against its own CRD.
Installed APIs remain part of cleanup and confirmation.
Verification failures keep cleanup unconfirmed.
### Leader election and metrics
The deployed manager enables `--leader-elect` using an agentic-specific Lease in
its install namespace. A namespaced Role/RoleBinding grants election
permissions. Local `make run` keeps election disabled and HTTP metrics on
`:18080`.
The default install requires OpenShift service-ca and the Prometheus Operator
ServiceMonitor CRD. Metrics are served over HTTPS on `:8443` using the TLS
certificate and private key from the `agentic-operator-metrics-certs` Secret.
OpenShift service-ca creates and rotates this Secret at the metrics Service's
request; the Deployment mounts it at `/etc/tls/private`. Controller-runtime
reloads rotated certificates. The ServiceMonitor validates the service-ca chain
and Service DNS name; a NetworkPolicy allows metrics ingress only from cluster
Prometheus pods in the `openshift-monitoring` namespace. Like the classic
operator, metrics require client mTLS using the CA certificate in the
`client-ca.crt` data key of the `metrics-client-ca` ConfigMap in the
`openshift-monitoring` namespace. The ServiceMonitor presents the certificate
and private key from Prometheus's `metrics-client-certs` Secret. The install
namespace is labelled for cluster monitoring. A namespace-scoped discovery Role
and RoleBinding allow the `prometheus-k8s` ServiceAccount in the
`openshift-monitoring` namespace to read Services, Pods, Endpoints, and
EndpointSlices in the operator namespace.
Metrics consume the effective TLS policy from the `tls-profile`,
`tls-min-version`, and `tls-cipher-suites` data keys in the
`lightspeed-agentic-configuration` ConfigMap; only the classic operator resolves
OLSConfig overrides and cluster defaults. The agentic operator neither reads
those policy APIs nor depends on their Go types.
Missing, deleted, or invalid handoff TLS policy rejects new metrics TLS
handshakes without blocking manager startup or health probes. Valid policy
arriving later (or changing) applies safely to new connections without
restarting the manager. Existing connections are not forcibly closed. A
dedicated metrics-policy watcher runs on all replicas, including leader-election
standbys, independently of potentially blocking sandbox/telemetry
reconfiguration. It does not eagerly publish an API-read snapshot before its
informer starts, avoiding stale policy after startup deletion. Go-unsupported
ciphers are logged and omitted; TLS 1.3 cipher suites are managed by Go.
Missing/invalid monitoring client CA fails secure startup; client CA changes
require a restart. Serving-certificate rotation is reloaded automatically.
The `config/metrics/` and `config/prometheus/` kustomize directories are
included by the `config/default/` kustomization for downstream bundle tooling.
The monitor's TLS server-name namespace follows the Service namespace through a
kustomize replacement.
### CRDs and manifests
Do not edit **`config/crd/bases/`** or generated **`config/rbac/role.yaml`** by hand. After changing **`api/v1alpha1/`** types or kubebuilder markers, run **`make manifests`**.
Downstream OLM installations must synchronize the updated role before relying on suspension confirmation.
`lightspeed-operator` consumes these permissions through its pinned-source `make sync-agentic-crds` and bundle regeneration workflow.
Updating this repository does not update an installed OLM bundle.
### Makefile defaults
**`OPERATOR_NAMESPACE`** defaults to **`default`**: used for **`make run`** (`--namespace`), and for **`make deploy`** / **`make undeploy`** (Namespace, Deployment, ServiceAccount, RBAC subject namespace). Manifests under `config/` use **`__OPERATOR_NAMESPACE__`**; the Makefile substitutes **`$(OPERATOR_NAMESPACE)`** before `kustomize build`. Local **`make run`** uses metrics **`:18080`** and health **`:18081`** by default; override with **`METRICS_BIND_ADDRESS`** and **`HEALTH_PROBE_BIND_ADDRESS`** if needed.
### Common targets
```bash
make manifests # controller-gen → config/crd/bases + config/rbac/role.yaml
make test # fmt-check + vet + go test (root + api module)
make install # kubectl apply CRDs from config/crd only
make run # install + install-agent-sandbox + vet + go run ./cmd
make uninstall # kubectl delete CRDs from config/crd (optional: ignore-not-found=true)
make deploy # apply only; IMG must be pullable (CI / released image)
make deploy-local # OpenShift (see Makefile)
make undeploy # same layout | kubectl delete (optional: ignore-not-found=true)
make api-lint # Kube API linter on api/ (golangci-lint custom + plugin; see below)
```
### Testing
**`make test`** runs **`fmt-check`** (fails if any file needs reformatting — run **`make fmt`** to fix), **`vet`**, root-module tests, and **`cd api && GOWORK=off go test ./... -count=1`** (the API tree is a separate module; **`GOWORK=off`** avoids a repo-root **`go.work`** hijacking module choice). The **`test/e2e`** package is excluded from this target — it requires **`-tags=e2e`** and a running mock agent.
`KUBEBUILDER_ASSETS=/path/to/kubernetes-1.35-binaries make test` also runs the isolated API admission regression with shipped CRD, policy, and binding.
It runs without a controller.
Without assets, the regression skips; the directory needs `kube-apiserver`, `etcd`, and `kubectl`.
The regression rejects resume/deletion while `Suspended=True/Stopping` and `SafeToResume=True`; it accepts current safe terminal outcomes.
**`make test-e2e`** runs the configuration failover tests first, then **`go test -tags=e2e ./test/e2e/...`** against a live cluster with the operator running. Prerequisites: **`make run TEMPLATE_NAME=lightspeed-agent-mock`** (or deployed operator with **`--template-name=lightspeed-agent-mock`**) and mock agent SandboxTemplate applied (**`kubectl apply -k test/agent/sandboxtemplate`**). See `test/e2e/` package doc for details.
### Configuration failover E2E
`make test-configwatch-e2e` starts two managers against a cluster with a shared
leader-election Lease in a temporary namespace. It verifies that the standby
loads the initial `lightspeed-agentic-configuration` ConfigMap, processes updates
and deletion before promotion, and cannot create a sandbox Pod after promotion
without configuration. The managers and temporary namespace are cleaned up.
The test requires permission to create/delete namespaces and read/write
ConfigMaps and Leases in the temporary namespace; it creates no workflow fixtures.
Each replica's configuration cache starts unavailable. Its ConfigMap watcher
runs regardless of leadership, and workflow reconciliation is blocked until the
cache has configuration. There is no separate startup read that could leave
stale configuration outside the watcher's event history.
### Metrics E2E
`make test-metrics-e2e` tests an installed operator without creating workflow
fixtures. It requires OpenShift cluster monitoring, the metrics Service and
ServiceMonitor, a ready operator Pod, and a valid TLS policy in the
`lightspeed-agentic-configuration` ConfigMap. Set `TEST_NAMESPACE` if the operator
is not installed in the default `openshift-lightspeed` namespace.
The tests verify the service-ca serving-certificate Secret, HTTPS access to
`/metrics` using Prometheus's client certificate, rejection of requests without a
client certificate, and fresh `up` and `go_goroutines` samples in cluster Prometheus.
The direct HTTPS checks use port-forwarding; the Prometheus checks exercise
ServiceMonitor discovery, mTLS, and the NetworkPolicy path.
The test user needs permission to get Services and list Pods in the operator and
`openshift-monitoring` namespaces, get the operator's serving Secret, and get the
`metrics-client-certs` Secret and `serving-certs-ca-bundle` ConfigMap in the
`openshift-monitoring` namespace. The user also needs permission to create Pod
port-forwards in both namespaces, create/delete a temporary ServiceAccount in the
operator namespace, create its token through the `serviceaccounts/token`
subresource, and create/delete a ClusterRoleBinding granting the
`cluster-monitoring-view` ClusterRole. Granting that role requires RBAC `bind`
authorization or equivalent existing privileges. The tests
query the `thanos-querier` Service in the `openshift-monitoring` namespace with a
short-lived token for that ServiceAccount. TLS certificate verification remains
enabled. Temporary RBAC resources are removed on completion.
`make test-metrics-e2e-build` compiles these tests without contacting a cluster.
`make test` runs their Prometheus-response parsing unit tests, but not the
live-cluster checks. These tests are separate from `make test-e2e`, whose workflow
suite creates and deletes shared agent and approval-policy fixtures.
### Product E2E (real providers)
Product E2E runs the troubleshooting scenarios against a real LLM provider. It deploys (or reuses) the operator, clones `rhobs/troubleshooting-scenarios`, and runs the `product_e2e`-tagged tests. For example, to run Claude scenarios:
```bash
IMG=quay.io/.../lightspeed-agentic-operator@sha256:<digest> \
VERTEX_PROVIDER_KEY_PATH=/path/to/service-account.json \
VERTEX_PROJECT_ID=my-gcp-project \
ARTIFACT_DIR="$PWD/artifacts" \
bash scripts/e2e-cluster.sh claude
```
`claude` and `gemini` require `VERTEX_PROVIDER_KEY_PATH` and `VERTEX_PROJECT_ID`; `openai` requires `OPENAI_PROVIDER_KEY_PATH`; `azure-openai` requires `AZUREOPENAI_PROVIDER_KEY_PATH`; `bedrock-deepseek` requires `BEDROCK_AWS_ACCESS_KEY_ID` and `BEDROCK_AWS_SECRET_ACCESS_KEY`; and `bedrock-claude` requires `BEDROCK_ROLE_AWS_ACCESS_KEY_ID` and `BEDROCK_ROLE_AWS_SECRET_ACCESS_KEY`. The default run order is `claude`, `gemini`, `openai`, `azure-openai`, `bedrock-deepseek`, then `bedrock-claude`.
| Variable | Purpose |
| --- | --- |
| `E2E_SCENARIO_TAGS=alert` | Run scenarios carrying the `alert` tag instead of the default `core` tag. Multiple tags are an **AND** filter: `core,alert` requires both. |
| `E2E_SKIP_SCENARIOS=pending_pvc_alert` | Skip comma-separated scenario directory names. |
| `E2E_SCENARIO_TIMEOUT=20m` | Per-scenario deadline (default: `20m`). |
| `E2E_OTEL_ENABLED=false` | Disable the persistent OTEL/Postgres collector used for product-E2E artifacts. It is enabled by default. |
| `E2E_OTEL_IMAGE=quay.io/.../collector:tag` | Override the OTEL collector image. |
| `ARTIFACT_DIR=/path/to/artifacts` | Directory where test and diagnostic artifacts are written. |
With `ARTIFACT_DIR` set, artifacts are arranged per provider:
```text
artifacts/<provider>/
├── product-e2e-output.log # Go test output
├── operator-logs.txt # controller-manager logs
├── pods.yaml # remaining operator-namespace pods
├── podlogs/ # best-effort direct sandbox-pod log backup
└── runs/<run-name>/
└── otel-sandbox.json # persisted OTEL collector records for the run
```
The runner configures the collector with its Postgres backend, because the default `nop` pipeline drops records and does not expose the admin API used to export per-run records. Direct pod-log capture is best effort: sandbox pods may be removed immediately after completion, so persisted OTEL records are the primary diagnostic source.
### Disconnected Gemma product E2E
`make product-e2e-disconnected` clones `lightspeed-service` at a verified full
commit SHA. Its own calling script orchestrates that checkout's existing RHOAI
scripts/manifests with `VLLM_MODEL_PROFILE=gemma-4-31b`, then runs the existing
core product suite under temporary sandbox/vLLM egress
NetworkPolicies. Model/image/template preparation happens while connected;
there is no unrestricted retry after the boundary is installed.
```bash
LIGHTSPEED_SERVICE_REF=<full-40-character-commit-with-compatible-Gemma-RHOAI-assets> \
HUGGING_FACE_HUB_TOKEN=<CI-secret> VLLM_API_KEY=<CI-secret> \
IMG=<operator-pullspec> \
SANDBOX_IMAGE=image-registry.openshift-image-registry.svc:5000/tests/sandbox:gemma \
E2E_SKILL_IMAGE_MAP="$PWD/mirrored-skills.json" \
ARTIFACT_DIR="$PWD/artifacts" \
make product-e2e-disconnected
```
Supply credentials through the CI secret environment, not checked-in files.
CI must mirror the sandbox and core-scenario skill images beforehand.
`mirrored-skills.json` maps original skill pullspecs to internal ones:
```json
{
"quay.io/example/skills:v1": "image-registry.openshift-image-registry.svc:5000/tests/skills:v1"
}
```
Unmapped references must already be internal; public references fail before
scenario setup or run creation. The actual configuration ConfigMap must use
`bare-pod` mode and local images. Use dedicated namespaces with no existing
egress policies or optional OTEL/MCP/RHOKP endpoints. The mirrored sandbox
image must include Python 3 for authenticated preflight probes.
`E2E_EGRESS_CANARY` optionally overrides the baseline HTTPS URL (default
`https://example.com/`). `E2E_SUITE_TIMEOUT` defaults to `12h`; the test rejects
a timeout too short for the discovered core scenarios. Scenario skips and
non-core tags are rejected. Policies and probe resources are removed after
redacted diagnostics are saved in `artifacts/disconnected/`. An invocation-labelled
shell cleanup path handles both normal teardown and hard test timeouts/INT/TERM.
A private run-UID journal keeps lingering sandboxes discoverable after run CR
removal. Cleanup archives run conditions/results and sandbox status/logs before
deletion, stops runs, waits for their sandbox Pods, and checks for residual Pods
before removing policies. Recovery snapshots use separate `fallback/retry.*`
directories to preserve initial evidence. The outer runner undeploys the operator
only after resource cleanup succeeds; failed recovery retains the operator/CRDs
and remaining policies, while preserving the original failure. The runner requires
Python 3, `setsid` (util-linux), `curl`, `envsubst` and GNU `timeout` (coreutils)
as well as the standard E2E tools. CI retains responsibility for GPU and model-serving resource teardown.
The pinned service commit must contain compatible `tests/rhoai/` assets:
`model-profile.sh` with `gemma-4-31b`, `bootstrap.sh`, `gpu-setup.sh`,
`fetch-vllm-image.sh`, `deploy-vllm.sh`, `get-vllm-pod-info.sh`, and their manifests.
Missing assets or incompatible profile settings fail clearly rather than falling
back to Llama or LSEval. There is no dependency on a separate service provisioning
entrypoint.
The operator-owned `scripts/e2e-rhoai.sh` creates the provisioning handoff from
actual Service/Pod data and an authenticated models request inside the inference
container using the caller's current API key via stdin. Existing inference
Deployments are restarted so their Pods pick up updated Secret-backed credentials.
The helper uses the Service port for the internal `/v1` URL and resolves the Pod
target port separately for NetworkPolicy. `E2E_RHOAI_READY_TIMEOUT` controls
rollout/InferenceService readiness waiting (default `60m`).
`E2E_RHOAI_PROVISION_TIMEOUT` bounds provisioning scripts and their children
(default `120m`); expiration preserves exit status 124 and saves redacted
inference logs alongside the failure diagnostics.
Cluster-free harness tests: `make test` and `make test-product-e2e-unit`.
See [the implementation contract](.ai/spec/how/disconnected-product-e2e.md).
For noisy debugging: **`go test ./controller/agenticrun/... -v`**, **`go test ./api/... -v`**, **`go test ./cli/... -v`**.
### API lint (Kube API linter)
**`make api-lint`** installs **`golangci-lint`** to **`bin/`** (version from **`.custom-gcl.yml`**), runs **`golangci-lint custom`** (builds **`bin/golangci-lint-kube-api-linter`**), then lints **`api/`** with **`GOWORK=off`** and **`.golangci-kal.yml`**.
### CEL validation (`XValidation`)
kubebuilder **`+kubebuilder:validation:XValidation`** markers become CEL rules in generated CRD YAML (**`make manifests`**).
- **`omitempty` fields**: they are absent from the stored object when unset. CEL must use **`has(field)`** before reading them, or validation fails with *no such key*. Example: **`has(old.denied) && old.denied`** instead of **`old.denied`** alone.
- **Marker placement**: struct-level **`XValidation`** can produce double-quoted strings in generated YAML. Put **`XValidation`** on the **field** in the parent struct, not on the nested type.
### `deploy`
**`kustomize build | kubectl apply`**. **`IMG`** is the image reference the Deployment uses; the image must already exist on the cluster.
### `deploy-local` (OpenShift)
**`make deploy-local`** — build, push to the cluster registry, and deploy the operator (**`oc`** and a push-capable registry required). Details live in the **`deploy-local`** recipe in the **`Makefile`**.
### `docker-build`
**`$(CONTAINER_TOOL) build -t $(IMG) .`** — see **`Dockerfile`**.