# lightspeed-agentic-operator
Kubernetes controller for the agentic proposal workflow (`agentic.openshift.io/v1alpha1`): AgenticRun, AgenticRunApproval, Agent, LLMProvider, ApprovalPolicy, and related resources.
- **How to work (agents)**: [`agent.md`](agent.md)
- **Two Go modules (`go.mod` / `go.sum` at root and under `api/`)**: below; **directory map, phases, conventions**: [`CLAUDE.md`](CLAUDE.md)
- **Tests, manifests, Makefile, cluster workflow, `make api-lint`, CEL / `XValidation`**: this file
## CLI Plugin (`oc-agentic`)
The `oc-agentic` binary is an `oc` CLI plugin for managing proposals. Place it in `$PATH` and `oc` auto-discovers it as `oc agentic <subcommand>`. It also works standalone.
### Install
```bash
# Linux amd64
curl -L https://github.com/openshift/lightspeed-agentic-operator/releases/latest/download/oc-agentic_linux_amd64.tar.gz | tar xz
sudo mv oc-agentic /usr/local/bin/
# macOS Apple Silicon
curl -L https://github.com/openshift/lightspeed-agentic-operator/releases/latest/download/oc-agentic_darwin_arm64.tar.gz | tar xz
sudo mv oc-agentic /usr/local/bin/
```
### Usage
```bash
# Create a proposal
oc agentic run create --request="Fix crashloop in my-app namespace"
# List proposals
oc agentic run list
# Inspect a proposal
oc agentic run get ag-abc12
# Approve steps (analysis → execution → verification)
oc agentic run approve ag-abc12 --stage=analysis
oc agentic run approve ag-abc12 --stage=execution --option=0
oc agentic run approve ag-abc12 --stage=verification
# Or approve all pending steps at once
oc agentic run approve ag-abc12 --all --wait
# Deny a step
oc agentic run deny ag-abc12 --stage=execution
# Watch phase transitions
oc agentic run watch ag-abc12
# Stream sandbox logs
oc agentic run logs ag-abc12 --step=Execution -f
# Delete a proposal
oc agentic run delete ag-abc12
# System operations
oc agentic status # check if system is active or suspended
oc agentic suspend --yes # halt all agentic operations
oc agentic resume # resume operations
oc agentic version
```
### Command Reference
| Command | Description |
| --------- | ------------- |
| `proposal create` | Create a new proposal (`--request`, `--agent`) |
| `proposal list` (`ls`) | List proposals (`-A`, `--phase`, `-o wide\|json\|yaml`) |
| `proposal get` | Show proposal details (`-o json\|yaml`) |
| `proposal approve` | Approve a step (`--stage`, `--option`, `--agent`, `--all`, `--wait`) |
| `proposal deny` | Deny a step (`--stage`, defaults to next pending) |
| `proposal watch` | Stream phase transitions until terminal |
| `proposal logs` | Stream sandbox pod logs (`--step`, `-f`) |
| `proposal delete` | Delete a proposal |
| `status` | Show system suspension state |
| `suspend` | Suspend all operations (`--yes` to skip prompt) |
| `resume` | Resume operations |
| `version` | Print plugin version |
Default namespace is `openshift-lightspeed` unless overridden with `-n` or kubeconfig context.
### Preserve a Failed Sandbox for Debugging
To retain the sandbox Pod and associated diagnostic resources after a failed run, add this annotation to the `AgenticRun`:
```yaml
metadata:
annotations:
agentic.openshift.io/preserve-sandbox: "true"
```
This is an opt-in debugging aid. Terminal TTL processing is disabled for the failed run while the annotation is present. Delete the `AgenticRun` when investigation is complete; normal cleanup then removes the preserved sandbox resources.
## Development
Run the manager against a cluster using the usual controller-runtime kubeconfig rules (`KUBECONFIG`, default kubeconfig path, or in-cluster as a pod). Auth plugins (OIDC, GCP, Azure, …) are registered via `k8s.io/client-go/plugin/pkg/client/auth`.
### Go modules (two `go.mod` trees)
This repo has **two Go modules**:
| Location | Module path | Role |
|----------|-------------|------|
| Repo root **`go.mod`** | `github.com/openshift/lightspeed-agentic-operator` | Controller, CLI, **`cmd/`**, **`controller/`**, etc. |
| **`api/go.mod`** | `github.com/openshift/lightspeed-agentic-operator/api` | CRD types and API helpers only—downstreams can **`require`** this module without pulling the full operator graph. |
The root module **`replace`s** `…/api` **`=> ./api`** for local builds. The **`Dockerfile`** copies **both** pairs of **`go.mod` / `go.sum`** before **`go mod download`**.
**When changing Kubernetes / controller-runtime versions**, keep the two **`go.mod`** files in sync (same PR / same intent) and run **`go mod tidy`** at the root and under **`api/`** as needed.
**`make test`** and **`make api-lint`** run tests and the kube API linter under **`api/`** with **`GOWORK=off`** so a repo-root **`go.work`** does not force the wrong module boundary.
### Agent Sandbox
Reconciling proposals uses [kubernetes-sigs/agent-sandbox](https://github.com/kubernetes-sigs/agent-sandbox) (`SandboxClaim`, `Sandbox`, `SandboxTemplate`). **`make run`** and **`make deploy`** both run **`install-agent-sandbox`** first: it only hits the network if the three Sandbox CRDs are missing (otherwise three cheap `kubectl get crd` calls). You can run **`make install-agent-sandbox`** alone to set **`AGENT_SANDBOX_VERSION`** without a full **`make run`**.
```bash
make install-agent-sandbox # optional; also run automatically before make run / make deploy
# Override release: make install-agent-sandbox AGENT_SANDBOX_VERSION=v0.4.5
```
Upstream install details: [Agent Sandbox installation](https://agent-sandbox.sigs.k8s.io/docs/getting_started/install_prerequisites/). The Makefile target applies **`manifest.yaml`** and **`extensions.yaml`** for **`AGENT_SANDBOX_VERSION`** (default pinned in the Makefile). You still need a **`SandboxTemplate`** named like **`TEMPLATE_NAME`** (default `lightspeed-agent`) in **`OPERATOR_NAMESPACE`** — usually from Lightspeed / your platform, not from this target.
### CRDs and manifests
Do not edit **`config/crd/bases/`** or generated **`config/rbac/role.yaml`** by hand. After changing **`api/v1alpha1/`** types or kubebuilder markers, run **`make manifests`**.
### Makefile defaults
**`OPERATOR_NAMESPACE`** defaults to **`default`**: used for **`make run`** (`--namespace`), and for **`make deploy`** / **`make undeploy`** (Namespace, Deployment, ServiceAccount, RBAC subject namespace). Manifests under `config/` use **`__OPERATOR_NAMESPACE__`**; the Makefile substitutes **`$(OPERATOR_NAMESPACE)`** before `kustomize build`. Local **`make run`** uses metrics **`:18080`** and health **`:18081`** by default; override with **`METRICS_BIND_ADDRESS`** and **`HEALTH_PROBE_BIND_ADDRESS`** if needed.
### Common targets
```bash
make manifests # controller-gen → config/crd/bases + config/rbac/role.yaml
make test # fmt-check + vet + go test (root + api module)
make install # kubectl apply CRDs from config/crd only
make run # install + install-agent-sandbox + vet + go run ./cmd/main.go
make uninstall # kubectl delete CRDs from config/crd (optional: ignore-not-found=true)
make deploy # apply only; IMG must be pullable (CI / released image)
make deploy-local # OpenShift (see Makefile)
make undeploy # same layout | kubectl delete (optional: ignore-not-found=true)
make api-lint # Kube API linter on api/ (golangci-lint custom + plugin; see below)
```
### Testing
**`make test`** runs **`fmt-check`** (fails if any file needs reformatting — run **`make fmt`** to fix), **`vet`**, root-module tests, and **`cd api && GOWORK=off go test ./... -count=1`** (the API tree is a separate module; **`GOWORK=off`** avoids a repo-root **`go.work`** hijacking module choice). The **`test/e2e`** package is excluded from this target — it requires **`-tags=e2e`** and a running mock agent.
**`make test-e2e`** runs **`go test -tags=e2e ./test/e2e/...`** against a live cluster with the operator running. Prerequisites: **`make run TEMPLATE_NAME=lightspeed-agent-mock`** (or deployed operator with **`--template-name=lightspeed-agent-mock`**) and mock agent SandboxTemplate applied (**`kubectl apply -k test/agent/sandboxtemplate`**). See `test/e2e/` package doc for details.
### Product E2E (real providers)
Product E2E runs the troubleshooting scenarios against a real LLM provider. It deploys (or reuses) the operator, clones `rhobs/troubleshooting-scenarios`, and runs the `product_e2e`-tagged tests. For example, to run Claude scenarios:
```bash
IMG=quay.io/.../lightspeed-agentic-operator@sha256:<digest> \
VERTEX_PROVIDER_KEY_PATH=/path/to/service-account.json \
VERTEX_PROJECT_ID=my-gcp-project \
ARTIFACT_DIR="$PWD/artifacts" \
bash scripts/e2e-cluster.sh claude
```
`claude` and `gemini` require `VERTEX_PROVIDER_KEY_PATH` and `VERTEX_PROJECT_ID`; `openai` requires `OPENAI_PROVIDER_KEY_PATH`; `azure-openai` requires `AZUREOPENAI_PROVIDER_KEY_PATH`; `bedrock-deepseek` requires `BEDROCK_AWS_ACCESS_KEY_ID` and `BEDROCK_AWS_SECRET_ACCESS_KEY`; and `bedrock-claude` requires `BEDROCK_ROLE_AWS_ACCESS_KEY_ID` and `BEDROCK_ROLE_AWS_SECRET_ACCESS_KEY`. The default run order is `claude`, `gemini`, `openai`, `azure-openai`, `bedrock-deepseek`, then `bedrock-claude`.
| Variable | Purpose |
| --- | --- |
| `E2E_SCENARIO_TAGS=alert` | Run scenarios carrying the `alert` tag instead of the default `core` tag. Multiple tags are an **AND** filter: `core,alert` requires both. |
| `E2E_SKIP_SCENARIOS=pending_pvc_alert` | Skip comma-separated scenario directory names. |
| `E2E_SCENARIO_TIMEOUT=20m` | Per-scenario deadline (default: `20m`). |
| `E2E_OTEL_ENABLED=false` | Disable the persistent OTEL/Postgres collector used for product-E2E artifacts. It is enabled by default. |
| `E2E_OTEL_IMAGE=quay.io/.../collector:tag` | Override the OTEL collector image. |
| `ARTIFACT_DIR=/path/to/artifacts` | Directory where test and diagnostic artifacts are written. |
With `ARTIFACT_DIR` set, artifacts are arranged per provider:
```text
artifacts/<provider>/
├── product-e2e-output.log # Go test output
├── operator-logs.txt # controller-manager logs
├── pods.yaml # remaining operator-namespace pods
├── podlogs/ # best-effort direct sandbox-pod log backup
└── runs/<run-name>/
└── otel-sandbox.json # persisted OTEL collector records for the run
```
The runner configures the collector with its Postgres backend, because the default `nop` pipeline drops records and does not expose the admin API used to export per-run records. Direct pod-log capture is best effort: sandbox pods may be removed immediately after completion, so persisted OTEL records are the primary diagnostic source.
### Disconnected Gemma product E2E
`make product-e2e-disconnected` clones `lightspeed-service` at a verified full
commit SHA. Its own calling script orchestrates that checkout's existing RHOAI
scripts/manifests with `VLLM_MODEL_PROFILE=gemma-4-31b`, then runs the existing
core product suite under temporary sandbox/vLLM egress
NetworkPolicies. Model/image/template preparation happens while connected;
there is no unrestricted retry after the boundary is installed.
```bash
LIGHTSPEED_SERVICE_REF=<full-40-character-commit-with-compatible-Gemma-RHOAI-assets> \
HUGGING_FACE_HUB_TOKEN=<CI-secret> VLLM_API_KEY=<CI-secret> \
IMG=<operator-pullspec> \
SANDBOX_IMAGE=image-registry.openshift-image-registry.svc:5000/tests/sandbox:gemma \
E2E_SKILL_IMAGE_MAP="$PWD/mirrored-skills.json" \
ARTIFACT_DIR="$PWD/artifacts" \
make product-e2e-disconnected
```
Supply credentials through the CI secret environment, not checked-in files.
CI must mirror the sandbox and core-scenario skill images beforehand.
`mirrored-skills.json` maps original skill pullspecs to internal ones:
```json
{
"quay.io/example/skills:v1": "image-registry.openshift-image-registry.svc:5000/tests/skills:v1"
}
```
Unmapped references must already be internal; public references fail before
scenario setup or run creation. The actual configuration ConfigMap must use
`bare-pod` mode and local images. Use dedicated namespaces with no existing
egress policies or optional OTEL/MCP/RHOKP endpoints. The mirrored sandbox
image must include Python 3 for authenticated preflight probes.
`E2E_EGRESS_CANARY` optionally overrides the baseline HTTPS URL (default
`https://example.com/`). `E2E_SUITE_TIMEOUT` defaults to `12h`; the test rejects
a timeout too short for the discovered core scenarios. Scenario skips and
non-core tags are rejected. Policies and probe resources are removed after
redacted diagnostics are saved in `artifacts/disconnected/`. An invocation-labelled
shell cleanup path handles both normal teardown and hard test timeouts/INT/TERM.
A private run-UID journal keeps lingering sandboxes discoverable after run CR
removal. Cleanup archives run conditions/results and sandbox status/logs before
deletion, stops runs, waits for their sandbox Pods, and checks for residual Pods
before removing policies. Recovery snapshots use separate `fallback/retry.*`
directories to preserve initial evidence. The outer runner undeploys the operator
only after resource cleanup succeeds; failed recovery retains the operator/CRDs
and remaining policies, while preserving the original failure. The runner requires
Python 3, `setsid` (util-linux), `curl`, `envsubst` and GNU `timeout` (coreutils)
as well as the standard E2E tools. CI retains responsibility for GPU and model-serving resource teardown.
The pinned service commit must contain compatible `tests/rhoai/` assets:
`model-profile.sh` with `gemma-4-31b`, `bootstrap.sh`, `gpu-setup.sh`,
`fetch-vllm-image.sh`, `deploy-vllm.sh`, `get-vllm-pod-info.sh`, and their manifests.
Missing assets or incompatible profile settings fail clearly rather than falling
back to Llama or LSEval. There is no dependency on a separate service provisioning
entrypoint.
The operator-owned `scripts/e2e-rhoai.sh` creates the provisioning handoff from
actual Service/Pod data and an authenticated models request inside the inference
container using the caller's current API key via stdin. Existing inference
Deployments are restarted so their Pods pick up updated Secret-backed credentials.
The helper uses the Service port for the internal `/v1` URL and resolves the Pod
target port separately for NetworkPolicy. `E2E_RHOAI_READY_TIMEOUT` controls
rollout/InferenceService readiness waiting (default `60m`).
`E2E_RHOAI_PROVISION_TIMEOUT` bounds provisioning scripts and their children
(default `120m`); expiration preserves exit status 124 and saves redacted
inference logs alongside the failure diagnostics.
Cluster-free harness tests: `make test` and `make test-product-e2e-unit`.
See [the implementation contract](.ai/spec/how/disconnected-product-e2e.md).
For noisy debugging: **`go test ./controller/agenticrun/... -v`**, **`go test ./api/... -v`**, **`go test ./cli/... -v`**.
### API lint (Kube API linter)
**`make api-lint`** installs **`golangci-lint`** to **`bin/`** (version from **`.custom-gcl.yml`**), runs **`golangci-lint custom`** (builds **`bin/golangci-lint-kube-api-linter`**), then lints **`api/`** with **`GOWORK=off`** and **`.golangci-kal.yml`**.
### CEL validation (`XValidation`)
kubebuilder **`+kubebuilder:validation:XValidation`** markers become CEL rules in generated CRD YAML (**`make manifests`**).
- **`omitempty` fields**: they are absent from the stored object when unset. CEL must use **`has(field)`** before reading them, or validation fails with *no such key*. Example: **`has(old.denied) && old.denied`** instead of **`old.denied`** alone.
- **Marker placement**: struct-level **`XValidation`** can produce double-quoted strings in generated YAML. Put **`XValidation`** on the **field** in the parent struct, not on the nested type.
### `deploy`
**`kustomize build | kubectl apply`**. **`IMG`** is the image reference the Deployment uses; the image must already exist on the cluster.
### `deploy-local` (OpenShift)
**`make deploy-local`** — build, push to the cluster registry, and deploy the operator (**`oc`** and a push-capable registry required). Details live in the **`deploy-local`** recipe in the **`Makefile`**.
### `docker-build`
**`$(CONTAINER_TOOL) build -t $(IMG) .`** — see **`Dockerfile`**.