# k8s-ai-bench
`k8s-ai-bench` is a benchmark for assessing the performance of LLM models for Kubernetes related tasks. It evaluates AI agents (like `kubectl-ai`) on their ability to perform real-world Kubernetes operations such as creating deployments, debugging crash loops, and scaling applications.
## 📊 Live Dashboard
See [k8s-ai-bench live leaderboard](https://gke-labs.github.io/k8s-ai-bench/) for the latest benchmark results.
The leaderboard shows run results for widely used proprietary and open models over a few run types:
* **Pass@1**: Can the agent solve the task on the first try? This measures raw capability and immediate correctness.
* **Pass@5**: Can the agent solve the task at least once in 5 attempts? This shows if the agent can eventually find a solution.
* **Pass^5**: Does the agent solve the task every single time? This measures reliability and consistency, which is crucial for autonomous usage.
## 🚀 Quick Start
### 1. Build the Binary
```sh
go build
```
### 2. Run an Evaluation
Run the benchmark against your agent binary. Results will be saved to the `.build` directory.
```sh
# Basic usage
./k8s-ai-bench run --agent-bin <path/to/kubectl-ai> --output-dir .build/k8s-ai-bench
# Run a specific task type (e.g., scaling tasks)
./k8s-ai-bench run --agent-bin <path/to/kubectl-ai> --task-pattern "scale" --output-dir .build/k8s-ai-bench
```
### Run with Docker
You can also build a container image and run the benchmark with mounted agent,
kubeconfig, Docker socket, and output directories:
```sh
docker build -t k8s-ai-bench .
docker run --rm \
-e GEMINI_API_KEY \
-v /var/run/docker.sock:/var/run/docker.sock \
-v "$(command -v kubectl-ai):/usr/local/bin/kubectl-ai:ro" \
-v "$PWD/.build:/bench/.build" \
k8s-ai-bench run --agent-bin kubectl-ai --output-dir /bench/.build/k8s-ai-bench
```
See [Docker usage](docs/docker.md) for kind, vCluster, matrix, and analyze
examples.
## 🛠 Usage Guide
### `run` Subcommand
The `run` subcommand executes the benchmark evaluations. It creates ephemeral clusters to ensure test isolation. We support two platforms for the test environment: **Kind** (default) and **vCluster**.
**vCluster Prerequisites:**
For a detailed step-by-step guide on setting up and using vCluster, see [vCluster Guide](docs/vcluster.md).
To use `vcluster`, you must have:
* The `vcluster` [CLI](https://www.vcluster.com/docs/vcluster/) installed.
* A running host Kubernetes cluster.
* A kubecontext to connect to the host cluster (passed via `--host-cluster-context`).
```sh
# Run with specific LLM provider and model
./k8s-ai-bench run \
--agent-bin <path/to/kubectl-ai> \
--llm-provider gemini \
--models gemini-2.5-pro-preview-03-25 \
--task-pattern fix \
--output-dir .build/k8s-ai-bench
```
**Common Flags:**
| Flag | Description | Default |
|------|-------------|---------|
| `--agent-bin` | Path to kubectl-ai binary (Required) | - |
| `--output-dir` | Directory to write results (Required) | - |
| `--task-pattern` | RegEx pattern to filter tasks (e.g. 'pod', 'fix') | - |
| `--llm-provider` | LLM provider ID (e.g. 'gemini', 'openai') | gemini |
| `--models` | Comma-separated list of models | gemini-2.5-pro... |
| `--concurrency` | Number of parallel tasks (0 = auto) | 0 |
| `--cluster-provider` | Cluster provider to use (`kind` or `vcluster`) | kind |
| `--host-cluster-context` | Host cluster context for vcluster (Required if provider is vcluster) | - |
### Agent/Skill/CLI Matrix Mode
For skill and CLI benchmarks, use a matrix file to declare agents, models, optional local skill/CLI directories, task directories, output directories, and run settings:
```sh
./k8s-ai-bench run \
--matrix-file examples/skill-cli/eval-matrix.yaml
```
Each task can select the agent and declare any required local skills and CLIs explicitly:
```yaml
agent: generic
skills:
- kube-debug/SKILL.md
clis:
- name: kube-inspector
path: kube-inspector
required: true
script:
- prompt: Diagnose and fix the failing pod using the provided skill.
cliExpect:
- name: kube-inspector
required: true
argvContains:
- inspect
- pod
verifier: verify.sh
difficulty: medium
```
Task lifecycle scripts can be written for the target operating system. Unix
tasks can use `setup.sh`, `verify.sh`, and `cleanup.sh`. On Windows, use the
corresponding `.ps1` files; the benchmark invokes them with PowerShell using
`-NoProfile -NonInteractive -ExecutionPolicy Bypass -File`. The lifecycle
environment, including `KUBECONFIG`, `K8S_AI_BENCH_TASK_OUTPUT_DIR`, and agent
configuration variables, is passed to both Bash and PowerShell scripts.
For generic Skill/CLI evaluation, build the bundled minimal agent and reference it with the `generic-stdin` adapter:
```sh
go build -o generic-llm-agent ./cmd/generic-llm-agent
```
The generic agent reads the injected prompt from stdin, asks the configured LLM to emit `<command>` or `<final>` blocks, and executes commands from `PATH`. Benchmark CLI wrappers still provide auditing.
Hermes can be evaluated through the unified connector bridge:
```sh
go build -o k8s-ai-agent-bridge ./cmd/k8s-ai-agent-bridge
./k8s-ai-bench run --matrix-file examples/agent-connectors/eval-matrix-hermes.yaml
```
The older `k8s-ai-hermes-bridge` binary remains available for compatibility.
Codex CLI, Claude Code, OpenClaw, and Hermes can use the same bridge. Set
`runs.agent` in a matrix file to select one configured connector. Codex and
Claude use headless CLI modes; OpenClaw and Hermes use their
OpenAI-compatible gateway endpoints. See [Agent Connectors](docs/agent-connectors.md).
To run one specific agent from a matrix that defines several agents, set
`runs.agent` to the configured agent ID:
```yaml
agents:
- id: codex
bin: ./k8s-ai-agent-bridge
adapter: generic-stdin
args: [--agent, codex]
- id: openclaw
bin: ./k8s-ai-agent-bridge
adapter: generic-stdin
args: [--agent, openclaw]
env:
OPENCLAW_BASE_URL: ${OPENCLAW_BASE_URL}
OPENCLAW_GATEWAY_TOKEN: ${OPENCLAW_GATEWAY_TOKEN}
OPENCLAW_AGENT_TARGET: ${OPENCLAW_AGENT_TARGET}
runs:
agent: codex
taskPattern: "create-pod"
```
Then run the matrix normally:
```sh
./k8s-ai-bench run \
--matrix-file examples/agent-connectors/eval-matrix-agents.yaml
```
Change `runs.agent` to `openclaw`, `claude`, or `hermes` to select another
configured connector. A run-level agent takes precedence over `agent` declared
inside an individual task. See the complete example at
[`examples/agent-connectors/eval-matrix-agents.yaml`](examples/agent-connectors/eval-matrix-agents.yaml).
See [docs/agent-skill-cli-matrix.md](docs/agent-skill-cli-matrix.md) for the full matrix workflow, adapter contracts, CLI wrapper audit behavior, and Hermes bridge details.
### `analyze` Subcommand
Process and summarize results from previous runs.
```sh
# Generate a Markdown report
./k8s-ai-bench analyze --input-dir .build/k8s-ai-bench --results-filepath report.md
# Generate JSONL for visualization
./k8s-ai-bench analyze --input-dir .build/k8s-ai-bench --output-format jsonl --results-filepath site/combined_results.jsonl
```
## 💻 Development Scripts
For a streamlined development loop, use the scripts in `dev/ci/periodics/`:
- **Run Evaluation Loop**: Runs evaluations multiple times to test consistency.
```sh
./dev/ci/periodics/run-eval-loop.sh --iterations 5 --task-pattern "create"
```
- **Run Single Evaluation**:
```sh
TEST_ARGS="--task-pattern=fix-probes" ./dev/ci/periodics/run-evals.sh
```
- **Analyze Results**:
```sh
./dev/ci/periodics/analyze-evals.sh --show-failures
```
## 📈 Visualizing Results Locally
The `site` directory contains a static website (Vue.js based) for visualizing benchmark results.
1. **Generate Data**:
```sh
./k8s-ai-bench analyze --input-dir .build/k8s-ai-bench --output-format jsonl --results-filepath site/combined_results.jsonl
```
2. **Serve Locally**:
```sh
cd site
python3 -m http.server
```
3. **View**: Open [http://localhost:8000](http://localhost:8000)
## 🤝 Contributions
We welcome contributions! Please check out the [contributions guide](contributing.md).