agent-aiops-on-aws

Plan, execute, and analyze LLM serving benchmarks across vLLM and SGLang configurations. Use when user says "run benchmarks", "benchmark config", "QPS sweep", "compare serving configs", "custbench", "benchmark analysis", or "generate benchmark report". Do NOT use for writing Terraform (use terraform-automation), deployment validation (use deployment-orchestrator), or GPU hardware diagnostics (use gpu-infra-troubleshooting).

Launch and manage detached agent runs on EKS via the agent-runner CLI. Use when the user says "launch an agent run", "agent runtime", "agent-runner", "run this spec on the cluster", "check run status", "agent run logs", or "stop a run". Do NOT use for writing the agent-runner code itself (edit the sibling ../agent-runner repo), Terraform of the foundation (use terraform-automation), or GPU serving deployments (use infra-deployer).

Evaluate upstream PRs or commits individually against the K2.6 baseline. Tests each change in isolation using the same L0-L4 pipeline, identifying which are helpful, harmful, or neutral for our specific workload. Use when new upstream PRs land (Alpha-MoE, Mega MoE, FP8 fusion) or when evaluating a bundle of commits. Never assume a bundle is net-positive — the EFA harness found bundled expert commits regressed LL performance by 10%.

Analyze a failed or underperforming kernel candidate using ncu profile data and telemetry history. Produces a structured diagnosis (what's wrong, why, what to try next) and updates the constraint database. Use after /verify-kernel returns a failure or an L3/L4 regression, or when the agent needs guidance on which optimization direction to pursue.

Generate an optimized kernel candidate using the current state vector approach. Injects all active constraints from the database, uses the specified DSL (Triton/TileLang/CUDA), and targets the highest-headroom region identified by profiling. Use after profiling identifies a target, or when the previous candidate fails and the state vector has advanced.

View, query, add, or demote constraints in the kernel optimization constraint database. The database is append-only — constraints are never deleted, only demoted in severity if contradicted by evidence. Use to inspect what the agent has learned, manually inject domain knowledge, or query relevant constraints before generating a candidate.

Profile a GPU kernel or full serving pipeline using NSight Compute. Produces roofline classification (compute/memory/latency-bound), identifies top-N kernels by wall-clock time, and maps each to pipeline stage (MoE dispatch, MLA decode, attention, comm). Use when starting optimization, after a new baseline is established, or when the freeze manager redirects to a new target region.

Run the cascaded L0-L4 verification pipeline on a kernel candidate. Each level gates the next — a candidate that fails L1 never wastes time on L4. Records telemetry at every stage regardless of pass/fail. Use after /generate-candidate produces a new kernel, or when evaluating an upstream PR cherry-pick.

Evaluate coding agent patches using the learned-verifier cascade (RF + multiprompt + debate). Use when user says "verify patch", "evaluate patch quality", "run cascade", "check if patch is correct", "batch evaluate traces", "bootstrap RF", "train verifier", or "flywheel". Do NOT use for writing or editing the verification rubrics themselves, running gold Docker tests, or modifying the cascade architecture. This skill is for USING the verifier as an evaluation overlay, not developing it.

Pre-flight validation, post-deployment verification, and failure recovery for AWS infrastructure deployments. Use when user says "pre-flight check", "readiness audit", "deployment checklist", "validate deployment", or "recover from failed deploy". Do NOT use for writing Terraform code (use terraform-automation) or for benchmark analysis (use benchmark-analyst agent).

Verify whether a coding agent's patch correctly fixes a bug. Scores patches on structured criteria (problem alignment, minimality, logic correctness, scope, completeness). Use when selecting the best patch from N candidates or filtering low-quality fixes.

Automate AWS infrastructure deployment using Terraform with security scanning, best practices, and iterative development. Use when user says "create infrastructure", "deploy with Terraform", "run Checkov scan", "validate Terraform", or "debug deployment failure". Do NOT use for reading or explaining existing Terraform code, answering general Terraform questions, or CloudFormation/CDK-only workflows.

Converts dense terminal output, benchmark data, and audit reports into interactive single-file HTML pages. Use when user says "visualize benchmarks", "generate diagram", "audit visual", "compound recap", or after benchmark-analyst, blueprint-reviewer, or compound-learner produce markdown output. Do NOT use for generating raw markdown reports, editing existing HTML files, or general web development tasks.