kensa

Set up evals for an agent codebase or check eval status after changes. Determines readiness, identifies what can be tested, and prepares the environment. Use when starting evals for the first time, returning after a code change, or figuring out what to do next. Also triggered by "set up evals", "is my agent ready?", "eval status", "what should I do next?", "init eval", "evaluate my agent", "test my agent", "help me eval this", "get started with evals", "where do I start", "how do I test this agent", "check my setup". This is the default entry point, use it whenever a user wants to evaluate an agent and you're unsure which skill to start with.

Design test scenarios for an agent, covering happy paths, edge cases, tool usage, error handling, and cost bounds. Use when writing new scenarios, editing existing ones, expanding coverage after a failed run, or when the user says "add a scenario", "improve my evals", "write scenarios", "I need test cases", "what should I test", "write test cases", "expand coverage", "add edge cases", "more scenarios". Also use when the user has a diagnosis from a failed run and needs targeted scenarios for specific failure patterns.

Understand what failed and why after an eval run. Reads judge results and traces, categorizes failures, identifies patterns, and recommends the next action. Use after running kensa eval or kensa judge, when results have failures, or when traces show anomalies. Also triggered by "what went wrong", "why did X fail", "diagnose my eval", "analyze errors", "my evals are failing", "everything failed", "check the results", "look at the failures", "debug my evals", "why is this failing", "evals broke". Use this whenever the user has run evals and wants to understand the results, even if they don't explicitly say "diagnose".

Create structured judge prompt files for subjective evaluation criteria that inline strings cannot handle. Produces .kensa/judges/*.yaml with binary pass/fail definitions and few-shot examples. Use when inline criteria produce inconsistent verdicts, when pass/fail boundaries need explicit definitions, or when multiple scenarios share an evaluation standard. Also triggered by "improve the judge", "write judge prompt", "judge is wrong", "calibrate judge", "generate judges", "judge keeps failing", "false positive", "false negative", "judge is too strict", "judge is too lenient", "inconsistent verdicts". Use this even when the user doesn't mention "judge" explicitly but describes evaluation inconsistency.

Validate a judge prompt against human-labeled examples. Measures TPR and TNR, identifies miscalibration, and iterates until both metrics meet threshold. Use after writing a structured judge with generate-judges, when verdicts seem wrong, or when you want confidence that the judge works before running evals at scale. Also triggered by "validate judge", "calibrate judge", "test the judge", "is my judge accurate", "judge quality", "judge metrics", "TPR", "TNR", "false positives", "false negatives", "the judge is wrong", "my judge sucks", "how good is my judge".