bkit-evals

# bkit Evals — Skill Quality Evaluation Runner > v2.1.11 Sprint β FR-β2. Wraps `evals/runner.js` with input validation, > result persistence, and structured reporting. Replaces the bare `node > evals/runner.js <skill>` invocation that previously required users to > remember argv structure and ignored timeout / sandbox concerns. ## Arguments | Argument | Description | Example | |----------|-------------|---------| | `run <skill>` | Execute the eval suite for one skill | `/bkit-evals run gap-detector` | | `list` | List all skills that have an `eval.yaml` definition | `/bkit-evals list` | If no argument is provided, render the same output as `list`. ## Behavior ### `run <skill>` 1. Validate `skill` against `/^[a-z][a-z0-9-]{0,63}$/`. Reject anything else (no shell metacharacters, no slashes, no spaces) — see Security below. 2. Spawn `node evals/runner.js --skill <skill>` via `child_process.spawnSync` (argv form, no shell). Default timeout 30 s, max 120 s. The `--skill` flag form is mandated by the runner CLI and locked by L3 contract test. 3. Capture stdout / stderr. Parse the trailing JSON block via balanced-brace fallback (string-aware). 4. Apply fail-closed defense: if `parsed === null` and stdout includes `Usage:`, return `reason: 'argv_format_mismatch'`; if `parsed === null` otherwise, return `reason: 'parsed_null'`. Exit code 0 alone NEVER implies success — the parsed JSON must be present. 5. Persist the structured result to `.bkit/runtime/evals-{skill}-{ISO timestamp}.json` with stdout/stderr tails (2000 chars each), `parsed` payload, and `reason` field. 5. Render a one-line summary in the chat: - exit code - parsed pass/fail counts (if available) - path of the persisted result file ### `list` 1. Read `evals/config.json` to enumerate skill classifications. 2. For each classification (`workflow`, `capability`, `hybrid`), list skills that have `evals/{classification}/{skill}/eval.yaml`. 3. Render a category-grouped table with skill name + a one-line note from the eval YAML (`description` field if present). ## Security - Skill name regex prevents argument injection. Anything outside `[a-z][a-z0-9-]{0,63}` is rejected with `reason: invalid_skill_name`. - argv-array spawn (no shell). No template-string concatenation into command lines. - Result file path is composed from a hardcoded base + sanitized skill name + timestamp; no traversal possible. - Subprocess timeout enforced (default 30 s, hard cap 120 s) so a buggy eval cannot block the session indefinitely. ## Module Dependencies | Module | Function | Usage | |--------|----------|-------| | `lib/evals/runner-wrapper.js` | `invokeEvals(skill, opts)` | Validate + spawn + persist | | `lib/evals/runner-wrapper.js` | `isValidSkillName(name)` | Regex pre-check shared with `list` | | `evals/runner.js` | (subprocess) | Existing eval execution engine | ## Result Schema `.bkit/runtime/evals-{skill}-{timestamp}.json`: ```json { "skill": "gap-detector", "invokedAt": "<ISO 8601>", "exitCode": 0, "timedOut": false, "stdoutTail": "...", "stderrTail": "...", "parsed": { /* whatever runner.js prints as JSON, or null */ } } ``` ## Examples ```bash # Single eval /bkit-evals run gap-detector # Discovery /bkit-evals list ``` ## Related - `/control trust` — eval results contribute to trust score - `/code-review` — uses eval data when assessing skills - `/bkit explore` (FR-β1) — explore evals as a category ARGUMENTS: