benchmark-category-development

# Benchmark Category Development ## Purpose Use this repo-specific workflow to develop benchmark task categories such as DB access-control, integrity, query, vector, storage, or auth. The goal is a clean, central, provider-neutral task set with trustworthy verifier results and reproducible workflow comparisons. ## Core Rules - Keep tasks central and provider-neutral under `tasks/<module>/<category>/<task_id>/`. - Do not preserve legacy paths, compatibility fallbacks, pilot concepts, or historical overlays. - Prompts describe the product goal and expected backend contract only. Workflow/tool restrictions belong in workflow profiles, not task prompts. - Verifiers test observable behavior, security properties, and production readiness. Do not require a specific implementation unless the task is explicitly about that implementation contract. - Use attack scripts for negative/security probes when the task can have false-positive pass cases. - Treat failures as data only after checking whether they are real agent failures, verifier false negatives, verifier false positives, harness bugs, timeout hangs, or isolation-rule issues. - Sandbox is the default. Do not depend on the user's local dev containers or external provider projects. - Avoid adding benchmark-specific instructions to product skills. Task prompts may state fair domain constraints, such as DB work staying in `public` schema when the benchmark target only supports that schema. ## Category Design Before writing tasks, define the category boundary in one sentence. Use these DB category meanings unless the user changes them: - `access-control`: row, relation, field, role, ownership, and sharing authorization. - `integrity`: invariants, lifecycle rules, constraints, triggers, derived state, state transitions, and maintenance logic. - `query`: reporting, aggregation, search, pagination, query-facing views/functions, and query performance as an optional metric. - `vector`: embeddings, similarity retrieval, hybrid search, metadata filters, and ranking. Build a candidate list, then reduce it. Keep only tasks that cover a distinct failure mode or product capability. Merge only when the combined task is still a natural real-world scenario, not two unrelated tasks glued together. For each category, prefer fewer high-quality tasks over many repetitive ones. Difficulty labels should reflect expected agent complexity and runtime: - `simple`: one core concept, quick implementation, useful as a baseline. - `medium`: two or three interacting requirements, likely one non-obvious edge case. - `hard`: realistic multi-step backend behavior, but still designed to finish within about 5 minutes. ## Task Authoring Checklist Each task directory should normally contain: ```text meta.json prompt.md prepare.ts verify.ts attack.ts cleanup.ts state.ts ``` Use `run.ts` only when a compact single-file task is clearly cleaner. `meta.json`: - `suite` is the module suite, such as `db`. - `capability` equals the category directory name. - `difficulty` is `simple`, `medium`, or `hard`. - `timeoutMs` should usually be `300000` for agent-backed benchmark tasks. `prompt.md`: - Tell the agent to read `task-state.json`. - Use exact names from state. - Keep the task provider-neutral. - Avoid direct HTTP/API/client test instructions; verifier owns user registration, login, uploads, reads, attacks, and production-readiness checks. - Require `implementation.json` with useful completion notes. `prepare.ts`: - Create deterministic randomized names and all seed users/data needed by verifier. - Keep provider differences out unless the provider architecture truly requires it. `verify.ts`: - Exercise the intended production path using realistic client/admin behavior. - Assert positive behavior, negative behavior, and edge cases. - Prefer outcome checks over introspecting exact SQL, policy names, or implementation shape. - Include diagnostics that make failures understandable without reading every log line. `attack.ts`: - Try bypasses that a broken implementation might accidentally allow. - For access control, include cross-user, cross-tenant, privilege escalation, protected-field mutation, and stale-access attempts when relevant. `cleanup.ts`: - Drop task-created objects and data only. - Be safe to run after partial agent failure. ## Calibration Workflow 1. Run `npm run typecheck`. 2. Run one task at `k=1` on the most relevant InsForge workflow. 3. If it fails, inspect logs before changing anything. 4. Classify the failure: - real agent failure - verifier false negative - verifier false positive risk - harness/sandbox issue - timeout/hang before meaningful work - workflow isolation rule issue 5. Fix only task or harness issues. Do not tune prompts or skills just to make one task pass. 6. Re-run the affected task after each fix. 7. Once each task is credible, run the selected cross-provider workflow set. Useful command shapes: ```bash npm run typecheck npm run benchmark:k -- --k 1 --provider insforge --workflow cli-skills --module db --capability <category> --task <task_id> --run-id <run_id> npm run benchmark:k -- --k 1 --provider insforge --workflow mcp --module db --capability <category> --task <task_id> --run-id <run_id> npm run benchmark:k -- --k 1 --provider supabase --workflow cli-skills --module db --capability <category> --task <task_id> --run-id <run_id> npm run benchmark:k -- --k 1 --provider supabase --workflow mcp --module db --capability <category> --task <task_id> --run-id <run_id> npm run benchmark:k -- --k 1 --provider supabase --workflow mcp-skills --module db --capability <category> --task <task_id> --run-id <run_id> ``` Run workflows one at a time for large `k` experiments unless the user explicitly wants a matrix shortcut. This keeps memory use predictable and makes interrupted runs easier to resume. ## Full K-Run Workflow For a finalized category: 1. Run `k=3` for each selected workflow, one workflow at a time. 2. After each workflow, scan failures immediately. 3. If a result is invalid because of harness/verifier/sandbox problems, fix the issue and rerun only the affected task/workflow/attempt set. 4. Keep the latest valid result set and remove stale invalid runs when the user asks for cleanup. 5. Generate or update a concise report with overall results, task-level reliability, time, token, cache-token rate when available, and failure notes. For the current RLS comparison, the common selected workflows are: - InsForge CLI+Skills - InsForge MCP - Supabase CLI+Skills - Supabase MCP - Supabase MCP+Skills Do not assume this list applies to every category; confirm the intended comparison surface from the current request and repo state. ## Report Shape Write reports like a benchmark report, not a work log. Include: - scope and date - workflow table with pass@k, pass^k, average time, P95 time, average tokens, and cache-token rate when available - task-level table sorted by difficulty and task id - concise failure-pattern analysis - task summary table with goal and primary verifier expectation - next steps grounded in observed failures Avoid: - run IDs unless the user asks for raw reproducibility details - replacement-task/process history unless it affects interpretation - compliance internals that require long explanation in public-facing reports - personal correction notes or chat-style apologies