Bmk-dev

Select a Python repository as a SWE-E2E benchmark task candidate. Use when evaluating whether a repo qualifies as a reconstruction task: checking hard/soft gates, recording the evidence brief in filter_notes.md, and logging retired candidates in CANDIDATES.md.

Write a behavioral specification (public packet) for a SWE-E2E reconstruction benchmark task. Use when drafting, iterating, or judging a spec for a candidate Python library before candidate evaluation. The spec describes the public API behavioral contract — what the library promises to callers. Source code may be read freely; the constraint is public intent (exported surface, user-facing docstrings, examples), not documentation coverage.

Judge whether a SWE-E2E benchmark task and evaluation run are valid, diagnose model failures, and record model weaknesses. Use after a candidate evaluation run to determine task status (QUALIFIED/BROKEN/CHEAT_DETECTED), produce a diagnosis report, and update the weakness table. Core principle: every scoring test must be spec-driven (traceable to a spec section) and behavioral (checks observable outcomes, not internal structure). Apply this principle before reading any score.

Orchestrate the full SWE-E2E benchmark task synthesis pipeline: candidate selection -> spec writing -> test filtering -> evaluation -> judging. Use when constructing a new benchmark task from scratch, resuming a pipeline run, or routing feedback between stages. This is the top-level coordinator - each stage delegates to its specialist skill.

Filter and classify a Python test suite for SWE-E2E benchmark scoring. Use when processing a raw test collection to produce a filtered subset with atomic/integration/system_e2e taxonomy and spec-test coverage map. Covers Step 3 (hard filtering + classification) and Step 3.5 (spec_test_map.md + coverage gate) of the benchmark pipeline.