NatureBench

Acquire and verify data for CNS papers that passed paper-filter. Clones repositories, downloads and organizes datasets per evaluation setting, performs Level 4 deep verification (acquisition integrity, initial state, separability, consistency, per-setting completeness), creates an organized data environment. Updates filter_result.json with verification results, acquisition details, and task_info corrections.

Verify data components in filter_result.json after data-check. Checks Algorithm A boundary, evaluation setting validity, separability correctness, data directory integrity, and description consistency. Outputs a verification report for human review — does not modify filter_result.json or data directories.

Fix Dockerfile issues identified by batch_verify.sh. Reads verify_result.txt to diagnose import failures, version conflicts, and missing dependencies, then repairs the Dockerfile and updates packages.json. May also fix scripts (Script-First Rule) when packages are unavailable.

Verify paper-filter results by checking pass/reject judgment correctness, validating task_info field accuracy, and detecting internal inconsistencies. Outputs actionable corrections to filter_result.json.

Filter CNS papers for suitability in extracting machine learning tasks. Uses a three-level cascade funnel (Task Nature → Evaluation System → Data Completeness) with progressive filtering; stops immediately upon rejection. Input is preprocessed paper data; output is the filtering decision result.

Preprocessing CNS papers (PDF + HTML) to extract text, figures, and links. Outputs structured data for use by the paper-filter skill.

Build a structured task package from a paper that has passed paper-preprocess, paper-filter, and data-check. Reorganizes data, generates problem description, evaluator, metadata, and environment Dockerfile.v3.

Fix issues identified by task-verify in a task package. Reads task_verify_result.json, prioritizes failed checks over warnings, applies targeted fixes following task-build rules, re-runs dynamic testing, and outputs a fix log.

Verify task packages built by task-build. Checks document consistency, benchmark design principles, and performs dynamic testing with a baseline solver to ensure the task package is valid and usable.

Apply corrections from filter-verify (verification_result.json) or data-verify (data_verify_result.json) to filter_result.json. User specifies source type explicitly. Applies field-level and structural corrections, performs inline verification for new/modified settings, writes change log.