name	paperharness
description	Use this skill to generate an agent-operable reproduce kit from a research paper and its GitHub repository using PaperHarness.

PaperHarness

Use this skill when the user provides a research paper and a repository and asks to reproduce, audit, inspect, plan, or evaluate experiments.

PaperHarness does not reproduce papers. PaperHarness generates the harness that lets agents reproduce papers responsibly.

What This Skill Does

This skill teaches the agent how to use the PaperHarness generator.

PaperHarness takes:

a paper PDF, markdown, text file, or extracted arXiv text
a local repository path or GitHub repository URL

and generates:

reprokit.yaml
a paper/repo-specific skill/SKILL.md
a generated repro CLI
experiment_plan.md
report_template.md
missing_info.md
reference summaries and experiment maps

Required Workflow

Check whether PaperHarness is installed:
```
paperharness --help
```
From a source checkout the equivalent is PYTHONPATH=src python -m paperharness.cli --help.
If the user gave a remote GitHub URL and remote cloning is not yet available in the installed version, clone the repository into a local working directory first.
Extract paper facts (preferred for PDFs). Run the deterministic prep step:
```
paperharness extract-text <paper-path> --out paper-extract
```
This produces in paper-extract/:
- paper.md — markdown via Marker if paperharness[parse] is installed, otherwise raw PyMuPDF text. For .txt / .md inputs the original text is copied through.
- paper.sections.json — GROBID-detected title/abstract/authors/sections, only if --grobid-url or $GROBID_URL is set.
- extraction_prompt.md — a prompt instructing you (the calling agent) to emit a strict JSON object describing the paper's facts.
- extraction_schema.json — the JSON schema the output must validate against.
Fill the extraction JSON yourself, using your own model. Read paper-extract/extraction_prompt.md, then emit one JSON object conforming to extraction_schema.json. Save it as paper-extract/extraction.json. Rules:
- Every results[*].source_span must be a verbatim quote from paper.md.
- Use confidence <= 0.4 for results you infer rather than read directly.
- Use null for fields the paper does not state; never invent values.
- Datasets and metrics must be named exactly as the paper writes them.
Build the reproduce kit with the extraction:
```
paperharness build \
  --paper <paper-path> \
  --repo <repo-path> \
  --extraction paper-extract/extraction.json
```
If you skip the extraction step, paperharness build falls back to its regex extractor; the generated kit will mark extraction_method: regex and may report No paper result candidates found. as missing info. That is the correct behavior — uncertainty is preferable to fabrication.

Validate the generated kit:

paperharness validate paperharness-output

Inspect the generated paper/repo-specific skill:

cd paperharness-output/skill
python scripts/repro.py inspect
python scripts/repro.py list

Run smoke tests before any full experiment:
```
python scripts/repro.py smoke
```
Do not run full training unless the user explicitly asks for it.

Critical Rules

Do not assume paper symbols and code variables have the same names.
Do not invent mappings for missing paper variables, hyperparameters, datasets, checkpoints, or seeds.
Do not rewrite target repository code just because an experiment is missing.
Do not create duplicate helper scripts when the target repository already has commands or scripts.
Treat symbol_mappings as hypotheses, not proof.
Treat implementation_status: partial_candidate as a warning that the released repository may only implement part of the paper experiment.
Treat implementation_status: missing as a blocker for full reproduction.
Prefer partial reproducibility reports over silent failure.

Reporting Requirements

Every report must include:

exact commands used
detected repo entrypoints
paper results and expected metrics
generated experiment IDs
paper symbol to code variable mappings
unmatched paper symbols
missing datasets, checkpoints, seeds, and hyperparameters
implementation gaps
logs and artifact paths
whether each result is mapped, runnable, blocked, failed, evaluated, or reported

Output Handoff

After generating a kit, the generated paperharness-output/skill/SKILL.md becomes the source of truth for operating that specific paper/repository reproduction workflow.

PaperHarness

Use this skill when the user provides a research paper and a repository and asks to reproduce, audit, inspect, plan, or evaluate experiments.

PaperHarness does not reproduce papers. PaperHarness generates the harness that lets agents reproduce papers responsibly.

What This Skill Does

This skill teaches the agent how to use the PaperHarness generator.

PaperHarness takes:

a paper PDF, markdown, text file, or extracted arXiv text

a local repository path or GitHub repository URL

and generates:

reprokit.yaml

a paper/repo-specific skill/SKILL.md

a generated repro CLI

experiment_plan.md

report_template.md

missing_info.md

reference summaries and experiment maps

Required Workflow

Check whether PaperHarness is installed:

paperharness --help

From a source checkout the equivalent is PYTHONPATH=src python -m paperharness.cli --help.

If the user gave a remote GitHub URL and remote cloning is not yet available in the installed version, clone the repository into a local working directory first.

Extract paper facts (preferred for PDFs). Run the deterministic prep step:

paperharness extract-text <paper-path> --out paper-extract

This produces in paper-extract/:

paper.md — markdown via Marker if paperharness[parse] is installed, otherwise raw PyMuPDF text. For .txt / .md inputs the original text is copied through.
paper.sections.json — GROBID-detected title/abstract/authors/sections, only if --grobid-url or $GROBID_URL is set.
extraction_prompt.md — a prompt instructing you (the calling agent) to emit a strict JSON object describing the paper's facts.
extraction_schema.json — the JSON schema the output must validate against.

Fill the extraction JSON yourself, using your own model. Read paper-extract/extraction_prompt.md, then emit one JSON object conforming to extraction_schema.json. Save it as paper-extract/extraction.json. Rules:

Every results[*].source_span must be a verbatim quote from paper.md.
Use confidence <= 0.4 for results you infer rather than read directly.
Use null for fields the paper does not state; never invent values.
Datasets and metrics must be named exactly as the paper writes them.

Build the reproduce kit with the extraction:

paperharness build \
  --paper <paper-path> \
  --repo <repo-path> \
  --extraction paper-extract/extraction.json

If you skip the extraction step, paperharness build falls back to its regex extractor; the generated kit will mark extraction_method: regex and may report No paper result candidates found. as missing info. That is the correct behavior — uncertainty is preferable to fabrication.

Validate the generated kit:

paperharness validate paperharness-output

Inspect the generated paper/repo-specific skill:

cd paperharness-output/skill
python scripts/repro.py inspect
python scripts/repro.py list

Run smoke tests before any full experiment:

python scripts/repro.py smoke

Do not run full training unless the user explicitly asks for it.

Critical Rules

Do not assume paper symbols and code variables have the same names.

Do not invent mappings for missing paper variables, hyperparameters, datasets, checkpoints, or seeds.

Do not rewrite target repository code just because an experiment is missing.

Do not create duplicate helper scripts when the target repository already has commands or scripts.

Treat symbol_mappings as hypotheses, not proof.

Treat implementation_status: partial_candidate as a warning that the released repository may only implement part of the paper experiment.

Treat implementation_status: missing as a blocker for full reproduction.

Prefer partial reproducibility reports over silent failure.

Reporting Requirements

Every report must include:

exact commands used

detected repo entrypoints

paper results and expected metrics

generated experiment IDs

paper symbol to code variable mappings

unmatched paper symbols

missing datasets, checkpoints, seeds, and hyperparameters

implementation gaps

logs and artifact paths

whether each result is mapped, runnable, blocked, failed, evaluated, or reported

Output Handoff

After generating a kit, the generated paperharness-output/skill/SKILL.md becomes the source of truth for operating that specific paper/repository reproduction workflow.