| name | recent-conjecture-evaluations |
| description | Run source-grounded Jacobian reliability evaluations using recently resolved conjectures as held-out probes. Use for source selection and deduplication, exact input-bound oracles, current-main math.find/math.run contract audits, frozen control/treatment comparisons, observable trajectory scoring, failure attribution, and independent review of another evaluation. Do not use merely to solve a conjecture or create benchmark-specific mathematical helpers. |
Recent Conjecture Evaluations
Treat conjectures as probes. The objective is to improve Jacobian from discovery
through final-answer use, not to accumulate solved examples.
This skill owns source selection, deterministic operation audits, optional
control/treatment evaluation, attribution, and repository-action decisions. It
does not create Harbor tasks. Use harbor-benchmarks and
verifier-evaluations when a selected case is later packaged as a benchmark.
Select a mode
- Probe: run one new source cycle.
- Review: independently audit a completed cycle without repeating model calls.
- Coordinate: check reservations, duplicates, and ownership before another evaluator proceeds.
Run one cycle by default. Continue looping only when the operator explicitly
requests it. A loop does not authorize unlimited model calls.
Preserve these invariants
- Prefer primary sources and record the precise theorem status and date.
- Bind every oracle, prompt, suite, and tool payload to the intended input.
- Reconstruct the gold result independently of evaluated model arms.
- Audit current main deterministically before any model call.
- Keep
COMPUTED, VERIFIED, source-imported, and unproved claims separate.
- Treat missing operation, timeout, cancellation, transport failure, and failure to find evidence as non-conclusions.
- Never attribute a model-authored fallback error to Jacobian when no Jacobian result produced it.
- Never weaken verification, artifact binding, scope, or assurance to make a case pass.
- Never merge a PR. Open only a draft PR when the localized-fix gate passes.
- Never create a benchmark-specific operation from one conjecture family.
Run the workflow
1. Inventory before research
Search saved reports, suites, trajectories, issues, PRs of every state,
branches, and active reservations. Search in layers: exact identifier/title,
artifact, mathematical family, required operation/provider, and suspected root
cause. One exact-source miss does not establish independence.
Run the local helper with positional arguments, then search GitHub separately:
python .agents/skills/recent-conjecture-evaluations/scripts/search_inventory.py \
"source or root-cause phrase" outputs benchmarks/results
Classify the source as selected, rejected, reserved elsewhere, or completed
duplicate. Read source-gating.md.
2. Establish the source gate
Read enough of the primary source to identify the prior conjecture, exact
resolution status, proof-carrying theorem or construction, finite obligations,
and imported boundaries.
Reject before model calls when the bounded task is illustrative, leaks the
answer, tests literature recall, requires private data, cannot represent the
decisive obligation faithfully, duplicates prior work, or only exercises a
near-match.
3. Build the independent oracle
Reconstruct every finite obligation with a clean-room script or exact
derivation. Hash the canonical input. Stop if it disagrees with the primary
source. Label author-supplied code as SOURCE-SUPPLIED REPLAY; reserve
INDEPENDENT ORACLE for separately implemented or derived evidence.
Record acceptable alternative witnesses and representations. Define what
remains imported or uncertain.
4. Audit current main
On a clean current-main worktree, preflight installed providers, authorized
checkers, input bounds, and estimated payload/runtime before materializing a
large artifact. Stop early when the decisive operation is unavailable and the
boundary is already documented.
Then:
- Search with natural task language.
- Treat bounded
math.find search as candidate retrieval, not complete inventory;
use operation://catalog when full installed membership matters.
- Inspect returned cards and exact IDs for schemas, examples, domains, bounds,
and provider availability. Treat these as operation facts, not workflow advice.
- Inspect the producer and independent checker as separate operation contracts.
A checker verdict must bind the exact claim and semantics; do not expect a
generic verification record or artifact URI from an ordinary operation.
- Execute the frozen input directly.
- Run the independent checker when installed.
- Replay relevant malformed, wrong-input, over-bound, and unsupported-domain cases.
- Run focused deterministic tests.
Persist complete machine-readable discovery and invocation outputs, including
failures, before summarizing them. Hash the input, probe, raw output, and
report. Never support an artifact URI, elapsed time, ranking, or diagnostic
only with terminal scrollback.
If this phase identifies the root cause, do not run model arms merely to
demonstrate it again.
5. Decide whether model arms have information value
Read evaluation-and-scoring.md. Run
paired arms only if every gate passes. Otherwise record a deterministic-only
cycle. Freeze the suite and its digest before execution; use the repository's
evaluation harness to validate the applicable suite shape.
Use fresh isolated contexts and identical settings except for Jacobian
availability. The control must not see Jacobian tools, skills, catalog content,
or routing hints. The treatment exposes the public surface only: the agent
still decides whether discovery is useful, which operation to call, and when to
stop.
6. Score observable evidence
Score final correctness, certificate validity, input binding, source fidelity,
scope, completeness, assurance, discovery, contract inspection, execution,
recovery, stopping, elapsed time, token use, and model-visible tool bytes.
Do not claim access to hidden chain-of-thought. Score observable calls, outputs,
retries, concise visible reasoning, and final answers.
7. Attribute before acting
Assign each failure to one primary class: Jacobian implementation; tool
interface or contract; discovery or routing; evaluator or telemetry;
infrastructure or transport; model transcription or input binding; model
mathematical reasoning; unsupported operation; or no issue.
Reproduce suspected Jacobian failures directly on current main and search
ownership again. Read action-policy.md before
opening an issue or draft PR.
8. Publish the cycle record
Write a persistent report using report-schema.md.
Include rejected and deterministic-only cycles because they prevent duplicate
work. Update the shared reservation ledger. End with a distinct next source
direction, not an unsupported claim that the search space is exhausted.
Manage cost and stalls
- Prefer deterministic checks.
- Require explicit model-call authorization and cost boundaries.
- Checkpoint after source gate, oracle, current-main replay, each model arm, scoring, and repository action.
- Inspect any phase that produces no checkpoint for 30 minutes.
- Do not retry a cancelled call without new diagnostic evidence.
- Stop collecting examples for an owned root cause once the agreed independence threshold is met.
- Do not use a weaker model merely to manufacture errors; use it only in a frozen comparison with a falsifiable question.
