criterion-synthesis

name	criterion-synthesis
description	Synthesize a task-specific executable scientific rubric from an instruction-derived skeleton, literature grounding, and task-data profile. Use in AutoSciRub when Codex needs criteria with linked goals, data sources, required analyses, metrics, comparisons, expected artifacts, provenance, and satisfaction conditions.

Criterion Synthesis

Purpose

Combine instruction goals, scientific practice, and task-data feasibility into an executable rubric. This implements phi_syn(G_i, K_i, P_i) from AutoSciRub.

The rubric is the contract for execution, verification, and revision.

All .autoscirub/ paths below refer to the state directory resolved by the controller. When invoked alone, use the user’s override, then AUTOSCIRUB_STATE_DIR, then project config state_dir, then .autoscirub/.

Inputs

Read:

1. .autoscirub/rubric_skeleton.json
2. .autoscirub/literature_grounding.json
3. .autoscirub/task_data_profile.json
4. optional task config/profile

Output

Write .autoscirub/executable_rubric.json:

{
  "schema_version": "1.0",
  "goals": [
    {
      "goal_id": "G1",
      "title": "...",
      "requirement": "..."
    }
  ],
  "criteria": [
    {
      "criterion_id": "C1",
      "goal_ids": ["G1"],
      "requirement": "Concrete requirement to satisfy.",
      "data_sources": ["relative/path or dataset field"],
      "required_analysis": ["experiment, computation, comparison, proof step, or diagnostic"],
      "metrics": ["metric, statistic, threshold, or qualitative check"],
      "comparisons": ["baseline, control, condition, region, time, model, or ablation"],
      "expected_artifacts": [
        {
          "type": "figure | table | number | file | code | text_analysis",
          "name": "semantic artifact name",
          "expected_path": "optional relative path",
          "must_show": "what the artifact must demonstrate"
        }
      ],
      "satisfaction_condition": "Observable condition for marking the criterion satisfied.",
      "priority": "high | medium | low",
      "provenance": {
        "instruction": ["goal or instruction evidence"],
        "literature": ["source id or summarized guidance"],
        "data_profile": ["supporting files or feasibility notes"]
      }
    }
  ],
  "claims_to_avoid": ["unsupported claim pattern"]
}

Procedure

1. Preserve every instruction-derived goal. If required inputs are missing, retain the goal and record the blocker; do not silently remove the user’s requirement.
2. Select feasible analyses that best satisfy each goal using literature-grounded practice.
3. Convert each requirement into a criterion with observable evidence and a satisfaction condition.
4. Link each criterion to goals, data sources, artifacts, and provenance.
5. Record unsupported or risky claims that the execution agent should avoid.

Rules

• Prefer criteria that can be checked against code, results, figures, tables, files, or report text.
• Use direct task data when available; only allow surrogate analyses when the data profile justifies them.
• Avoid generic criteria such as "analyze thoroughly" or "discuss results".
• Do not introduce requirements for unavailable labels, tools, or private data. If the user explicitly requires an unavailable input, retain that requirement as blocked and explain what is needed.
• Define satisfaction by correctness of the method and evidence, not agreement with a desired result. Negative or inconclusive findings can satisfy an appropriately framed research question.
• Keep the rubric compact enough to execute, without a fixed minimum number of criteria.