eval-author-task-create

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> <!-- SPDX-License-Identifier: Apache-2.0 --> # Eval Author: create task ## Trace source mode For a user-requested task from an ATIF, MLflow, NeMo Gym or normalized trace, use the shared [trace source handoff](../eval-author/references/trace-input.md): privacy review, candidate admission, environment preparation, construction and native qualification. This source mode does not require an audit gap and does not claim coverage closure. Skip the audit selector and gap-only scaffold/acceptance steps; use the selected provider's native task initialization and the handoff's qualification procedure. For audit-gap requests, retain Steps 1–8 and their measured eligibility rules. Proposal-only requests still stop before construction. ## Purpose Read `eval-author` for the shared evidence standard and boundaries. Read `eval-author-audit` for measurement and aggregation. Start with the proposal step to turn audit evidence into prioritized dataset improvements across tools, capabilities, and failure cases. For proposal-only requests, stop after Step 1. Shared lifecycle decisions and pending work belong only in `.eval-author/.working/journey-state.json`; recommendations remain a user-facing deliverable and native evidence remains immutable. Enter that step directly from the core's proposal route, carrying existing audit findings and prior answers. Accepting the audit's offer to propose tasks also enters Step 1 directly; reuse the reviewed Audit coverage report and agreed priorities. When handing off from a guided audit, wait for an explicit answer accepting that offer; report acceptance alone is not permission to begin proposals. For task creation, consume the core's shared prerequisite result, including direct invocation. Proposal-only requests keep their supplied-input scope. Reuse the confirmed Ethos and prerequisites. A general expansion request follows the core's expansion handoff; this skill creates tasks from actionable audit tool gaps. If required audit inputs are missing, obtain those inputs through the audit flow and return to proposals; do not restart first-eval onboarding or infer permission to create or run tasks. For task-creation requests, continue with an eligible tool gap through the existing execution path: ```text actionable uncovered tool → NeMo Gym-native or Harbor-native draft → native positive and negative controls → repeated real-agent traces → target tool covered in every report ``` Create and prove one tool gap at a time. Keep every generated artifact under `.eval-author/`; do not edit existing tasks or customer source. For an existing suite without sufficient audit findings, preserve the requested outcome: audit or proposal work obtains its missing coverage inputs through `eval-author-audit`; readiness work belongs in `eval-author-discover`. A missing report alone is not a first-eval request. A reviewed specification and Markdown report can support proposal-only work without measurement JSON: label such ideas as unmeasured candidates and carry the measurement limitation forward. Do not repeat an already-deferred measurement just to propose candidates. Automatic tool-gap task creation still requires the aggregate JSON report. ## Available Scripts Run `uv run <skill_dir>/scripts/task_pipeline.py <command>`; its inline metadata declares PyYAML and jsonschema for audit binding verification, including in a customer repository without those packages. Run `task_evidence.py` with the selected runtime's Python; Harbor preparation imports Harbor. | Script | Purpose | Arguments | |---|---|---| | `scripts/task_evidence.py` | Record current-revision native controls and agent runs | `prepare --task-dir --contract --out`; `run --manifest --case --run-id --result --out [--trace] -- <command>` | | `scripts/candidate_admission.py` | Check trace candidacy, privacy and tool-access gates | `--task-dir [--out <private-receipt>]` | | `scripts/task_evidence.py workspace` | Retain trace-workspace evidence and finalize/check its status | `<probe\|record-run-inputs\|record-validation\|record-repair\|finalize\|check> --task-dir`; use `<command> --help` | | `scripts/task_pipeline.py` | Select, scaffold, and verify one audit-gap task | `select --report`; `scaffold --report --target` plus task metadata; `verify --before --after --target --evidence` | The three deterministic commands have these verdicts: | Command | Verdict | |---|---| | `select` | Lists only tool items with `reason: not_covered_by_any_input_report` and emits a deterministic `task_slug` plus artifact paths | | `scaffold` | Calls NeMo Gym's native scaffolder with `--provider gym` or Harbor's `harbor task init`, requires matching draft/proposal names for that slug, and installs the supplied instruction | | `verify` | Exits 0 only with current-revision control and agent receipts, matching task/trace identities, and tool coverage in both distinct agent runs | The script prints one JSON object. Exit code 0 is success; do not replace its verdict with model judgment. ## Step 1: propose dataset improvements Apply [private construction declarations](../eval-author/references/construction-plan.md) to each proposal: name grading authority, copied versus authored starting inputs, and the intended requirement-to-check mapping. Retain unresolved authority as an open design question; proposal-only work still stops here without scaffolding. Use the shared [Coverage planning](../eval-author/references/coverage-planning.md) procedure for source scope, priorities, and proposed example breadth. Reuse the audit's confirmed corpus and exclusions when inspecting evidence; a discovered path is not permission to include another corpus. Ask about an unresolved source before searching or reading it. Proposal-only work can use the available reviewed findings without requiring new traces or a creation-scope checkpoint. When available, read the **Audit coverage report** (`.eval-author/audit-coverage-report.md`) for test mappings, findings, limitations, and agreed next actions. Check its applicability against the underlying specification and reports; a mapped test does not establish measured coverage. The Markdown report supplies context, while measurement JSON remains the input for deterministic tool-gap selection and verification. Its absence alone does not require repeating an otherwise usable audit. Keep the audit's [artifact names](../eval-author-audit/references/coverage-report.md#artifact-names) in proposal replies and links: Audit specification (`audit.md`), Audit coverage report (`audit-coverage-report.md`), and Coverage measurements (JSON) (`audit-coverage-report.json`). Review status does not change these names. Read available aggregate measurements, relevant per-trace `details.json` and capability judgments, and the selected source traces or verifier results needed to explain the findings. When measurement is unavailable, ground candidate proposals in the reviewed specification and inspected eval definitions instead. Check existing task instructions, fixtures, and verifiers before claiming a scenario is absent or proposing a duplicate. A capability covered in one run can still fail in another; review observed failures even when the item is absent from `uncovered_items`. Distinguish the basis for each recommendation: | Basis | What it supports | |---|---| | Observed failure | A trace or verifier shows incorrect behavior on an exercised scenario. Distinguish agent failure from verifier false acceptance/rejection or ungraded execution before recommending an agent fix. Preserve a genuine failing task as a regression; propose verifier repair when the evidence identifies a grading defect. | | Coverage gap in inspected inputs | A measured item is not demonstrated and the inspected tasks or traces lack the intended scenario. Propose a concrete new task or extension; state the inspected scope rather than claiming the entire dataset lacks coverage. | | Unmeasured or insufficient evidence | The method was not selected or is unsupported, judgments are missing or unclear, or the scenario's trigger is unverified. Recommend measurement or inspection first. Ethos-backed scenarios may still be proposed as candidates, with their unmeasured status explicit. | `not_covered_by_any_input_report` alone does not distinguish an absent scenario, an agent failure, or missing judgments. `not_measured_by_any_method` is a measurement limitation, not proof of a dataset deficiency. Failure-case coverage requires the audit skill's `failure_cases` measurement; manual observations alone do not change measurement status. Failure cases remain ineligible for automatic tool-gap generation. Rank by expected value and strength of evidence, explaining why the first action comes first. Apply established user priorities, including cost when relevant, and distinguish them from suggested priorities still awaiting review. Prefer a short, useful list over one suggestion per uncovered item. A written recommendation is not a generated, validated, or accepted NeMo Gym or Harbor task. Expand only the recommendation the user selects, using the [report workflow](../eval-author-audit/references/coverage-report.md#findings-limitations-and-next-actions). Fill its `details` in the authored report content, then revalidate, regenerate the Markdown, and retain its input bindings for renewed review. Keep generated Markdown and measurement JSON read-only. Unselected recommendations keep their one-sentence `proposed_task`. Author these fields: - **Change and scenario:** the recommendation's `proposed_task`. Say whether to add a task, strengthen a named task, retain a regression, or gather evidence, and describe the concrete request and the fixture or failure trigger. - **Verification:** `details.verification`. The outcome to check, how a verifier distinguishes correct from incorrect behavior, and any unresolved trust boundary. For failure cases, identify the trigger evidence and the expected response. - **Breadth:** `details.breadth`. Distinct inputs, behavior variations, difficulty, positive and negative scenarios, exclusions, and cost assumptions. Label counts and tradeoffs provisional until agreed. Repeated runs are not new examples. - **Next step:** `details.next_step`: `auto_tool_gap` when automatic tool-gap task creation applies, `manual_design` when a person must design the task, or `more_measurement` when evidence comes first. Do not author these in the report content: - **Basis and evidence.** They already live on the finding. Mark proposed fixture details as proposals, not observed facts. - **The Verifier link.** The renderer links each affected task's README Verifier section once it exists. Document the applicable [simple-cheat controls](../eval-author/references/task-validation.md#screen-simple-cheats) and expected rejection only in that README section while authoring the task. - **NeMo Gym version and rerun plan.** Put the [version and rerun plan](references/gym-tasks.md#version-and-rerun-plan) in the task README and run configuration: exact pins supported by the available evidence, unresolved versions, and the settings and reset procedure. The report links to the task instead of repeating it. Carry the selected plan into task creation; a proposed pin is not a verified lock. Lead the proposal response with the highest-value recommendations and enough scenario and expected-behavior detail to act on them. Follow with supporting coverage counts, measurement limits, and links to the recommendation entries in the Audit coverage report. Full tool coverage or an unsupported task-generation path must not suppress useful capability or failure-case suggestions. Do not relabel those suggestions as tool gaps to pass the selector. If evidence supports no dataset change, say why and identify any useful measurement or agent-fix action instead of inventing additions. Do not scaffold or run tasks for a proposal-only request. ## Step 2: select one actionable gap ```bash uv run <skill_dir>/scripts/task_pipeline.py select \ --report .eval-author/audit-coverage-report.json ``` Choose one item from `actionable_tools`. Stop task creation when the list is empty, and surface the dataset recommendations from Step 1. An empty selector means no eligible tool gaps, not that there are no useful dataset improvements. Capability and failure-case items are not task-generation inputs in v1, even when capability coverage was measured. Each actionable tool includes a deterministic `task_slug` of the form `cover-<tool-name>` and a `paths` object for the proposal, draft, and measurement directories. Use those paths verbatim for the rest of this flow. Do not invent alternate slugs or filenames. ## Step 3: design the smallest objective task After the scope checkpoint below and draft creation, materialize the private [construction plan](../eval-author/references/construction-plan.md), validate its input hashes and native check mappings, and bind it when preparing run evidence. Declare prerequisites only when they matter to the task; plan a bypass control that targets the dependent check. A declaration is not proof of necessity. Before generating task instructions, fixtures, or scaffolds, use the shared [coverage-planning checkpoint](../eval-author/references/coverage-planning.md). Use the core's [provider-selection rule](../eval-author/SKILL.md#select-the-evaluation-provider) to resolve NeMo Gym or Harbor when mapping scope to execution counts. NeMo Gym repeats selected dataset rows, while Harbor repeats tasks. Keep estimates conditional if that choice remains open. Present the proposed priorities and breadth for review, including an option to start with a minimal pilot and expand. Reuse already agreed scope rather than asking again. Save the agreement in `.eval-author/.working/journey-state.json`: selected corpus and exclusions, priorities and cost constraints, distinct input-example counts per behavior, variations and difficulty, positive and negative scenarios, expected outcomes, and pilot or expansion status. Reflect the agreed scope in the selected recommendation in the Audit coverage report. Keep this planning record out of the agent-facing instruction file. Answer questions or complete scoped detours, then resume this checkpoint with the existing decisions intact. Explain how the agreement maps to the selected tool gap. The deterministic pipeline creates one tool-gap draft at a time; repeated runs test closure on the selected inputs and do not add distinct input examples. Keep any agreed examples beyond the current draft visible as pending work. A capability or failure-case proposal outside the generator's support remains an explicit gap requiring manual design, not a completed example or a relabeled tool gap. If the reviewed priorities change the target, return to Step 2 before generation. Read the selected item's `description`, `focus`, `needed_tools`, and `evidence_required`. Read one nearby task for domain conventions only. Do not copy its directory: a sibling can carry an obsolete Harbor schema, placeholder verifier, or unrelated solution. Write the instruction only to `paths.proposal` from Step 2. State the observable goal, paths, and constraints without naming the target tool or leaking verifier logic. The task should naturally require the selected tool and no unrelated capability. Decide the verifier before scaffolding. Prefer deterministic shell or pytest. The verifier must grade the task outcome, not the tool call; trace measurement proves tool coverage separately. ## Step 4: scaffold with NeMo Gym or Harbor Select the provider from the user's request and existing suite. For **NeMo Gym**, follow [Author and prove a NeMo Gym evaluation](references/gym-tasks.md) for native scaffolding, verifier controls, and real-agent runs. Keep Steps 1–3's selected gap and instruction, then return to the shared coverage verification steps. Do not apply Harbor file layouts or Oracle CLI flags to NeMo Gym. For either provider, build the task's environment with [`eval-author-environment`](../eval-author-environment/SKILL.md) and pass its smoke task before writing the verifier, reusing the agent's existing proven environment kit when one exists, and derive expected values with an independent reference query over the kit's data. For **Harbor**, use the following native scaffold: ```bash uv run <skill_dir>/scripts/task_pipeline.py scaffold \ --report .eval-author/audit-coverage-report.json \ --target <tool-name> \ --out .eval-author/tasks/<task-slug> \ --task-name <org>/<task-slug> \ --description "<one-line description>" \ --author "<author>" \ --instruction-file .eval-author/.working/proposals/<task-slug>-instruction.md ``` Use the Step 2 `task_slug` for `<task-slug>` in every path above. `scaffold` rejects mismatched draft, task-name, or proposal filenames. For either provider, record the scaffold as **Draft** using [Task status](../eval-author/references/shared-prerequisites.md#task-status) before refreshing the report. Then complete Harbor's generated files: - `environment/`: a copy of the agent's environment kit plus this task's own records, buildable on its own rather than `FROM` a local kit image, and never the solution. - `tests/test.sh`: deterministic reward writer using absolute paths. - `solution/solve.sh`: executable Oracle solution. - `task.toml`: nonempty keywords, metadata, realistic timeouts and resources. - `README.md`: purpose, behavior under test, expected result, environment, verifier, layout, and persistent suite and individual-item run instructions. For either provider, refresh `.eval-author/report.html` and the marked report section of `.eval-author/README.md` after authoring and after controls or actual-agent runs using [Report generation](../eval-author/references/suite-readme.md#report-generation). Finish the task README, including its [Verifier section](../eval-author/references/suite-readme.md#task-documentation) and setup/run instructions, before preparing validation evidence. Maintain later results and rerun summaries in the suite README outside generated markers. Reporting failure does not change acceptance or execution evidence. For either provider, maintain the generated suite's top-level review and rerun guide using [Review and rerun guidance](../eval-author/references/suite-readme.md#review-and-rerun-guidance). For a standalone task, use `.eval-author/README.md` and link its finalized task README; when adding it to a generated collection, refresh the shared report and maintain the collection guide, preserving the existing coverage mapping. Present and link the case inventory when the cases are authored, with execution pending, then refresh it after native controls and actual-agent runs with their available results and execution traces. Do not leave generated placeholders, `pass`, unconditional reward 1, or empty keywords. ## Step 5: prove task correctness with native controls Confirm the environment kit's proof passed first; a proof failure is an environment defect to fix, not a task result, and that proof stays in the kit's plan rather than among the task's receipts. The task's own controls never count as environment proof. Apply [Task validation and execution evidence](../eval-author/references/task-validation.md). Prepare the result contract and revision manifest before execution. Record each native control and each agent attempt as its own `scripts/task_evidence.py run` invocation, with fresh native results and receipts outside the task tree. Harbor requires a reference, a realistic incorrect control, and an empty/no-op control; include applicable valid-alternative and side-effect cases. A failing command or missing evidence is not a passing negative control. Use the same Harbor Python runtime for preparation and runs. Follow [Execution recovery](../eval-author/references/execution-recovery.md) for native controls, real-agent trials, and reporting. Compatibility repairs must preserve the selected provider and original grading semantics. For NeMo Gym, use the positive and negative controls and native execution in the [NeMo Gym guide](references/gym-tasks.md). For Harbor, record the Oracle reference before spending model credentials. `harbor trial start` writes `result.json` and `agent/trajectory.json` under its `--trials-dir` and `--trial-name`, giving the recorder fixed paths; a Harbor job names its trials randomly: ```bash TRIALS="$PWD/.eval-author/jobs/<task-slug>" <harbor_python> <skill_dir>/scripts/task_evidence.py run \ --manifest .eval-author/task-measurements/<task-slug>/revision-1.json \ --case reference --run-id reference-1 \ --result "$TRIALS/reference-1/result.json" \ --out .eval-author/task-measurements/<task-slug>/reference-1.json \ -- harbor trial start -p .eval-author/tasks/<task-slug> -a oracle \ --trial-name reference-1 --trials-dir "$TRIALS" ``` Continue only when the receipt reports `passed`, with no exception and reward 1.0. Record the incorrect and empty controls separately the same way, choosing their agents as described in [Review the verifier before running](../eval-author/references/task-validation.md#review-the-verifier-before-running). Fix the task, solution, or verifier when Oracle fails; do not weaken the verifier merely to make it pass. ## Step 6: run the real agent twice Running a model spends credentials. Do it only when the user asked for the run or approved it. Use the repository's proven agent configuration and point it at the draft. Record each attempt as its own invocation with fresh native output: one NeMo Gym repeat per recorded run, or one Harbor trial per run with a distinct trial name. One job with two attempts cannot back two receipts. Keep all outputs under `.eval-author/`: ```bash <harbor_python> <skill_dir>/scripts/task_evidence.py run \ --manifest .eval-author/task-measurements/<task-slug>/revision-1.json \ --case agent --run-id repeat-1 \ --result "$TRIALS/repeat-1/result.json" \ --trace "$TRIALS/repeat-1/agent/trajectory.json" \ --out .eval-author/task-measurements/<task-slug>/agent-1.json \ -- harbor trial start -p .eval-author/tasks/<task-slug> \ -a <agent> -m <model> --trial-name repeat-1 --trials-dir "$TRIALS" ``` Repeat with `repeat-2` and `agent-2.json`. Carry a proven Harbor job's agent settings into the equivalent trial options or a trial `--config`. Require both trials to: 1. finish without an exception, 2. receive the intended verifier reward, and