eval-author-audit

<!-- SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. --> <!-- SPDX-License-Identifier: Apache-2.0 --> # Eval Author: audit ## Purpose Read the [eval-author core skill](../eval-author/SKILL.md) for the shared standard, vocabulary, and boundaries. This sub-flow generates and validates a finite coverage denominator from `<ethos_path>` and reviewed audit items, then can measure ATIF traces and aggregate coverage reports against it. It identifies coverage gaps and recommends improvements grounded in the findings and evidence limits, then writes a human-readable `audit-coverage-report.md` and reviews it with the user. Detailed task proposals and task creation remain in the separate improvement handoff. ## Start a guided audit Before classifying a full-audit request as fresh, follow the core's [Return to an audit](../eval-author/references/shared-prerequisites.md#return-to-an-audit). A full audit enters the core's shared prerequisite flow, including direct invocation. For a fresh audit, use the core's [onboarding renderer](../eval-author/references/conversations/onboarding.md) for `orientation` with journey `audit`, then end the turn. Read only this skill, the core, the shared rendering contract, and that onboarding reference as needed before the response; the core's narrow renderer and audit-entry lookup exceptions apply. Repository inspection, Ethos validation, prerequisite questions, and state writes wait for the actual readiness reply. Complete and save the core's state-only prerequisite pass before any target Ethos inspection, then reconcile artifacts and include the completed findings in the final `results` response. Reuse confirmed agent, prerequisites, and Ethos on continuation. Apply the shared flow's audit checks: local audit tooling and source access, without evaluation runner, Docker, live agent service, or credential probes. Confirm that traces from completed evaluations exist before continuing to Ethos; unknown or missing availability pauses onboarding under **Blockers**. Follow the [canonical trace blocker](../eval-author/references/pre-reqs/audit.md#traces-from-completed-evaluation-runs), supplying the exact loaded audit skill link for its renderer-owned attribution; do not append a separate pause explanation. Check conversion and trajectory-parser dependencies only after evidence selection. After fresh onboarding's completion pause and the user's continuation, or the returning-audit handoff above, follow [Guided audit milestones](references/guided-audit.md): confirm evals and traces, define intended coverage, measure selected evidence, then generate and review `audit-coverage-report.md`. Keep source and content reviews explicit. Use bounded agent and eval reads. Do not enumerate or read traces until their source and location are selected. An existing accepted source choice needs no repeated confirmation. Render downstream checkpoints through the [audit conversation contract](../eval-author/references/conversations/audit.md). Explicit generate, validate, measure, and aggregate requests keep their scope. Generation needs confirmed Ethos. Existing-spec validation, measurement, and aggregation use their supplied inputs without new onboarding. File-only and provider-independent operations do not require runtime installation. Read-only and suggestion requests remain read-only. ## Ethos prerequisite Use the confirmed result from [Ethos confirmation](../eval-author/references/ethos-confirmation.md). An explicit `--ethos <path>` overrides a prior handoff path; recheck applicability and confirmation for changed content. Before drafting audit items or generating `audit.md`, require the selected agent's confirmed intended behavior. If it is missing or changed, invoke the canonical `ethos-orchestrate` through that adapter; do not run a separate audit interview, writer, or review loop. ## Coverage specification The audit-spec approach has three item kinds in v1: | Kind | Meaning | |---|---| | `tool` | A canonical tool name the agent may call | | `capability` | A high-level behavior the agent should exercise | | `failure_case` | Expected safe behavior when a capability cannot proceed normally | Every item uses `name` as its stable coverage key. Names must be unique across the whole file; do not add sequential numeric IDs. Tool references in `required_tools`, `expected_tools`, and `evidence_required[].tool` must match the `name` of a declared `tool` item. `prohibited_tools` may name any syntactically valid tool name, including tools the agent must never call and therefore should not declare as allowed tools. Write audit artifacts under `.eval-author/`. Audit operations do not edit the customer's source, existing evals, source-of-truth documents, or `ETHOS.md`; prerequisite Ethos work belongs to the shared procedure in the pre-flight above. ## Audit outputs Follow [report presentation](references/coverage-report.md#artifact-names): only the Audit coverage report is introduced for guided review; other outputs are internal. | Name | File under `.eval-author/` | Purpose | |---|---|---| | Audit specification | `audit.md` | Structured source of checks and required evidence, reviewed through the report's Intended coverage section | | Audit coverage report | `audit-coverage-report.md` | Readable intended coverage, test mappings, measured coverage, gaps, evidence limits, and next actions | | Coverage measurements (JSON) | `audit-coverage-report.json` | Script-generated measurements aggregated over the selected runs, when available | | Journey state (internal) | `.working/journey-state.json` | Shared decisions, audit progress, and resumption | For a full audit, create the Audit coverage report with the validated Audit specification, refresh it after measurement, and update it for final review. Follow the [report guidance](references/coverage-report.md) and [template](templates/audit-coverage-report.md) even when measurement is deferred. Whenever the Markdown report is created or updated, explicitly say so and link it in the turn-ending response, including intermediate decision checkpoints. Focused operations need this report only when requested or continuing a full audit; keep their scope. Read-only and `suggest` requests remain read-only. ## Scripts Audit-spec mechanics live under `scripts/audit_spec/`: Read `scripts/audit_spec/README.md` for the current measurement assumptions: ATIF input, Harbor trajectory parsing, v1 `tool_calls`, `capabilities`, and `failure_cases` coverage, and coverage aggregation from `coverage.json` files. | Script | Use it to | |---|---| | `scripts/audit_spec/generate.py` | Create, reconcile, replace, or preview `.eval-author/audit.md` from `ETHOS.md` and reviewed item proposals | | `scripts/audit_spec/measure.py` | Measure one ATIF trace or Harbor trial directory against `audit.md` and write coverage/details files for each selected method | | `scripts/audit_spec/report.py` | Aggregate per-trace `coverage.json` files into one coverage report with uncovered audit items | | `scripts/audit_spec/validate.py` | Validate the marked audit-spec block in `audit.md` | Shared helpers, measurement method contracts, schemas, and examples are documented in `scripts/audit_spec/README.md`. Runtime dependencies are listed in `requirements.txt`. ## Step 1: Draft Or Update Audit Items Before drafting or updating `.eval-author/.working/audit-items.yaml`, read `templates/audit.md` and `schemas/audit.schema.json`. Use the template as the worked example and the JSON Schema descriptions as the field definitions. Do not use validation as the primary way to discover the format; validation is the enforcement and repair step after drafting. Read `<ethos_path>` and draft audit items at the level between Ethos and runnable tasks: canonical tools, high-level capabilities, and material failure cases. Keep the list finite. Do not create separate items for prompt paraphrases, fixture variants, or ordinary happy-path permutations. For `tool` items, use the names that appear in the actual runtime traces or tool registry, including eval-specific tools that may be more precise than product tools named in Ethos prose. If Ethos describes a generic tool such as `sqlite` but measurement traces expose `execute_sql` and `submit_sql`, declare the runtime tool names and connect capabilities or failure cases to those names. Do not invent tool names that will not appear in the measurement surface. Read traces for this purpose only after user confirmation of their source and location. Without usable traces, use an authoritative tool registry or agent configuration; keep unresolved names as open questions rather than inventing tool items. Explain any resulting limits on the proposed specification. Save the reviewed item proposals as `.eval-author/.working/audit-items.yaml`. The items file may be either a mapping with an `items` key or the item list itself. It should use the same item shape shown in `templates/audit.md` and enforced by `schemas/audit.schema.json`. For an initial audit, this file should contain the full proposed denominator. For an update, it may contain only the proposed additions or edits. Existing reviewed `audit.md` items remain the source of truth in reconcile mode. Capabilities that do not need tools, such as policy refusals or out-of-scope handling, should use `required_tools: []`. Failure cases attach to capability names through `applies_to`; tool-level failure expectations stay on the tool item as `expected_failure_behavior`. For failure cases, make the trigger and safe response explicit in `evidence_required`. Include prohibited output classes in an `output` evidence description when their absence must gate coverage. Measurement uses `prohibited_tools` as a deterministic gate; `applies_to`, `expected_tools`, `trigger`, `expected_behavior`, and `prohibited_outputs` otherwise provide the rubric and authoring context rather than separate hidden checks. ## Step 2: Generate Or Reconcile Audit.md Create or update `.eval-author/audit.md` from `<ethos_path>` and the reviewed item proposals: ```bash uv run --with pyyaml --with jsonschema \ <skill_dir>/scripts/audit_spec/generate.py \ --ethos <ethos_path> \ --items .eval-author/.working/audit-items.yaml \ --out .eval-author/audit.md ``` The default mode is `reconcile`. If `.eval-author/audit.md` does not exist, it creates the file. If it already exists, the generator parses the existing marked block, updates source metadata such as the Ethos digest, preserves existing item bodies by stable `name`, appends new proposed items, and reports proposed edits without silently rewriting them. Hand-authored prose outside the marked block is preserved. By default, the generator treats `.eval-author/.working/audit-items.yaml` as a partial update proposal. Missing existing items are not stale in that mode, because the items file may contain only additions or edits. Use `--items-mode full` only when the items file is intended to be the complete denominator; then existing items omitted from the proposal are reported as `possibly_stale_items`. Use the explicit modes when the default is not what the user wants: ```bash uv run --with pyyaml --with jsonschema \ <skill_dir>/scripts/audit_spec/generate.py \ --ethos <ethos_path> \ --items .eval-author/.working/audit-items.yaml \ --out .eval-author/audit.md \ --mode suggest uv run --with pyyaml --with jsonschema \ <skill_dir>/scripts/audit_spec/generate.py \ --ethos <ethos_path> \ --items .eval-author/.working/audit-items.yaml \ --out .eval-author/audit.md \ --mode reconcile \ --items-mode full uv run --with pyyaml --with jsonschema \ <skill_dir>/scripts/audit_spec/generate.py \ --ethos <ethos_path> \ --items .eval-author/.working/audit-items.yaml \ --out .eval-author/audit.md \ --mode replace ``` `suggest` performs the same comparison as `reconcile` but writes nothing. `replace` rewrites the whole file from the item proposal file, including prose outside the marked block, and should be used only when the user wants to discard the existing generated audit file. The generator prints a JSON summary containing `written`, `added_items`, `conflicting_items`, `conflicting_items_applied`, and `possibly_stale_items`. Treat `conflicting_items` as items where the proposal differs from the reviewed audit item; reconcile mode preserves the reviewed item and reports `conflicting_items_applied: false` so the user can accept the change manually or through a future editor. If reconcile adds items, finds conflicts, reports stale items in full mode, or detects an agent-name change, an approved audit is demoted to `status: draft` unless the user passes `--status approved`. The generator adds an optional `sources` entry for Ethos with `name: ethos`, a path relative to `audit.md`, and a real `sha256` digest. It uses the frontmatter `name` from `ETHOS.md` when `--agent` is omitted. If that inferred name differs from the existing audit's `agent`, reconcile preserves the reviewed audit name and reports `agent_change` unless the user passes `--agent` explicitly. `source_refs` are advisory provenance notes in v1; the validator preserves them but does not resolve them against `sources` until a future generator or grammar defines that reference format. ## Step 3: Validate Run validation after every generated or hand-edited audit file: ```bash uv run --with pyyaml --with jsonschema \ <skill_dir>/scripts/audit_spec/validate.py --audit .eval-author/audit.md ``` `schemas/audit.schema.json` is the canonical structural schema. The Python validator applies that schema first, then checks any source digests that are provided and cross-item references such as `required_tools` and `applies_to`. Validation proves only structure and references, not that the denominator is complete or correct. ## Step 4: Measure One ATIF Trace After validation and the guided audit's specification review (when applicable), measure one completed trial directory or one ATIF trajectory file from the user-confirmed source. If that input is missing or unusable, follow the guided audit's unavailable-evidence guidance; do not search elsewhere without the user selecting another location. Use `--measure` to select one or more comma-separated measurement methods; the default is `tool_calls`. Prefer `--trial-dir` for Harbor trials so `measure.py` stamps `subject.evidence_class` from `result.json`. Measure the **agent under test** only—no Oracle/NOP trials, synthetic control ATIF, or control runs in audit artifacts (see [coverage report guidance](references/coverage-report.md)). With `--trace`, pass `--evidence-class agent_under_test` or `--skip-evidence-class` for legacy inputs only. ```bash uv run --with-requirements <skill_dir>/requirements.txt \ <skill_dir>/scripts/audit_spec/measure.py \ --audit .eval-author/audit.md \ --trial-dir <harbor-job-dir>/<trial-dir> \ --measure tool_calls \ --out-dir .eval-author/audit-measurements ``` Harbor trial dirs normally contain `agent/trajectory.json`; pass `--evidence-class` to override stamping. `--trace` does not infer runner role from ATIF bytes alone. ```bash uv run --with-requirements <skill_dir>/requirements.txt \ <skill_dir>/scripts/audit_spec/measure.py \ --audit .eval-author/audit.md \ --trace <path-to>/trajectory.json \ --task-id <task-id> \ --run-id <run-id> \ --evidence-class agent_under_test \ --measure tool_calls \ --out-dir .eval-author/audit-measurements ``` `--measure` may be passed more than once or as CSV, for example `--measure tool_calls,capabilities,failure_cases`. The default is `tool_calls`; include the other methods when the user wants the same trace to count against capability or failure-case items. The script loads the trajectory once, then runs each selected method against the same parsed Harbor trajectory model. Unknown method names fail before the trace is loaded. When capability evidence contains non-tool kinds such as `user_intent`, `output`, `outcome`, `policy_boundary`, or `verifier`, inspect the trace and write a structured judgment file under `.eval-author/` using `schemas/audit_capability_judgments.schema.json`. Each judgment must target the capability `name` plus the zero-based `evidence_required` index, copy the evidence `kind` and `description` exactly, and judge only non-tool evidence. Do not write judgments for `tool_call`; the script measures those deterministically. Set the sidecar's required `trace_sha256` to `sha256:` followed by the lowercase SHA-256 digest of the exact ATIF trajectory file you inspected. Measurement rejects the sidecar if those trace bytes have changed or another trace is used. Use the capability description, evidence description, and concrete trace content as the rubric: mark `satisfied` only when the trace clearly demonstrates the requirement, `missing` when it clearly does not, and `unclear` when the trace is ambiguous or insufficient. Include brief rationale and supporting trace references when available. A subjective judgment can satisfy only the non-tool evidence it targets; it cannot override missing required tools or missing `tool_call` evidence. Pass the sidecar when measuring capabilities: ```bash uv run --with-requirements <skill_dir>/requirements.txt \ <skill_dir>/scripts/audit_spec/measure.py \ --audit .eval-author/audit.md \ --trace <path-to>/trajectory.json \ --task-id <task-id> \ --run-id <run-id> \ --measure capabilities \ --capability-judgments .eval-author/capability-judgments.json \ --out-dir .eval-author/audit-measurements ``` Capability coverage is conjunctive: every deterministic requirement must be satisfied, and every judged evidence requirement must be satisfied. Missing judgments leave the capability uncovered. Stale judgments fail measurement before the script writes a coverage report, including judgments bound to a different trace digest. Failure-case coverage follows the same pattern. Inspect the trace and write `schemas/audit_failure_case_judgments.schema.json`, targeting each non-tool evidence requirement by failure-case `name`, zero-based index, exact `kind`, and exact `description`. Judge only what `evidence_required` states, using the failure case's trigger, expected behavior, and prohibited outputs as context. Then measure it with: ```bash uv run --with-requirements <skill_dir>/requirements.txt \ <skill_dir>/scripts/audit_spec/measure.py \ --audit .eval-author/audit.md \ --trace <path-to>/trajectory.json \ --task-id <task-id> \ --run-id <run-id> \ --measure failure_cases \ --failure-case-judgments .eval-author/failure-case-judgments.json \ --out-dir .eval-author/audit-measurements ``` A failure case is covered only when every evidence requirement is satisfied and none of its `prohibited_tools` appears anywhere in the trace. Missing judgments leave it uncovered, and a subjective judgment cannot override missing `tool_call` evidence or an observed prohibited tool. The same trace-digest and stale-target checks used for capability judgments apply. The script writes one folder per task, run, and method. Task and run ids are encoded as single path components so ids containing `/` cannot create nested or escaping paths: ```text