quark-onnx-eval-runner

name	quark-onnx-eval-runner
description	Manually verify that the Quark ONNX skill family behaves correctly across the four contract categories (routing, planning, artifact, recovery). Use when maintainers need to confirm that ONNX routing, planning, artifact generation, or error recovery skills still work as expected. Trigger for "verify the ONNX skills", "smoke-test ONNX routing", "check ONNX skill behavior", or before tagging a release that touches `quark-onnx-*` skills. This is a governance tool for skill maintainers, not for end users running ONNX model accuracy evaluation — for the latter use the upstream Quark ONNX evaluation tooling.
layer	meta
backend	onnx
primary_artifact	validation_report.md
source_knowledge	["examples/onnx/yolo_quantization/quantize_yolo.py","examples/onnx/model_support.md","docs/source/onnx/basic_usage_onnx.rst","docs/source/onnx/user_guide_config_description.rst","docs/source/onnx/user_guide_auto_search_pro.rst","quark/onnx/quantization/config/custom_config.py"]

quark-onnx-eval-runner

Purpose

Walk a maintainer through manual verification of the ONNX skill family across the four contract categories: routing, planning, artifact, and recovery. Run this after modifying any quark-onnx-* skill, after a Quark ONNX upgrade, or before tagging a release.

Inputs

• The current ONNX skill files under .claude/skills-impl/{l1-atomic,l2-workflows,l3-recipes}/onnx/
• The entry stubs under .claude/skills/quark-onnx-*
• The contract schemas under .claude/skills-impl/shared/contracts/
• The example prompts under examples/agent_skills/prompts/ (add ONNX-specific cases as the catalog grows)

Outputs: validation_report.md

A markdown report recording per-category pass/fail and concrete evidence for each finding.

Schema: validation_report.schema.json

# ONNX Skill Verification Report

## Summary
| Category | Cases Run | Pass | Fail |
|----------|-----------|------|------|
| routing  | N         | N    | 0    |
| planning | N         | N    | 0    |
| artifact | N         | N    | 0    |
| recovery | N         | N    | 0    |

## Failures
### <category> / <case name>
- **Expected**: ...
- **Got**: ...
- **Impact**: ...
- **Fix**: ...

Manual Verification Protocol

For each category below, run at least one case and record the result in the report. As ONNX prompts are not yet enumerated in examples/agent_skills/prompts/, the cases below double as the seed catalog — add more as the ONNX skill set grows.

1. Routing

Goal: verify that quark-onnx-router (and Claude's auto-routing via the descriptions in .claude/skills/quark-onnx-*) maps natural-language ONNX goals to the correct downstream skill and never silently routes ONNX requests through a torch skill.

Manual procedure:

1. Pick a user-style prompt that names a .onnx artifact or ONNX-specific vocabulary.

2. In a fresh Claude Code session at the Quark repo root, paste the prompt.

3. Observe which skill Claude invokes first.

4. Compare against the expected target skill. Examples of expected mappings:

   • "Quantize my ./models/yolov8n.onnx to XINT8 for AMD NPU CNN" → quark-onnx-ptq (which loads quark-onnx-ptq-workflow)
   • "Run AutoSearchPro on this .onnx with the XINT8_SEARCH preset" → quark-onnx-autosearch-pro
   • "Analyze this .onnx — what opset is it, is it NPU-compatible, is it already QDQ?" → quark-onnx-model-intake
   • "Validate my quantized model.onnx — did QDQ insertion happen, are the non-quantized initializers byte-identical?" → quark-onnx-result-validator
   • "onnxruntime-gpu import fails, CUDAExecutionProvider not in providers list" → quark-onnx-install (or quark-onnx-debug if the user already attempted install)
   • "quantize_static failed with custom-op library load failure for BFPQuantizeDequantize" → quark-onnx-debug
   • "Is onnxruntime-rocm installed correctly? Show me the install matrix" → quark-onnx-install

5. Cross-backend guard: also run one negative prompt that mentions a .onnx path and confirm Claude does not route to quark-torch-* (e.g., "quantize ./models/foo.onnx with FP8" must not land on quark-torch-ptq).

Pass criteria: the first skill invoked matches the expected target, and no ONNX prompt is routed to a torch skill.

2. Planning

Goal: verify that quark-onnx-quant-plan produces internally consistent plans for typical ONNX inputs and that the deployment-target gates are respected.

Manual procedure:

1. Construct (or take from a prior session) a model_analysis.json produced by quark-onnx-model-intake for a representative model (e.g., YOLOv8n exported at opset 17, Conv-heavy, 6.2 MB inline).
2. Hand it to quark-onnx-quant-plan with a target preset (e.g., XINT8) and a deployment target (e.g., AMD NPU CNN).
3. Inspect the produced quant_plan.json for:
   • preset matches the requested preset
   • activation_spec and weight_spec are consistent with the preset (e.g., both XInt8Spec for XINT8)
   • EnableNPUCnn=True is set when the target is AMD NPU CNN
   • use_external_data_format is True iff the model is >2 GB
   • algo_config is a non-empty list when CLE or AdaRound was requested or recommended
   • exclude is a list (may be empty) and never contains an op the plan also quantizes
   • requires_confirmation is set when the plan deviates from preset defaults
   • Negative gate: the plan refuses incompatible combos (e.g., BFP16 + AMD NPU CNN) rather than silently downgrading

Pass criteria: the plan validates against quant_plan.schema.json, contains no internal contradictions, and explicitly rejects unsupported deployment-target / preset combinations.

3. Artifact

Goal: verify that ONNX workflow output artifacts conform to their JSON schemas and that the generated standalone script + manifest produced by quark-onnx-ptq-workflow agree with each other.

Manual procedure:

1. Take the quant_plan.json from the planning case.
2. Run quark-onnx-ptq-workflow to produce a run_manifest.yaml and the standalone <name>_ptq.py script in the user's working directory.
3. Validate the manifest against .claude/skills-impl/shared/contracts/run_manifest.schema.json (use any JSON-schema validator, e.g., the jsonschema Python package).
4. Spot-check that:
   • All required fields are present in the manifest
   • The manifest's command references the generated script path and uses python3
   • The manifest's resolved QConfig matches the plan's preset / algo_config / EnableNPUCnn / use_external_data_format
   • The generated script imports only from quark.onnx and the standard ORT calibration API (no editing of upstream examples/onnx/ or quark/onnx/ files)
   • Output paths reflect any .onnx_data sidecar when external data is enabled

Pass criteria: schema validation passes; the manifest's resolved config matches the plan; the generated script is self-contained in the user's working directory.

4. Recovery

Goal: verify that quark-onnx-debug correctly diagnoses known ONNX-side error patterns and that handoffs to quark-onnx-install happen for runtime/provider issues.

Manual procedure:

1. Pick a known ONNX error scenario. Examples:
   • RuntimeError: CUDAExecutionProvider not in available providers after installing onnxruntime (CPU build) instead of onnxruntime-gpu.
   • Custom-op library load failure for BFPQuantizeDequantize or MXQuantizeDequantize (missing C++ build, ABI mismatch).
   • model.onnx >2 GB and the run fails with "external data not found" because use_external_data_format was not set or the sibling .onnx_data was not staged.
   • OOM during calibration on a vision model with num_calib_data=1000 and batch_size=4.
   • AdaRound divergence with default learning rate.
   • NPU CNN run fails because activation scales are not power-of-two.
2. Present the error (full traceback) to quark-onnx-debug.
3. Verify the diagnosis:
   • Root cause is correctly identified
   • A concrete fix command or config change is provided
   • For runtime/provider issues, the skill explicitly hands off to quark-onnx-install instead of silently swapping execution providers
   • For OOM, the skill suggests the documented ladder: reduce num_calib_data → drop batch_size to 1 → move calibration to CPU (OptimDevice="cpu")
   • For custom-op load failures, the skill names the expected library path / build step rather than recommending the user ignore the op

Pass criteria: the diagnosis names the actual root cause, suggests a fix that would actually work, and respects the "never silently fall back to CPU" rule from quark-onnx-ptq-workflow.

Rules

• Run at least one case per category before any ONNX release — even small wording changes in preset names or custom-op names can shift routing or break generated scripts.
• A failing case means the skill is broken, not the procedure — investigate the skill first. Only update the expected behavior if the skill change was intentional.
• Report all results, not just failures. A clean run is positive evidence and worth recording.
• Document new cases inline in the report. As the ONNX skill set grows (e.g. new deployment target, new preset, new AutoSearchPro preset), add cases that cover the new triggers.
• Cross-backend isolation is part of the protocol. Every routing case must include a guard that confirms ONNX requests stay on quark-onnx-* skills.

Interaction Flow

1. Select scope: which categories to verify — all four, or a subset affected by a recent change?
2. Pick or write cases: start from the examples above; add ONNX prompt files under examples/agent_skills/prompts/ as the catalog grows.
3. Run each case manually following the protocol above in a fresh Claude Code session.
4. Record results in validation_report.md using the template.
5. Hand off failures to quark-onnx-skill-sync if they look like upstream drift, to quark-onnx-doc-drift-check if they look like stale user-facing facts, or directly to the affected skill's owner if it's a content bug.

Recovery

• If a case can't run because a prerequisite artifact is missing (e.g., no model_analysis.json for the planning case), report the missing producer skill and stop — do not fabricate the input.
• If the ONNX custom-op binaries are not built locally, the recovery case for the custom-op load failure may produce a real failure rather than a simulated one — note this distinction in the report so future runs don't confuse genuine environment gaps with skill bugs.
• If Claude Code itself is misbehaving (ONNX skill not discovered, stub not loading), that's an infrastructure issue separate from skill quality — record it distinctly in the report.
• If an ONNX prompt routes to a torch skill (or vice versa), this is a routing-category failure and must block the release — cross-backend mis-routing produces wrong artifacts silently.