hipfire-eval-harness

name	hipfire-eval-harness
description	Use when running or interpreting the unified Hipfire eval harness, choosing hipfire eval tiers/batteries/suites, validating quantized model candidates, collecting admission evidence, or deciding whether fast/medium/long/extensive eval results are sufficient for a quantization claim.

Hipfire Eval Harness

Use this skill when the user asks to evaluate a Hipfire model, compare quantized candidates, admit/reject a quant, run hipfire eval/hipfire-eval, inspect eval reports, or decide which tier/battery/suite is appropriate.

Core Rule

Treat hipfire eval as the unified evidence ledger for quantization work, not as just another benchmark script. Do not claim a candidate is better, promoted, or ship-ready unless the report has the relevant quality, performance, runtime, and provenance evidence for that claim.

Output Locations

Default report roots:

~/.hipfire/eval-results/runs/   # normal hipfire-eval runs
~/.hipfire/eval-results/smoke/  # harness smoke scripts
~/.hipfire/eval-results/cache/  # reusable per-battery result cache

Each run directory should contain:

manifest.json
results.jsonl
summary.md
artifacts/

Use --out <dir> only when the user explicitly wants a custom location or a test needs a controlled path. Prefer the default cache root over writing under the source tree.

Result Cache

The harness writes aggregate reports per run, but expensive model-backed rows are cached under ~/.hipfire/eval-results/cache/. Cache keys include the battery, model/draft/baseline/reference hashes, prompt fixture hashes for static batteries, Hipfire version, git commit/branch/dirty state, binary hash, ROCm version, GPU arch, executor, KV mode, max tokens, DFlash/profile modes, and relevant dataset or evidence inputs.

Use cache controls deliberately:

• --force: ignore cache hits for this run, but write replacement entries.
• --regenerate: delete matching cache entries before running and write new ones.
• --no-cache: disable cache reads and writes.
• --result-cache <dir>: use a custom cache root for tests or isolation.

Rows reused from cache carry metrics.cache_hit=true, plus cache_key and cache_path. Treat cached evidence as valid only when its key metadata matches the claim being made; use --regenerate after changing prompts, runtime behavior, or evidence collection semantics.

Benchmark Repeats

Default evals use one scored pass per battery. Use repeat controls when the claim depends on performance stability:

• --runs <N>: collect N scored samples for each selected battery.
• --warmup-runs <N>: run and discard N warmup passes before scored samples.
• --benchmark: shorthand for --runs 5 unless --runs is explicitly set.

Repeated runs keep raw JSONL rows with metrics.benchmark_sample=true, run_index, and run_count. The harness also emits ::aggregate rows with median/mean/stddev/min/max/p10/p90/cv_pct metrics. Use aggregate medians for regression checks; inspect raw rows when stddev or cv_pct is high.

For performance admission or regression evidence, prefer:

hipfire eval speed candidate.hfq --benchmark --force

Use --force or --regenerate for benchmark reruns so stale cached evidence does not hide runtime changes.

Passive Profiling

--profile off is the default. --profile passive collects profiling evidence after scored rows are collected. Profiling is evidence-only and must not affect pass/fail scoring.

When rocprofv3 is available, passive profiling runs a representative speed anchor under rocprof and stores raw files under the run's artifacts/rocprof/. The normalized records are ingested into artifacts/profiling.json. If rocprof or an examples executor is unavailable, the harness records an explicit skip instead of failing the eval.

Tier Selection

• fast: CI-suitable small-model smoke and admission canaries. Target: under 60s.
• medium: bounded model-eval subset for local validation. Target: under 5 min.
• long: broader quality and long-context coverage. Target: under 20 min.
• extensive: full native/evidence suite with no fixed wall-clock target.

Use the smallest tier that can prove the decision:

• For PR/no-GPU integrity: --executor mock plus no-GPU smoke.
• For a local candidate sanity check: --tier fast on a small model.
• For quant admission: at least candidate-vs-baseline quality and speed evidence.
• For DFlash claims: AR anchor must pass before DFlash rows are accepted.
• For MoE claims: include or ingest MoE router histogram evidence.

Common Commands

Build the runner and examples:

cargo build -p hipfire-runtime --bin hipfire-eval --example run --example dflash_spec_demo

No-GPU harness integrity:

HIPFIRE_EVAL_BIN=target/debug/hipfire-eval bash scripts/smoke/eval-harness-nogpu-smoke.sh

Fast local eval:

hipfire eval smoke ~/.hipfire/models/qwen3.5-0.8b-mq4.hfq

Human default speed check:

hipfire eval ~/.hipfire/models/qwen3.5-0.8b-mq4.hfq

Regenerate cached rows for a model:

hipfire eval smoke ~/.hipfire/models/qwen3.5-0.8b-mq4.hfq --regenerate

Candidate-vs-baseline admission:

hipfire eval \
  admit candidate.hfq baseline.hfq \
  --quality-json result-data.json \
  --performance-json perf.json \
  --evidence-dir runtime-evidence \

Opt-in Hugging Face dataset fetch:

hipfire eval --model candidate.hfq --battery barrage --suite gpqa --fetch-datasets

Offline/cached dataset run:

hipfire eval --model candidate.hfq --battery barrage --suite gpqa --offline

Do not pass --fetch-datasets and --offline together.

Report Audit Checklist

Before using a report as evidence, inspect:

• manifest.json: runner version, Hipfire version, git commit/branch/dirty, binary hash, arch, ROCm, model/draft/baseline/reference hashes, datasets.
• results.jsonl: one row per subcase with status, prompt hash where applicable, model hash, timing, KV mode, and metrics.
• summary.md: human-readable Models, Datasets, Comparisons, Admission, Evidence Artifacts, and Rows sections.
• artifacts/admission.json: verdict and findings.
• artifacts/comparisons.json: candidate deltas against baseline/reference.
• artifacts/quality.json: KLD/PPL/barrage accuracy evidence.
• artifacts/performance.json: tok/s, TTFT, prefill/decode metrics.
• artifacts/dflash_trace.json: AR and DFlash trace records.
• artifacts/launch_counts.json: kernel/graph/memcpy launch evidence.
• artifacts/moe_router_histogram.json: expert-hit/router evidence for MoE.
• artifacts/profiling.json: passive profiling evidence when requested.

Treat skipped evidence as explicit information, not as a pass. If required evidence is skipped or missing, the admission should not be promoted.

Validation Gates

Run the smallest gate that matches the change:

cargo test -p hipfire-runtime eval_harness --lib
bun test cli/eval.test.ts
bash -n scripts/smoke/eval-harness-nogpu-smoke.sh scripts/smoke/eval-harness-gpu-smoke.sh scripts/smoke/eval-harness-model-eval-smoke.sh

For full no-GPU coverage:

./scripts/no-gpu-ci.sh

For real GPU validation:

HIPFIRE_EVAL_BIN=target/debug/hipfire-eval \
HIPFIRE_EVAL_SMOKE_MODEL=~/.hipfire/models/qwen3.5-0.8b.mq4 \
scripts/smoke/eval-harness-gpu-smoke.sh

For candidate-vs-baseline smoke:

HIPFIRE_EVAL_BIN=target/debug/hipfire-eval \
HIPFIRE_EVAL_MODEL_SMOKE_CANDIDATE=~/.hipfire/models/qwen3.5-0.8b.mq4 \
HIPFIRE_EVAL_MODEL_SMOKE_BASELINE=~/.hipfire/models/qwen3.5-0.8b.mq6 \
scripts/smoke/eval-harness-model-eval-smoke.sh

For paired DFlash validation, provide a real draft:

HIPFIRE_EVAL_BIN=target/debug/hipfire-eval \
HIPFIRE_EVAL_SMOKE_GROUP=dflash \
HIPFIRE_EVAL_SMOKE_MODEL=~/.hipfire/models/qwen3.5-27b-mq4.hfq \
HIPFIRE_EVAL_SMOKE_DRAFT=~/.hipfire/models/qwen3.5-27b-dflash-mq4.hfq \
scripts/smoke/eval-harness-gpu-smoke.sh

Claim Discipline

• Quality claim: cite exact report path, model hashes, prompt/dataset hashes, quality artifact, and comparison/admission artifact.
• Performance claim: include binary hash, git commit, GPU arch, ROCm, KV mode, prompt hash, and performance artifact.
• DFlash claim: include AR anchor, DFlash anchor, draft hash, and DFlash trace.
• MoE claim: include router histogram evidence and explain expert-hit coverage.
• Profiling claim: profiling is evidence only; it must not affect pass/fail.

If evidence was generated with mock executor or generated smoke fixtures, call it an integrity check only, not a quality/performance result.