quark-torch-result-validator

name	quark-torch-result-validator
description	Validate Quark quantization output using four lightweight checks: auxiliary file copy alignment, excluded tensor MD5 byte-identity, config.json deep comparison after stripping quantization keys, and safetensors header pattern/dtype summaries. Intended for post-export or file2file validation. Trigger for "validate quantization result", "check quantized model output", "verify exported weights".
layer	l1-atomic
primary_artifact	validation_report.md
source_knowledge	["examples/torch/language_modeling/llm_ptq/quantize_quark.py","quark/torch/export/api.py"]

quark-torch-result-validator

Purpose

Run four lightweight checks on a completed Quark export or file2file output. All checks read only safetensors headers or small metadata files — no full tensor payload is loaded except for the bounded MD5 spot-check. Results feed into a structured validation_report.md.

Runtime Assumptions

All scripts (quant_validation.py, run_validation.py) live in the same directory as this SKILL.md, under .claude/skills-impl/l1-atomic/torch/quark-torch-result-validator/.

Resolve SKILL_DIR from the repo root before running any command:

SKILL_DIR=.claude/skills-impl/l1-atomic/torch/quark-torch-result-validator

run_validation.py writes JSON to stdout; quant_validation.py diagnostics go to stderr. Never treat stderr as structured output.

Contracts

• Input: session_context.json, quant_plan.json (for exclude rules and model paths)
• Output: validation_report.md (structured per the report template below)
• Schemas: shared/contracts/validation_report.schema.json

Inputs

Field	Source	Required
source_model_dir	user or session_context	Yes
quantized_model_dir	user or run_manifest	Yes
quant_config	quant_plan.json or user-supplied JSON	For step 2 only

Outputs

validation_report.md with one section per executed step. Unexecuted steps are marked skipped.

Interaction Flow

1. Confirm SKILL_DIR, source_model_dir, and quantized_model_dir are available.
2. Run self-test to verify scripts are intact: python3 "$SKILL_DIR/run_validation.py" self-test
3. Execute steps in cheap-to-expensive order (4 → 1 → 3 → 2).
4. Collect JSON from stdout for each step; write validation_report.md.
5. Surface any ok: false steps with their errors and mismatches.

Four Validation Steps

Order	Function	CLI subcommand	Purpose
1	check_auxiliary_files_copied	auxiliary	Compare non-weight auxiliary files between source and quantized dirs
2	check_non_quantized_tensors_md5_unchanged	md5	MD5 spot-check payload bytes of exclude-listed tensors
3	check_config_json_equal_except_quantization	config	Deep compare config.json after stripping quantization keys
4	get_fuzzy_tensor_names	fuzzy	Summarize canonical header patterns and dtype_counts

Run in cost order: 4 → 1 → 3 → 2.

Agent Execution Contract

Self-Test

SKILL_DIR=.claude/skills-impl/l1-atomic/torch/quark-torch-result-validator
python3 "$SKILL_DIR/run_validation.py" self-test

Exits 0 and prints exported symbols self-test (__all__): ok on success.

CLI Commands

SKILL_DIR=.claude/skills-impl/l1-atomic/torch/quark-torch-result-validator

# 4. get_fuzzy_tensor_names (cheapest — header only)
python3 "$SKILL_DIR/run_validation.py" fuzzy \
  --model-path ./quantized-model

# 1. check_auxiliary_files_copied
python3 "$SKILL_DIR/run_validation.py" auxiliary \
  --source-model-dir ./original-model --quantized-model-dir ./quantized-model \
  --ignore 'README*'

# 3. check_config_json_equal_except_quantization
python3 "$SKILL_DIR/run_validation.py" config \
  --original-config ./original-model/config.json \
  --quantized-config ./quantized-model/config.json

# 2. check_non_quantized_tensors_md5_unchanged (most expensive)
python3 "$SKILL_DIR/run_validation.py" md5 \
  --source-model-dir ./original-model --output-model-dir ./quantized-model \
  --quant-config '{"exclude":["*.lm_head.weight"],"max_samples":50}'

For md5, --quant-config accepts a JSON string or a path to a JSON file. If SKILL_DIR or model paths cannot be resolved, mark the affected step skipped.

Public API (for direct Python import)

from quant_validation import (
    check_auxiliary_files_copied,
    check_non_quantized_tensors_md5_unchanged,
    check_config_json_equal_except_quantization,
    get_fuzzy_tensor_names,
)

All four functions are in __all__. Other public-named helpers are internal utility surface.

Recovery

Failure	Recovery
Self-test exits non-zero	Report script integrity failure; do not run further steps
source_model_dir missing	Mark steps 1, 2, 3 as skipped; run step 4 on quantized dir only
No safetensors files found	Mark steps 2 and 4 as skipped
quant_config missing exclude rules	Mark step 2 as skipped
File too large for SHA256	Recorded in hash_skipped_files; size comparison still runs

Report Template

## Validation Report — quark-torch-result-validator

**Step 4 — fuzzy tensor names**: ok / FAIL / skipped
  - patterns: <count>, mixed-dtype patterns: <count>
  - Notable: <pattern> → <dtype_counts>

**Step 1 — auxiliary files**: ok / FAIL / skipped
  - missing: <count>, mismatched: <count>, extra: <count>

**Step 3 — config.json**: ok / FAIL / skipped
  - mismatches: <count>
  - <key>: <source_value> / <quantized_value>

**Step 2 — MD5 spot-check**: ok / FAIL / skipped
  - candidates: <count>, checked: <count>, sampled: true/false
  - mismatches: <count>

Canonical Name Rules

• Replace only pure numeric path segments with *: layers.12 → layers.*
• Do not alter digits embedded in non-numeric names: norm1, w2 stay unchanged
• dtype_counts is aggregated per pattern; multiple dtypes in one pattern signals mixed precision risk

Optional Dtype Hints

Heuristics only — not mandatory pass/fail rules:

• FP8: dtypes such as F8_E4M3
• MXFP4: often uint8-packed
• INT schemes: depend on export naming convention