quark-torch-export

name	quark-torch-export
description	Prepare export and downstream evaluation handoff for a planned or completed Quark PTQ run. Use when the user wants to export a quantized model to HuggingFace SafeTensors, ONNX, or GGUF format, package for deployment, or set up post-quantization evaluation. Trigger for "export model", "save quantized model", "convert to GGUF", "export quantized model", "export to HuggingFace format", or when the user has a completed or planned PTQ run and needs deployment outputs.
layer	l1-atomic
primary_artifact	run_manifest.yaml
source_knowledge	["examples/torch/language_modeling/llm_ptq/quantize_quark.py","quark/torch/export/api.py","examples/contrib/llm_eval/llm_eval.py"]

quark-torch-export

Purpose

Translate a confirmed quantization plan into export expectations and downstream evaluation requirements. Export is a post-quantization step — the model must be quantized first (or have a plan to be quantized) before export decisions make sense. This skill ensures the right export format is chosen and evaluation is properly configured.

Inputs

• quant_plan.json from quark-torch-quant-plan
• workspace_context.json for the output directory
• run_manifest.yaml (optional, existing manifest to extend)

Outputs: run_manifest.yaml

Export does not own this artifact — it updates the workflow's manifest with export and evaluation config.

Schema: run_manifest.schema.json

(Export updates the workflow's run_manifest.yaml with export and evaluation fields rather than producing a new artifact.)

export:
  formats:
    - hf_format
  output_dir: ./output/qwen3-8b-fp8
  weight_format: real_quantized
  custom_mode: quark

evaluation:
  skip: false
  metrics:
    - ppl
  dataset: wikitext
  tasks: null
  batch_size: auto

Supported Export Formats

1. HuggingFace SafeTensors (hf_format) — Default

• Produces: config.json + *.safetensors files with quantization_config metadata
• Compatible with: HuggingFace transformers loading, vLLM, TGI
• CLI flag: --model_export hf_format
• Weight format options:
  • real_quantized (default) — compressed, actual quantized weights
  • fake_quantized — full-precision weights with quantization metadata only

2. ONNX (onnx)

• Produces: quark_model.onnx with optimization passes applied
• Compatible with: ONNX Runtime, TensorRT (with conversion)
• CLI flag: --model_export onnx
• Supports INT4/UINT4 conversion pass automatically

3. GGUF (gguf)

• Produces: GGUF format file for llama.cpp and compatible inference engines
• Requires: gguf>=0.10.0 package and tokenizer path
• CLI flag: --model_export gguf
• Best with: uint4_wo_32 scheme + AWQ algorithm

Multiple formats can be exported simultaneously: --model_export hf_format --model_export gguf

Export CLI Arguments

python quantize_quark.py \
  --model_dir /path/to/model \
  --output_dir /path/to/output \
  --quant_scheme fp8 \
  --model_export hf_format \                    # Export format(s)
  --export_weight_format real_quantized \        # Compression mode
  --custom_mode quark \                          # Export mode: quark|awq|fp8
  --pack_method reorder                          # Weight packing: order|reorder

Evaluation Options

Post-quantization evaluation can be configured as part of the export step:

Perplexity (PPL)

• Default dataset: wikitext
• Flag: included by default unless --skip_evaluation is set

Task-Based Evaluation (via lm-eval harness)

--tasks hellaswag,winogrande,arc_easy
--eval_batch_size auto
--num_fewshot 0

ROUGE/METEOR (for generation models)

# Evaluated on cnn_dailymail by default
--use_mlperf_rouge  # For MLPerf-compatible ROUGE scoring

KV Cache Evaluation

--use_ppl_eval_for_kv_cache
--ppl_eval_for_kv_cache_context_size 1024
--ppl_eval_for_kv_cache_sample_size 512

Skip Evaluation

--skip_evaluation  # Skip all post-quantization evaluation

Model Reload for Separate Evaluation

If quantization and evaluation are done in separate steps:

# Step 1: Quantize and export
python quantize_quark.py --model_dir MODEL --quant_scheme fp8 \
  --model_export hf_format --output_dir output/ --skip_evaluation

# Step 2: Reload and evaluate
python quantize_quark.py --model_dir MODEL --model_reload \
  --output_dir output/ --skip_quantization

Rules

• Never invent export paths that conflict with the existing run_manifest.yaml or quant_plan.json.
• Export depends on quantization. If the model has not been quantized yet, this skill produces an export plan attached to the manifest — it does not run quantization.
• Match export format to deployment target. Ask the user where the model will run: HuggingFace ecosystem → hf_format, llama.cpp → gguf, ONNX Runtime → onnx.
• GGUF works best with UINT4. If the user wants GGUF but the plan uses FP8, flag the mismatch — GGUF is primarily designed for integer quantization.

Interaction Flow

1. Check prerequisites: Is there a quant_plan.json? Has quantization been run or is this plan-only?
2. Choose format: Ask where the model will be deployed and recommend the right export format.
3. Configure evaluation: Ask if the user wants post-quantization evaluation and which metrics.
4. Emit: Update run_manifest.yaml with export and evaluation configuration.

Recovery

• If export is requested before quantization, return the missing prerequisites and attach the export request to the manifest under pending_exports.
• If GGUF export fails, check that gguf>=0.10.0 is installed and that the tokenizer is accessible.
• If ONNX export fails on a complex model, suggest trying hf_format first as a fallback.