quark-torch-file2file-quantization

name	quark-torch-file2file-quantization
description	Low-memory file2file quantization for very large safetensors LLMs that cannot be loaded whole. Use when the user wants to run file2file quantization, adapt a new safetensors checkpoint without loading the full model, register an external LLMTemplate, inspect sharded checkpoint naming, generate wrapper or conversion scripts, or validate low-memory sharded quantization outputs. Trigger for "run file2file quantization", "quantize without loading the model", "large safetensors low-memory quantization", "file2file for DeepSeek/Qwen/MoE", "safetensors naming incompatible", "register LLMTemplate externally".
layer	l1-atomic
primary_artifact	run_manifest.yaml
source_knowledge	["quark/torch/quantization/file2file_quantization.py","quark/torch/quantization/weight_convert.py","quark/torch/utils/llm/model_preparation.py","docs/source/pytorch/file2file_quantization.rst","examples/torch/language_modeling/llm_ptq/quantize_quark.py"]

quark-file2file-quantization-runner

Purpose

Quantize very large safetensors checkpoints without loading the full model into memory, using ModelQuantizer.direct_quantize_checkpoint(). Covers the full adaptation path: checkpoint inspection, external LLMTemplate registration, optional naming normalization via a one-time conversion script, minimum-scale experiment gating, full file2file execution, and output validation via quark-torch-result-validator.

Default policy: solve naming or layer-selection mismatches with external adapters (LLMTemplate.register_template(), weight_converters) or a temporary conversion script. Do not modify Quark source unless the required capability is absent, the external path has been ruled out, and the user explicitly agrees.

Inputs

• pretrained_model_path — local directory of the safetensors checkpoint
• save_path — output directory
• quant_scheme — quantization scheme (e.g., w_fp8_a_fp8, w_int4_a_bf16)
• device — e.g., cuda:0, cpu
• User intent: direct quantization, checkpoint conversion first, HF-format export, or script generation only
• model_analysis.json from quark-torch-model-intake (optional but recommended)

Outputs: run_manifest.yaml

Records the adaptation path taken, scripts generated, experiment results, and validation status.

pretrained_model_path: /models/DeepSeek-V3
save_path: /output/DeepSeek-V3-fp8
quant_scheme: w_fp8_a_fp8
device: cuda:0
adaptation_path: external_template          # direct | external_template | weight_converters | conversion_script
conversion_script: null                     # path if generated
wrapper_script: /tmp/run_ds_v3_f2f.py
min_experiment:
  status: passed                            # passed | failed | skipped
  layers_covered: [layer_0_expert_0, ...]
  moe_covered: true
full_run_status: completed                  # pending | completed | failed
validation_status: passed                  # passed | failed | not_run

Interaction Flow

Step 1 — Inspect checkpoint

Read keys and config without loading weights:

python - <<'PY'
import json, os
from glob import glob
from safetensors.torch import safe_open

model_dir = "<pretrained_model_path>"
index_path = os.path.join(model_dir, "model.safetensors.index.json")
print("config:", os.path.exists(os.path.join(model_dir, "config.json")))
print("index:", os.path.exists(index_path))
files = sorted(glob(os.path.join(model_dir, "*.safetensors")))
print("safetensors:", len(files))
if files:
    with safe_open(files[0], framework="pt", device="cpu") as f:
        keys = list(f.keys())
    print("sample_keys (first 80):")
    for k in keys[:80]: print(" ", k)
if os.path.exists(index_path):
    with open(index_path) as f:
        wm = json.load(f).get("weight_map", {})
    print("index_keys:", len(wm))
PY

Verify: model_type, weight-name suffixes (*.weight, *_scale_inv, *.scale), shard count, MoE expert / shared-expert / gate naming, and whether scale tensors are co-located with weights.

Step 2 — Choose adaptation path

Situation	Action
Names already match Quark template	Direct file2file; tune exclude_layers only
Layer naming differs from built-in template	External LLMTemplate.register_template()
Only weight suffixes differ	weight_converters / _apply_weight_converters
Scale naming or dtype incompatible pre-recovery	Generate normalization conversion script first

_apply_weight_converters limits: suited for post-recovery single-suffix rename or one-source split. Not suited for multi-source merge, cross-shard scale pairing, or FP4→FP8 dtype conversion.

Step 3 — Generate wrapper or conversion script

For external template registration, generate a wrapper script (do NOT modify quantize_quark.py):

from quark.torch import ModelQuantizer
from quark.torch.utils.llm import LLMTemplate

template = LLMTemplate(
    model_type="<model_type>",
    kv_layers_name=["<pattern>"],
    q_layer_name=["<pattern>"],
    exclude_layers_name=["embed", "head", "<other>"],
)
LLMTemplate.register_template(template)

quantizer = ModelQuantizer(config)
quantizer.direct_quantize_checkpoint(
    pretrained_model_path=pretrained_model_path,
    save_path=save_path,
    keep_excluded_layers_as_original_model_state=False,
    weight_converters=weight_converters,
    device=device,
)

Wrapper must print: registered model_type, input/output dirs, quant scheme, and exclude rules. Conversion scripts must stream safetensors (no full-model load), include explicit remap_name(), scale/weight pairing validation, shard output in HF style, index rebuild, and atomic output.

Step 4 — Minimum-scale experiment (mandatory gate)

This step is not optional. Full file2file must not run until the minimum experiment passes.

Construct the minimum input:

1. If num_hidden_layers is safely reducible, copy config.json with the smallest value that still covers at least one MoE layer (for MoE models, use first_moe_layer_id + 1).
2. If not, generate a subset checkpoint filtered by --key-regex covering at least one complete MoE expert + its scale tensor + shared expert/router/gate + adjacent non-quantized tensors.
3. Never set num_hidden_layers=1 for a MoE model if layer 0 is dense — verify from config or key patterns which layer is the first actual MoE layer.

Run the minimum experiment, then call quark-torch-result-validator with: inspect_safetensors, summarize_dtypes, check_index_consistency, check_scale_pairs, get_fuzzy_tensor_names, and auxiliary-file copy check (if source dir available).

If validation fails, fix template / naming / subset and re-run. Do not proceed to full file2file.

Step 5 — Full file2file

Set cache paths to avoid polluting home quota:

export TMPDIR=/path/to/run/tmp
export TORCH_EXTENSIONS_DIR=/path/to/run/torch_extensions
export TRITON_CACHE_DIR=/path/to/run/triton_cache

Run the wrapper script. After completion, re-run quark-torch-result-validator on the final output with the same checks as Step 4.

Step 6 — Deliver

Emit run_manifest.yaml and present to the user:

• Script paths and execution commands
• Minimum experiment summary: layers selected, MoE coverage, validation outcome
• Full run validation summary
• Any outstanding risk items

Recovery

Failure	Action
Missing Triton / compressed-tensors	Install dependency, retry
Incomplete source shards	Repair shards and index before proceeding
Template not recognized	Register via external LLMTemplate; do not patch Quark source
Scale/weight cannot be paired	Normalize checkpoint first; add unit test if Quark recovery is extended
Output index inconsistent	Rebuild index or fix shard write logic; re-validate
Minimum experiment fails	Fix adapter/naming/subset; never skip to full run

Rules

• Never load full weights for inspection — read safetensors headers only.
• External adapters first: use LLMTemplate.register_template() or weight_converters before any Quark source change.
• Minimum experiment is a gate, not a hint — full file2file is blocked until it passes.
• MoE minimum experiment must include actual MoE expert weights + their scale tensors. Setting num_hidden_layers=1 when layer 0 is dense is invalid.
• No cluster paths in shared scripts — keep one-time paths in temporary wrapper scripts only.
• Source changes require tests — if Quark source must change, add a test/test_for_torch/ unit test and run the relevant pytest before committing.

Notes

• DeepSeek-V4-family: prefer external LLMTemplate; check whether inference-format naming needs a conversion script (embed_tokens→embed, self_attn→attn, q_proj→wq, etc.) before file2file.
• FP4/e2m1fn expert weights must be converted to FP8/e4m3fn in the conversion script, not via _apply_weight_converters.
• Sibling scale naming ({base}.scale) must be resolvable before recovery; post-recovery suffix conversion cannot substitute for pre-recovery scale identification.
• MTP embedding/head, attention, gate, hc_* auxiliary tensors must be explicitly excluded in the template or conversion script.