quark-torch-model-intake

name	quark-torch-model-intake
description	Inspect a target model and prepare metadata for Quark PTQ planning. Use when the user needs model path validation, architecture detection, quantization target discovery, layer counting, risk assessment, or transformer compatibility checks before planning PTQ. Trigger for "analyze my model", "check this model", "what architecture is this", "can Quark quantize X", "is this model supported", or when any quantization step needs model facts that are missing.
layer	l1-atomic
primary_artifact	model_analysis.json
source_knowledge	["examples/torch/language_modeling/llm_ptq/quantize_quark.py","quark/torch/utils/llm/model_preparation.py","quark/torch/quantization/config/template.py"]

quark-torch-model-intake

Purpose

Validate the target model and extract the structural facts that quark-torch-quant-plan needs to make correct quantization decisions. This step exists because different model architectures have different quantization requirements — MoE models need expert module replacement, some models require trust_remote_code, and certain architectures have known compatibility issues with specific transformers versions.

Inputs

• Model path (HuggingFace ID or local directory)
• env_context.json for Python and accelerator constraints
• workspace_context.json for the validated model path

Outputs: model_analysis.json

Captures architecture facts, quantizable layer count, transformer compatibility, and risks.

Schema: model_analysis.schema.json

{
  "analysis_status": "complete",
  "model": {
    "model_path": "Qwen/Qwen3-8B",
    "model_type": "qwen3",
    "trust_remote_code": false,
    "transformers_version_required": ">=4.48.0,<5.3",
    "loading_class": "AutoModelForCausalLM",
    "is_moe": false,
    "num_hidden_layers": 36,
    "hidden_size": 4096,
    "estimated_size_gb": 16.0
  },
  "quantization_targets": {
    "linear_layer_count": 224,
    "has_non_linear_experts": 
     
     
  
   
    
       
       
       
    
  

Supported Model Architectures

Quark has built-in templates for 36 model families:

Category	Models
Standard LLMs	llama, mistral, opt, phi, phi3, qwen, qwen2, gptj, cohere, olmo
Advanced LLMs	qwen3, qwen3_next, deepseek, deepseek_v2, deepseek_v3, deepseek_v32
MoE Models	mixtral, dbrx, llama4, qwen2_moe, qwen3_moe, qwen3_5_moe, gpt_oss, granitemoehybrid, glm4_moe
Vision-Language	mllama, deepseek_vl_v2, qwen3_vl_moe
Other	chatglm, gemma2, gemma3, gemma3_text, grok-1, instella, kimi_k25, minimax_m2

Models not in this list may still work if they follow standard HuggingFace AutoModelForCausalLM patterns, but need extra attention.

What to Extract

From config.json

• model_type — must match a Quark template name (e.g., "llama", "qwen3", "mistral")
• num_hidden_layers — determines the number of quantizable linear layers
• hidden_size, intermediate_size — affects memory estimates
• num_attention_heads, num_key_value_heads — relevant for KV cache quantization
• Architecture-specific fields (e.g., num_experts for MoE models)

Quantization Targets

• Count total linear layers (these are what gets quantized)
• Identify default exclusions — lm_head is almost always excluded
• For MoE models, note that expert modules need special preparation via prepare_for_moe_quant()

Transformer Compatibility

Some models require specific minimum transformers versions:

• llama4 → transformers >= 4.51.0
• gpt_oss, granitemoehybrid → transformers >= 4.55.1
• qwen3_vl_moe → transformers >= 4.57.0
• qwen3_5_moe → transformers >= 5.2.0
• General requirement for LLM PTQ: transformers < 5.3

Special Loading Requirements

• deepseek_vl_v2 → uses AutoModel instead of AutoModelForCausalLM
• mllama → uses MllamaForConditionalGeneration
• gpt_oss → needs Mxfp4Config(dequantize=True) for loading
• Models with custom code → need trust_remote_code=True

Risks

Flag anything that could cause failures downstream:

• Model type not in Quark's template list
• Transformer version incompatibility
• Very large models that may need --multi_gpu or --file2file_quantization
• MoE models that need module replacement

Concrete Actions

Action 1: Read model config (NEVER load full weights)

python3 -c "
from transformers import AutoConfig
import json
config = AutoConfig.from_pretrained('<MODEL_PATH>', trust_remote_code=True)
info = {
    'model_type': config.model_type,
    'num_hidden_layers': config.num_hidden_layers,
    'hidden_size': config.hidden_size,
    'intermediate_size': getattr(config, 'intermediate_size', None),
    'num_attention_heads': config.num_attention_heads,
    'num_key_value_heads': getattr(config, 'num_key_value_heads', None),
    'vocab_size': config.vocab_size,
    'num_experts': getattr(config, 'num_local_experts', getattr(config, 'num_experts', None)),
    'torch_dtype': str(getattr(config, 'torch_dtype', 'unknown')),
}
print(json.dumps(info, indent=2))
"

Action 2: Check model size (for local models)

du -sh /path/to/model/
ls -lh /path/to/model/*.safetensors

Action 3: Present summary table to user

After running the above, format the results as:

Model Analysis:
  Model path:       <path or HuggingFace ID>
  Model type:       <model_type from config>
  Hidden layers:    <num_hidden_layers>
  Linear layers:    ~<estimated count>
  MoE:              Yes/No
  Exclude defaults: [lm_head]
  Risks:            <list or "None">
  Compatibility:    OK / <version constraints>

This table is what the user sees at Checkpoint 1 of quark-torch-llm-ptq-workflow.

Rules

• Run or reference quark-workspace-validate first to confirm that model paths are valid before attempting to read config.json.
• Do not load the full model during intake. Reading config.json and listing files is enough — loading weights is expensive and belongs to the quantization step.
• Preserve ambiguity when a model reference could be local or remote. Note both possibilities and let the user resolve.
• Surface new environment constraints (e.g., model requires transformers >= 4.57.0) by recording them in model_analysis.json under risks and asking quark-torch-router to add them to session_context.json's open_questions. Do not write directly to env_context.json.

Interaction Flow

1. Confirm model source: Is it a local directory or a HuggingFace ID? Check if quark-workspace-validate already confirmed the path.
2. Read config: Extract architecture facts from config.json. Present a summary table to the user.
3. Assess compatibility: Check model type against Quark's template list. Flag any transformer version requirements.
4. Identify risks: Note anything that could affect downstream PTQ planning.
5. Emit: Write model_analysis.json. Surface any new constraints back to quark-torch-router so they land in session_context.json.

Recovery

• If analysis_status: "partial" — some facts were extracted but the model could not be fully inspected. Common cause: model needs trust_remote_code=True but the user has not approved it.
• If model type is not in Quark's template list — report this clearly. The user may need to register a custom template (see LLMTemplate.register_template() in quantize_quark.py).
• If transformer version is incompatible — show the exact version mismatch and the upgrade/downgrade command.