qwen-mtp-gguf

name

qwen-mtp-gguf

description

Complete agent-ready workflow for Qwen-family MTP or nextn GGUF conversion and release. Use when Codex or another coding agent needs to inspect a user's machine, estimate disk/RAM requirements from Hugging Face model sizes and requested quant formats, bootstrap llama.cpp and Python dependencies, extract MTP heads from a matching official/base Qwen model, merge them into a fine-tuned or target safetensors model, run local HF/GGUF smoke tests with Qwen chat formatting, quantize to GGUF, and optionally upload or resume Hugging Face releases.

Qwen MTP GGUF

Operating Principle

Run this as a staged release pipeline, not a blind conversion:

1. Identify the target model and matching MTP source model.
2. Preflight the environment, disk, RAM, tools, token access, and config compatibility.
3. Ask the user to choose upload strategy when it affects disk use.
4. Prepare the target HF directory and inject only missing MTP/nextn tensors.
5. Convert to a temporary F16 GGUF, smoke-test locally, then quantize.
6. Upload with resume support, or keep local artifacts if the user chooses local-only.

MTP/GGUF conversion and llama.cpp quantization do not require a GPU. GPU acceleration can make later inference tests faster, but the default smoke test should use CPU mode (-ngl 0) so it works on common machines.

Required User Inputs

Ask for missing items only when they cannot be inferred safely:

• Target model: HF repo ID or local HF model directory.
• Matching MTP source model: the official/base Qwen-family repo with the same architecture, size, tokenizer, and MTP layout.
• Output mode: local-only, stream upload, or batch upload.
• Output repo ID when uploading.
• Quantization formats. Default full matrix: q2_k,q3_k_s,q3_k_m,q3_k_l,iq4_xs,q4_k_s,q4_k_m,q5_k_s,q5_k_m,q6_k,q8_0,bf16.

If the target and MTP source configs disagree on core architecture fields, stop and ask before continuing.