track-models

name	track-models
description	Track popular/new open-source LLMs and update docs/model_coverage.mdx with their kernel support status. Use when discovering new models to add to the coverage tracker, checking if a specific model is covered, or refreshing model coverage documentation.

Track Models

Discover popular or newly released open-source LLMs, determine their kernel coverage in FlashInfer-Bench, and update docs/model_coverage.mdx with summary and detailed per-kernel status tables.

Usage

# Auto-discover new popular models not yet tracked
/track-models --discover

# Add a specific model by name
/track-models --model-name mistral-7b
/track-models --model-name gemma-3-27b --hf-repo-id google/gemma-3-27b-it

# Refresh coverage status for all already-tracked models
/track-models --refresh-status

# Do everything: discover new models + refresh existing ones
/track-models --discover --refresh-status

Parameters

• --discover (optional): Auto-discover new popular models not yet in model_coverage.mdx
• --model-name (optional): Specific model to add or refresh (e.g., "mistral-7b", "gemma-3-27b")
• --hf-repo-id (optional): Override the HuggingFace repo ID (e.g., "google/gemma-3-27b-it")
• --refresh-status (optional): Re-check definition files for all already-tracked models and update ✅/❌ status

Prerequisites

• docs/model_coverage.mdx must exist (it does — managed by this skill)
• tmp/flashinfer-trace/definitions/ must exist with current definition JSON files (this is the HuggingFace dataset clone — populated by /clone-repos; the in-repo flashinfer_trace/ directory was removed in the trace-dataset refactor)
• Run /clone-repos first if you need SGLang or sgl-cookbook configs for a model that isn't in the existing patterns, or to ensure the trace dataset clone is current

---

What This Skill Does

Phase 1: Model Discovery

When --discover is set, find popular open-source LLMs not yet tracked in model_coverage.mdx.

Discovery Sources (in order)

1. SGLang day-0 additions (highest priority — newly supported models):

   # Models added to SGLang in the last 30 days (new .py files in models dir)
   git -C tmp/sglang log --since="30 days ago" --name-status --diff-filter=A \
       -- "python/sglang/srt/models/*.py" | grep "^A" | awk '{print $2}'
   

   These are models with brand-new SGLang support — they are the most likely to have novel kernels not yet tracked. Parse each new filename to derive the model slug.

2. SGLang supported models list (full catalog):

   ls tmp/sglang/python/sglang/srt/models/
   

   SGLang models are a curated list of production-quality LLMs. Every model in SGLang is a candidate.

3. sgl-cookbook (models with recommended serving configs):

   ls tmp/sgl-cookbook/data/models/generated/v0.5.6/
   

   Models with a YAML config are actively deployed — highest priority.

4. Inference API provider model lists (strong popularity signal): A model served by 2+ commercial inference APIs is definitively production-critical. Fetch or browse the following provider pages and extract the open-weight model list:

   Provider	URL	Notes
   Together AI	https://www.together.ai/pricing	Full model catalog with pricing tiers
   Fireworks AI	https://fireworks.ai/pricing	Open-source models section
   Groq	https://groq.com/pricing	Smaller curated list, very high-traffic models
   OpenRouter	https://openrouter.ai/models	Largest aggregator; filter by free or sort by throughput
   DeepInfra	https://deepinfra.com/models	Broad open-source catalog
   Hyperbolic	https://app.hyperbolic.xyz/models	GPU-native provider
   Cerebras	https://inference.cerebras.ai	Wafer-scale inference, limited but popular subset

   Heuristic: Any model listed by ≥ 2 providers should be added to coverage tracking. Models on ≥ 3 providers are the highest priority (most widely deployed in production).

   Known high-priority models from provider crawls (update this list over time):

   • Llama 4 Scout (17Bx16E) — Together, Groq, sgl-cookbook
   • Llama 4 Maverick (17Bx128E MoE) — Together, Groq
   • Llama 3.1 405B — Together, Fireworks
   • gpt-oss 120B / gpt-oss 20B — Together, Fireworks, Groq (OpenAI open-source)
   • GLM-4.5 / GLM-4.6 / GLM-5 — Together, Fireworks; sgl-cookbook has glm46.yaml
   • MiniMax M2 / M2.5 — Together, Fireworks
   • Kimi K2 / K2.5 — Together, Fireworks, Groq
   • DeepSeek-V3.1 / R1-0528 — Together, Fireworks (updated DeepSeek variants)
   • Qwen3 235B A22B — Together, Fireworks, Groq
   • Mistral Small 3 (24B) — Together; sgl-cookbook has mistral.yaml

5. HuggingFace website (primary web source — fetch directly): Use WebFetch to retrieve the HuggingFace trending models page and extract newly released open-weight LLMs. This gives the most up-to-date signal for models gaining traction before they appear on inference provider lists.

   # Fetch trending models (text/LLM category)
   WebFetch: https://huggingface.co/models?pipeline_tag=text-generation&sort=trending
   

   Also check the HuggingFace "new" filter for models released in the last 7 days:

   WebFetch: https://huggingface.co/models?pipeline_tag=text-generation&sort=created
   

   Focus on models that are:

   • Open-weight (not API-only or gated without public access)
   • From established labs (Meta, Google, Mistral, Alibaba/Qwen, DeepSeek, Moonshot, etc.)
   • Have ≥ 1K downloads/week or trending in the top 50
   • Transformer-based decoder (not vision-only or encoder-only)

   For each candidate, fetch its model page to confirm the architecture:

   WebFetch: https://huggingface.co/{org}/{model}/blob/main/config.json
   

6. HuggingFace trending (supplemental web search): Search for "huggingface trending LLM open source {current_year}" to find newly popular models. Focus on models that are:

   • Open-weight (not API-only)
   • LLM/VLM with transformer-based decoder
   • Have ≥ 1K downloads/week or are from a major lab (Meta, Google, Mistral, Alibaba, etc.)

Filtering Already-Tracked Models

Read the current docs/model_coverage.mdx and extract all model names from the ## Summary table. Skip any model whose name is already listed.

Architecture Priority

Prefer to discover models that use kernel types already supported by FlashInfer-Bench:

• GQA attention (most LLMs)
• MLA attention (DeepSeek family)
• GDN linear attention (Qwen3-Next family)
• MoE FFN (Mixtral, Qwen-MoE, DeepSeek, etc.)
• Mamba/SSM (NemotronH, GraniteMoe-Hybrid)
• Dense FFN (Llama, Gemma, Mistral)

---

Phase 2: Architecture Extraction

For each new model, extract architecture details. Use the same approach as the extract-kernel-definitions skill.

Step 1: Get config.json from HuggingFace

from huggingface_hub import hf_hub_download
import json

config_path = hf_hub_download(repo_id="meta-llama/Llama-3.1-8B", filename="config.json")
with open(config_path) as f:
    config = json.load(f)

Key fields to extract:

Field	Used For
hidden_size	RMSNorm definition names, GEMM K dim
num_hidden_layers	Layer count
num_attention_heads	GQA/MLA q_heads
num_key_value_heads	GQA kv_heads
head_dim or hidden_size / num_attention_heads	Attention head dim
intermediate_size	GEMM N dim for MLP
vocab_size	Sampling definition names
architectures	Model class name → infer op types
num_experts / num_local_experts	MoE expert count
num_experts_per_tok	MoE topk

Step 2: Determine attention type from architecture

Use the architectures field in config.json or the SGLang model class:

Architecture String	Attention Type	Notes
LlamaForCausalLM, MistralForCausalLM, GemmaForCausalLM, Qwen2ForCausalLM	GQA	Standard
DeepseekV2ForCausalLM, DeepseekV3ForCausalLM	MLA	Has q_lora_rank, kv_lora_rank, qk_rope_head_dim
Qwen3NextForCausalLM	GDN + GQA hybrid	Has gdn_* config keys
NemotronHForCausalLM	GQA + Mamba2 hybrid	Has ssm_* config keys
MixtralForCausalLM	GQA + MoE	Has num_local_experts

Step 3: Find sgl-cookbook TP/EP config

find tmp/sgl-cookbook/data/models/generated/v0.5.6/ -name "*.yaml" | xargs grep -l "{model_keyword}"

Parse the YAML to extract all unique tp and ep values. Use these to compute per-TP head counts.

If no sgl-cookbook config exists, use TP=1 (single GPU baseline).

---

Phase 3: Map Kernels to Definitions

For each model, compute the full list of expected definition names, then check which ones exist in tmp/flashinfer-trace/definitions/ (the HF dataset clone — the only location where definition JSONs live).

3a: Compute expected definitions

Follow these rules (same as CLAUDE.md):

RMSNorm (not TP-dependent):

• rmsnorm_h{hidden_size}
• fused_add_rmsnorm_h{hidden_size}
• If MLA: also rmsnorm_h{q_lora_rank} and rmsnorm_h{kv_lora_rank}

GEMM (not TP-dependent):

• gemm_n{intermediate_size}_k{hidden_size} (gate/up proj)
• gemm_n{hidden_size}_k{intermediate_size} (down proj)
• gemm_n{hidden_size}_k{hidden_size} (o_proj, if square)
• For MLA: gemm_n{q_lora_rank * num_heads}_k{hidden_size} etc. (check SGLang impl)

GQA (TP-dependent, per TP value):

• gqa_paged_prefill_causal_h{q//TP}_kv{kv//TP}_d{head_dim}_ps1
• gqa_paged_prefill_causal_h{q//TP}_kv{kv//TP}_d{head_dim}_ps64
• gqa_paged_decode_h{q//TP}_kv{kv//TP}_d{head_dim}_ps1
• gqa_paged_decode_h{q//TP}_kv{kv//TP}_d{head_dim}_ps64
• gqa_ragged_prefill_causal_h{q//TP}_kv{kv//TP}_d{head_dim}

MLA (TP-dependent, per TP value):

• mla_paged_prefill_causal_h{num_heads//TP}_ckv{ckv_dim}_kpe{kpe_dim}_ps1
• mla_paged_prefill_causal_h{num_heads//TP}_ckv{ckv_dim}_kpe{kpe_dim}_ps64
• mla_paged_decode_h{num_heads//TP}_ckv{ckv_dim}_kpe{kpe_dim}_ps1
• mla_paged_decode_h{num_heads//TP}_ckv{ckv_dim}_kpe{kpe_dim}_ps64

Where ckv_dim = kv_lora_rank + qk_rope_head_dim and kpe_dim = qk_rope_head_dim.

DSA (for DeepSeek V3.2-style, TP-dependent):

• dsa_topk_indexer_fp8_h{num_heads//TP}_d{head_dim}_topk{topk}_ps64
• dsa_sparse_attention_h{num_heads//TP}_ckv{ckv_dim}_kpe{kpe_dim}_topk{topk}_ps1
• dsa_sparse_attention_h{num_heads//TP}_ckv{ckv_dim}_kpe{kpe_dim}_topk{topk}_ps64

GDN (for Qwen3-Next-style, TP-dependent, per TP value):

• gdn_prefill_qk{q_heads//TP}_v{v_heads//TP}_d{head_dim}_k_last
• gdn_decode_qk{q_heads//TP}_v{v_heads//TP}_d{head_dim}_k_last
• gdn_mtp_qk{q_heads//TP}_v{v_heads//TP}_d{head_dim}_k_last

Mamba2 SSU (for NemotronH-style, TP-dependent):

• mamba_ssu_decode_h{nheads//TP}_d{head_dim}_s{dstate}_ng{ngroups//TP}
• Constraints: head_dim ∈ [64,128,256], dstate ∈ [64,128,256], nheads/ngroups ∈ [1,8,16]

MoE (EP-dependent):

• moe_fp8_block_scale_ds_routing_topk{topk}_ng{n_group}_kg{topk_group}_e{num_experts//EP}_h{hidden_size}_i{intermediate_size}

RoPE (not TP-dependent):

• rope_with_cos_sin_cache_neox_style_d{head_dim}_rd{rotary_dim}
• rope_with_cos_sin_cache_gptj_style_d{head_dim}_rd{rotary_dim}

Sampling (not TP-dependent):

• top_k_sampling_from_probs_v{vocab_size}