onboard-model

# Onboard Model Thin orchestrator that runs the five-phase pipeline for adding a new LLM to FlashInfer-Bench. Each phase delegates to a focused skill — this file is the contract that chains them together via a shared run manifest. ## Overview of phases | Phase | Skill | Output | |-------|-------|--------| | 0 | [`/clone-repos`](../clone-repos/SKILL.md) | `tmp/sglang/`, `tmp/flashinfer/`, `tmp/sgl-cookbook/`, `tmp/flashinfer-trace/` cloned and current | | 1 | [`/discover-models`](../discover-models/SKILL.md) | manifest `kernels[]` populated with `phase1_status`, `fi_status`, `fi_trace_template`, `sgl_status` | | 2 | [`/extract-kernel-definitions`](../extract-kernel-definitions/SKILL.md) (+ inline `gh issue create` for `fi_missing`) | definition JSONs in `tmp/flashinfer-trace/definitions/` — auto-dumped via `FLASHINFER_TRACE_DUMP=1` for kernels with `fi_trace_template=true`, otherwise hand-written; manifest `phase2_status=done` (and `fi_issue_url` for fi_missing) | | 3 | [`/collect-workloads`](../collect-workloads/SKILL.md) (+ inline SGLang PR for `sgl_missing`) | workloads + blobs in `tmp/flashinfer-trace/`; manifest `phase3_status=done` | | 4 | [`/submit-onboarding-prs`](../submit-onboarding-prs/SKILL.md) | one HF PR + one bench PR per definition; manifest `phase4` populated | The state contract between skills is the run manifest at `tmp/onboard_{model_slug}_{date}.json` — see "Run manifest" below. ## Usage ```bash # Discover new models and fully onboard any that are ready /onboard-model --discover # Onboard a specific known model end-to-end /onboard-model --model-name qwen3-235b-a22b --hf-repo-id Qwen/Qwen3-235B-A22B # Run only specific phases (e.g. skip workload collection for now) /onboard-model --model-name kimi-k2 --phases 0,1,2 # Dry-run: discover and report what would be done without making changes /onboard-model --discover --dry-run ``` ## Parameters - `--discover` (optional): Auto-discover new models from SGLang day-0 additions and sgl-cookbook. Compatible with `--model-name`. - `--model-name` (optional): Specific model slug to onboard (e.g. `qwen3-235b-a22b`). - `--hf-repo-id` (optional): HuggingFace repo override. Inferred from `--model-name` if omitted. - `--phases` (optional): Comma-separated list of phases to run (default: `0,1,2,3,4`). - `--dry-run` (optional): Print what would be done without writing files or submitting PRs. - `--skip-workload` (optional): Skip Phase 3 (e.g. when no GPU is available). - `--submit-prs` (optional): Submit Phase 4 PRs (default: true). --- ## Phase 0: Update local repos Delegate to the [`clone-repos`](../clone-repos/SKILL.md) skill. ```bash /clone-repos ``` After the pull, capture the current SHAs and write them to the manifest's `repo_shas`: ```bash git -C tmp/sglang rev-parse --short HEAD git -C tmp/flashinfer rev-parse --short HEAD git -C tmp/sgl-cookbook rev-parse --short HEAD git -C tmp/flashinfer-trace rev-parse --short HEAD ``` Report the SHAs in the Phase 0 summary so the user can reproduce the run. --- ## Phase 1: Discover model + classify kernels Delegate to the [`discover-models`](../discover-models/SKILL.md) skill, which produces the manifest's `kernels[]` array — each entry tagged with `phase1_status` (existing/new), `fi_status` (fi_supported/fi_missing/n-a), and `sgl_status` (sgl_integrated/sgl_missing/n-a). ```bash # In auto-discover mode /discover-models --discover --manifest tmp/onboard_{model_slug}_{date}.json # In single-model mode /discover-models --model-name {model_slug} --hf-repo-id {hf_repo_id} \ --manifest tmp/onboard_{model_slug}_{date}.json ``` Subsequent phases iterate over `kernels` and act based on the per-entry classification. --- ## Phase 2: Generate kernel definitions For each kernel with `phase1_status=new`, generate a Definition JSON and write it into `tmp/flashinfer-trace/definitions/{op_type}/`. ### 2b: fi_supported → trace-dump from a short SGLang pass (or manual fallback) Delegate to [`extract-kernel-definitions`](../extract-kernel-definitions/SKILL.md). When the kernel's FlashInfer API carries an `@flashinfer_api(trace=...)` decorator (i.e. `fi_trace_template=true` in the manifest), one short SGLang inference pass with `FLASHINFER_TRACE_DUMP=1` and `attention_backend="flashinfer"` produces complete Definition JSONs for every shape it touches — `axes`, `inputs`, `outputs`, `tags` (`fi_api:*`, `status:verified`), and `reference` are filled in by the dumper. For decorated kernels that the model didn't exercise (e.g. an unused page-size variant) or for FlashInfer APIs not yet decorated, the same skill falls back to manual extraction from sgl-cookbook + HF config. ```bash /extract-kernel-definitions --model-name {sglang_model_name} ``` Verify each new definition now exists: ```bash find tmp/flashinfer-trace/definitions/ -name "{definition_name}.json" ``` Update each kernel's `phase2_status=done` in the manifest. ### 2a: fi_missing → manual SGLang-sourced reference + file kernel-request issue When FlashInfer does not yet implement the kernel, the trace-dump path doesn't apply (no decorated API to fire on). Generate the definition manually with SGLang's vanilla forward as the reference (`extract-kernel-definitions` Path B), then file an issue against `flashinfer-ai/flashinfer`. Mark the definition with the `status:unverified` tag. ```bash gh issue create \ --repo flashinfer-ai/flashinfer \ --title "Kernel request: {op_type} for {model_name}" \ --label "enhancement,kernel-request" \ --body "$(cat <<'EOF' ## Kernel Request **Model**: {model_display_name} ({hf_repo_id}) **Op type**: {op_type} **Definition name**: {definition_name} ### Motivation This kernel is required for serving **{model_display_name}** with FlashInfer. A FlashInfer-Bench definition has been staged at: `tmp/flashinfer-trace/definitions/{op_type}/{definition_name}.json` (landing in the HuggingFace dataset PR for `flashinfer-ai/flashinfer-trace`) ### Kernel Parameters {formatted parameter table from definition axes} ### Reference Implementation A plain-PyTorch reference `run()` is available in the definition JSON above. The SGLang implementation is at: `python/sglang/srt/layers/{layer_path}` ### Requested Work - [ ] CUDA/Triton kernel implementation matching the definition schema - [ ] FlashInfer Python API (`flashinfer.{module}.{function}`) - [ ] Unit test in `tests/test_{op_type}.py` ### Links - HuggingFace dataset PR: (link once PR 2 is open) - SGLang model: `tmp/sglang/python/sglang/srt/models/{model_file}` - HuggingFace model: https://huggingface.co/{hf_repo_id} EOF )" ``` Record the issue URL in the manifest as `fi_issue_url` for that kernel and add it to the definition's `description`: ``` "description": "... See flashinfer-ai/flashinfer#<issue_number> for kernel implementation request." ``` **Do not proceed to Phase 3 for fi_missing kernels** — workload collection requires the FlashInfer kernel to exist. --- ## Phase 3: Workload collection Skip entirely for `fi_missing` kernels. ### 3a: Skip if workloads already exist ```bash ls tmp/flashinfer-trace/workloads/{op_type}/{definition_name}.jsonl 2>/dev/null ``` If the JSONL exists and is non-empty, mark `phase3_status=done` and skip. ### 3b: sgl_missing → submit SGLang integration PR first For kernels classified `sgl_missing`, wire `{fi_api}` into the appropriate SGLang layer file and open a PR against `sgl-project/sglang`. The change is typically small (import + conditional dispatch). ```python # In layers/attention/flashinfer_backend.py or equivalent try: from flashinfer.{module} import {KernelClass} FLASHINFER_{KERNEL}_AVAILABLE = True except ImportError: FLASHINFER_{KERNEL}_AVAILABLE = False def forward(...): if FLASHINFER_{KERNEL}_AVAILABLE and use_flashinfer: return {KernelClass}(...).run(...) else: return vanilla_forward(...) ``` ```bash cd tmp/sglang git checkout -b feat/flashinfer-{op_type}-integration-{model_slug} git add python/sglang/srt/layers/... git commit -m "feat: integrate FlashInfer {op_type} for {model_name} Wire {fi_api} into SGLang's FlashInfer backend to enable optimized {op_type} for {model_display_name}. Needed for: flashinfer-ai/flashinfer-bench (workload collection) FlashInfer API: {fi_api} " pre-commit run --all-files git push origin HEAD gh pr create \ --repo sgl-project/sglang \ --title "feat: integrate FlashInfer {op_type} for {model_name}" \ --body "$(cat <<'EOF' ## Summary - Integrates `{fi_api}` into SGLang's FlashInfer backend - Enables optimized `{op_type}` kernel for **{model_display_name}** - Required by flashinfer-ai/flashinfer-bench for workload collection ## Changes - `python/sglang/srt/layers/{path}`: add FlashInfer dispatch for {op_type} ## Test plan - [ ] SGLang unit test passes with `--attention-backend flashinfer` - [ ] Inference output matches non-FlashInfer baseline - [ ] Memory usage within expected bounds EOF )" ``` Record the SGLang PR URL on the kernel entry. Pause Phase 3 for that kernel until the PR merges; resume the run with `--phases 3,4` once it lands. ### 3c: sgl_integrated → run workload collection Delegate to [`collect-workloads`](../collect-workloads/SKILL.md): ```bash /collect-workloads \ --definition-names {definition_name} \ --model-name {model_name} \ --submit-pr false # PRs are submitted in Phase 4 ``` Verify outputs and update the manifest: ```bash ls tmp/flashinfer-trace/workloads/{op_type}/{definition_name}.jsonl ls tmp/flashinfer-trace/blob/workloads/{op_type}/{definition_name}/ ``` Mark `phase3_status=done` and record `workload_entries`. --- ## Phase 4: Submit PRs Delegate to [`submit-onboarding-prs`](../submit-onboarding-prs/SKILL.md). It creates the per-definition worktrees, spawns one agent per definition, and opens PR 2 (HuggingFace dataset) followed by PR 1 (flashinfer-bench coverage doc) for each. ```bash /submit-onboarding-prs --manifest tmp/onboard_{model_slug}_{date}.json ``` The skill writes back to the manifest's `phase4` block with the resulting PR URLs. The PR Review Checklist and Agent TASK.md template both live inside that skill — refer to `submit-onboarding-prs/SKILL.md` rather than duplicating them here. --- ## Run manifest The contract between skills. Stored at `tmp/onboard_{model_slug}_{date}.json`: ```json { "model_slug": "qwen3-235b-a22b", "hf_repo_id": "Qwen/Qwen3-235B-A22B", "date": "2026-04-27", "repo_shas": { "sglang": "abc1234", "flashinfer": "def5678", "sgl_cookbook": "ghi9012", "flashinfer_trace": "jkl3456" }, "kernels": [ { "definition_name": "gqa_paged_decode_h40_kv8_d128_ps1", "op_type": "gqa_paged", "phase1_status": "new", "fi_status": "fi_supported", "fi_trace_template": true, "sgl_status": "sgl_integrated", "phase2_status": "done", "phase2_method": "trace_dump", "phase3_status": "done", "workload_entries": 8 }, { "definition_name": "rope_with_cos_sin_cache_neox_style_d128_rd128", "op_type": "rope", "phase1_status": "new", "fi_status": "fi_supported", "fi_trace_template": false, "fi_trace_template_request_url": "https://github.com/flashinfer-ai/flashinfer/issues/3001", "sgl_status": "sgl_integrated",