菜单
Validate the correctness and completeness of a FlashInfer Trace dataset. Use when checking dataset quality, verifying definitions/workloads/solutions/traces, debugging data issues, or preparing a dataset for release.
用 Codex 或 Claude 帮你安装 复制这段 Prompt，粘贴到 Codex、Claude 或其他助手里，让它检查 Skill 页面并帮你完成安装。
基于 SOC 职业分类
Auto-collect workloads from SGLang inference runs using FlashInfer logging API. Dumps tensors, sanitizes them according to kernel definitions, and submits PR to flashinfer-trace workload repo.
Discover candidate LLMs and produce a kernel inventory — required definitions, classified as existing/new and fi_supported/fi_missing — for onboarding. Use as Phase 1 of /onboard-model, or standalone to plan onboarding work.
Generate Definition JSON files for the flashinfer-trace HuggingFace dataset by harvesting them from a short SGLang inference pass (FlashInfer's @flashinfer_api(trace=...) dumper) — or, as a fallback, by manually transcribing the schema from SGLang sources when FlashInfer doesn't yet have a trace template. Use when adding a new model, extracting GPU kernels (MLA, MoE, GQA, RMSNorm, GEMM, GDN, RoPE, sampling), or filling gaps in the dataset.
End-to-end pipeline for discovering new LLMs with novel kernels and onboarding them into FlashInfer-Bench. Orchestrates repo updates, model discovery, kernel definition generation, workload collection, and PR submission.
Add pytest tests to validate reference implementations in the flashinfer-trace HuggingFace dataset against FlashInfer or SGLang ground truth. Use when validating kernel definitions, adding tests for new op_types, or verifying reference implementations are correct.
Clone SGLang, FlashInfer, sgl-cookbook, and flashinfer-trace repositories to tmp/. Use when setting up the project, preparing for kernel extraction, or when the user needs the source repositories.
| name | validate-dataset |
| description | Validate the correctness and completeness of a FlashInfer Trace dataset. Use when checking dataset quality, verifying definitions/workloads/solutions/traces, debugging data issues, or preparing a dataset for release. |
Run structural, schema, and coverage checks on a FlashInfer Trace dataset. Produces a JSON report and optional text summary.
# Full validation (including GPU benchmark)
/validate-dataset
# Skip GPU benchmark
/validate-dataset --disable-gpu
# Validate specific op_type or definition
/validate-dataset --op-types mla_paged --disable-gpu
/validate-dataset --definitions rmsnorm_h4096 --disable-gpu
--dataset: Dataset root (default: FIB_DATASET_PATH env var)--op-types: Filter by op_type (union with --definitions)--definitions: Filter by definition name--checks: Comma-separated subset of: layout,definition,workload,solution,trace,baseline,benchmark--disable-gpu: Skip benchmark checks (default: GPU enabled)--outputs: Comma-separated output targets: stdout,json,text (default: all three)--output-folder: Report output directory (default: <dataset>/reports/)| File | Purpose |
|---|---|
flashinfer_bench/data/validate.py | Core validation logic: validate_dataset() |
flashinfer_bench/data/validate_render.py | Report models and rendering: render_report() |
flashinfer_bench/cli/main.py | CLI entry: flashinfer-bench validate, flashinfer-bench validate-render |
Run validation via CLI:
flashinfer-bench validate --dataset tmp/flashinfer-trace --disable-gpu
Or call Python API directly:
from flashinfer_bench.data.validate import validate_dataset
report = validate_dataset(dataset="tmp/flashinfer-trace", disable_gpu=True)
Re-render an existing report:
flashinfer-bench validate-render reports/report-20260413-120000.json
| Category | What it checks |
|---|---|
layout | Duplicate definition names, directory structure, path-field consistency, blob file existence |
definition | Pydantic schema, reference Python syntax, axis references, descriptions, build reference (GPU) |
workload | Schema, axes > 0, shape inference, safetensors blobs, no solution/evaluation |
solution | Pydantic schema, entry_point format, sources, path-field consistency |
trace | Schema, must have solution+evaluation, solution existence, workload coverage |
baseline | Baseline solution/trace existence, build baseline solutions (GPU), PASSED status, workload coverage |
benchmark | GPU: run baseline + reference via Benchmark.run_all() (lightweight config) |
report-YYYYMMDD-HHMMSS.json/.txt) to prevent overwriting<dataset>/reports/ by defaultdocs/flashinfer-trace/validate.mdx for full report structure and examples