| name | validate-dataset |
| description | Validate the correctness and completeness of a FlashInfer Trace dataset. Use when checking dataset quality, verifying definitions/workloads/solutions/traces, debugging data issues, or preparing a dataset for release. |
Validate Dataset
Run structural, schema, and coverage checks on a FlashInfer Trace dataset. Produces a JSON report and optional text summary.
Usage
/validate-dataset
/validate-dataset --disable-gpu
/validate-dataset --op-types mla_paged --disable-gpu
/validate-dataset --definitions rmsnorm_h4096 --disable-gpu
Parameters
--dataset: Dataset root (default: FIB_DATASET_PATH env var)--op-types: Filter by op_type (union with --definitions)--definitions: Filter by definition name--checks: Comma-separated subset of: layout,definition,workload,solution,trace,baseline,benchmark--disable-gpu: Skip benchmark checks (default: GPU enabled)--outputs: Comma-separated output targets: stdout,json,text (default: all three)--output-folder: Report output directory (default: <dataset>/reports/)
Implementation
Files
| File | Purpose |
|---|
flashinfer_bench/data/validate.py | Core validation logic: validate_dataset() |
flashinfer_bench/data/validate_render.py | Report models and rendering: render_report() |
flashinfer_bench/cli/main.py | CLI entry: flashinfer-bench validate, flashinfer-bench validate-render |
Steps
-
Run validation via CLI:
flashinfer-bench validate --dataset tmp/flashinfer-trace --disable-gpu
-
Or call Python API directly:
from flashinfer_bench.data.validate import validate_dataset
report = validate_dataset(dataset="tmp/flashinfer-trace", disable_gpu=True)
-
Re-render an existing report:
flashinfer-bench validate-render reports/report-20260413-120000.json
Check Categories
| Category | What it checks |
|---|
layout | Duplicate definition names, directory structure, path-field consistency, blob file existence |
definition | Pydantic schema, reference Python syntax, axis references, descriptions, build reference (GPU) |
workload | Schema, axes > 0, shape inference, safetensors blobs, no solution/evaluation |
solution | Pydantic schema, entry_point format, sources, path-field consistency |
trace | Schema, must have solution+evaluation, solution existence, workload coverage |
baseline | Baseline solution/trace existence, build baseline solutions (GPU), PASSED status, workload coverage |
benchmark | GPU: run baseline + reference via Benchmark.run_all() (lightweight config) |
Report Output
- JSON and text reports are timestamped (
report-YYYYMMDD-HHMMSS.json/.txt) to prevent overwriting
- Reports go to
<dataset>/reports/ by default
- See
docs/flashinfer-trace/validate.mdx for full report structure and examples
See Also
