Run any Skill in Manus with one click

benchmark

Stars60

Forks10

UpdatedMay 31, 2026 at 08:57

Reference for the active benchmark harness — what it IS and how it behaves (the active benchmark is flashinfer-bench). Covers config.toml structure (`[solution]`/`[build]`/`[benchmark]` tables), the status enum, the workload / fresh-inputs model, reference-baseline + scoring mechanics, tolerance keys, TVM-FFI builder link limits, and the silent-skip cascade. Invoke whenever you decode a bench status string, hit an unfamiliar config.toml field, suspect "correctness passed but the headline is implausible", or need to know what's frozen before proposing a bench edit — do NOT guess field or status semantics. Bench commands and noise methodology (A/B compare, variance check, drift cancellation) live in the `bench` skill.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

TongmingLAIC

TongmingLAIC/AKO4X

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

File Explorer

2 files

SKILL.md

readonly

name

benchmark

description

Benchmark

The active benchmark harness — what this project's bench skill is a frontend for. Reference for what the benchmark is and how it behaves — distinct from the bench skill which covers how to drive it from the harness shim. The active benchmark is flashinfer-bench; this skill is the single home for its schema, status, scoring, and frozen-for-comparability behavior. Detailed body: benchmark.md.

When this skill applies:

Looking up a config.toml field you don't recognize.
Decoding a status string from bench output (COMPILE_ERROR, INCORRECT_NUMERICAL, etc.).
Reasoning about why correctness "passed" but the headline looks too good (silent-skip cascade ends here).
Checking what's frozen for bench comparability (scoring/baseline/tolerance) before proposing a harness edit.
Understanding the TVM-FFI link limit for C++/CUDA (no compile-time linking) and which kernel libraries are disallowed (cuBLAS / cuDNN) vs. allowed building blocks (CUTLASS, the DSLs) — see "Valid solution: write your own kernel".

More from this repository

same repository

bench

TongmingLAIC/AKO4X

Run performance benchmarks to get a verdict on whether a kernel change actually helped. Use --ab-compare for sub-1x deltas (drift cancels in same container), --variance-check for noise floor, and the subset modes (--first, --smoke; --extremes is modal-backend only) for compile-correctness probes only — those are NOT performance verdicts. The only authoritative source for "is this change real". Benchmark specifics (config.toml schema, status enum, scoring formula, baseline rule, fresh-inputs contract) live in the `benchmark` skill.

2026-05-3160

cpp

TongmingLAIC/AKO4X

Host-side C++ kernels (`[build].language = cpp`) — the `solution/{kernel.cpp, binding.py}` two-file TVM-FFI binding pattern, and when to reach for it (host-side orchestration of a kernel you write, or composing CUTLASS headers into it). Distinct from the `cuda` skill which covers `.cu` device kernels compiled by nvcc. Use when writing a host-side C++ kernel or deciding cpp-vs-cuda.

2026-05-3160

cuda

TongmingLAIC/AKO4X

CUDA C++ (.cu) kernel reference — TVM-FFI direct-export vs Python-binding entry points, the chevron-launch null-stream pitfall under CUDA-graph capture, the sm_100 fp8→bf16 cvt PTX gap, the load_inline name-cache trap, and the __launch_bounds__ register-spill lever. Also the single-source for generic per-call-overhead / CUDA-graph-capture / PDL theory (Waves-Per-SM decision table) regardless of DSL. Use when writing or debugging hand-written CUDA C++ kernels, or whenever you need generic CUDA-graph / sync-audit / PDL reasoning.

2026-05-3160

cute-dsl

TongmingLAIC/AKO4X

CuTe DSL reference — NVIDIA's CUTLASS Python DSL. Use whenever writing or debugging a CuTe DSL kernel — covers the `@cute.kernel` + `@cute.jit` + `.launch()` host pattern, `from_dlpack` tensor conversion, API-probing against the installed wheel, the launch-kwarg table, and the well-known graph-capture pitfall (CuTe's TVM-FFI stream binding does NOT pick up CUDA-graph capture mode — flashinfer-bench issue 414).

2026-05-3160

profiler-ncu

TongmingLAIC/AKO4X

Run NVIDIA Nsight Compute (NCU) per-kernel profiling for register pressure, occupancy, stall-reason, memory-throughput, and IPC analysis. Use BEFORE architecting an optimization fix — not only after — when a hypothesis about microarchitectural behavior needs verification. NCU `Duration` is NOT comparable to bench timing; use NCU for ratios only.

2026-05-3160

sanitizer

TongmingLAIC/AKO4X

Run NVIDIA compute-sanitizer (memcheck / racecheck / initcheck / synccheck). FIRST action on `INCORRECT_NUMERICAL` or flaky output — rolling back destroys the evidence; sanitizer often points at the exact line. Also useful for diagnosing race conditions in multi-block kernels and uninitialized-memory reads.

2026-05-3160

name

benchmark

description

Benchmark

When this skill applies:

Looking up a config.toml field you don't recognize.
Decoding a status string from bench output (COMPILE_ERROR, INCORRECT_NUMERICAL, etc.).
Reasoning about why correctness "passed" but the headline looks too good (silent-skip cascade ends here).
Checking what's frozen for bench comparability (scoring/baseline/tolerance) before proposing a harness edit.
Understanding the TVM-FFI link limit for C++/CUDA (no compile-time linking) and which kernel libraries are disallowed (cuBLAS / cuDNN) vs. allowed building blocks (CUTLASS, the DSLs) — see "Valid solution: write your own kernel".