benchmark-kernel

# Tutorial: Benchmarking FlashInfer Kernels This tutorial shows you how to accurately benchmark FlashInfer kernels. ## Goal Measure the performance of FlashInfer kernels: - Get accurate GPU kernel execution time - Compare multiple backends (FlashAttention2/3, cuDNN, CUTLASS, TensorRT-LLM) - Generate reproducible benchmark results - Save results to CSV for analysis ## Timing Methods FlashInfer supports two timing methods: 1. **CUPTI (Preferred)**: Hardware-level profiling for most accurate GPU kernel time - Measures pure GPU compute time without host-device overhead - Requires `cupti-python >= 13.0.0` (CUDA 13+) 2. **CUDA Events (Fallback)**: Standard CUDA event timing - Automatically used if CUPTI is not available - Good accuracy, slight overhead from host synchronization **The framework automatically uses CUPTI if available, otherwise falls back to CUDA events.** > **Autotuner timing (separate from the benchmark framework above).** The > `AutoTuner`'s internal per-tactic timing has its own selector, > `FLASHINFER_AUTOTUNE_TIMER`: `globaltimer` forces the GPU `%globaltimer` > register, `cuda_event` forces `cudaEvent`, and unset/auto uses `%globaltimer` > only when Confidential Computing (CC) is detected. Under CC > `cudaEventElapsedTime` is unreliable (can go negative), which would corrupt > tactic ranking — the globaltimer path avoids that. CC auto-detection can be > overridden with `FLASHINFER_CONFIDENTIAL_COMPUTE=0/1`. (Full env-var reference > in `CLAUDE.md`.) ## Installation ### Install CUPTI (Recommended) For the most accurate benchmarking: ```bash pip install -U cupti-python ``` **Requirements**: CUDA 13+ (CUPTI version 13+) ### Without CUPTI If you don't install CUPTI, the framework will: - Print a warning: `CUPTI is not installed. Falling back to CUDA events.` - Automatically use CUDA events for timing - Still provide good benchmark results ## Method 1: Using flashinfer_benchmark.py (Recommended) ### Step 1: Choose Your Test Routine Available routines: - **Attention**: `BatchDecodeWithPagedKVCacheWrapper`, `BatchPrefillWithPagedKVCacheWrapper`, `BatchPrefillWithRaggedKVCacheWrapper`, `BatchMLAPagedAttentionWrapper`, `trtllm_batch_decode_sparse_mla_dsv4` - **GEMM**: `bmm_fp8`, `gemm_fp8_nt_groupwise`, `group_gemm_fp8_nt_groupwise`, `mm_fp4` - **MOE**: `trtllm_fp4_block_scale_moe`, `trtllm_fp8_block_scale_moe`, `trtllm_fp8_per_tensor_scale_moe`, `cutlass_fused_moe` ### Step 2: Run a Single Benchmark Example - Benchmark decode attention: ```bash # CUPTI will be used automatically if installed python benchmarks/flashinfer_benchmark.py \ --routine BatchDecodeWithPagedKVCacheWrapper \ --backends fa2 fa2_tc cudnn \ --page_size 16 \ --batch_size 32 \ --s_qo 1 \ --s_kv 2048 \ --num_qo_heads 32 \ --num_kv_heads 8 \ --head_dim_qk 128 \ --head_dim_vo 128 \ --q_dtype bfloat16 \ --kv_dtype bfloat16 \ --num_iters 30 \ --dry_run_iters 5 \ --refcheck \ -vv ``` Example - Benchmark FP8 GEMM: ```bash python benchmarks/flashinfer_benchmark.py \ --routine bmm_fp8 \ --backends cudnn cublas cutlass \ --batch_size 256 \ --m 1 \ --n 1024 \ --k 7168 \ --input_dtype fp8_e4m3 \ --mat2_dtype fp8_e4m3 \ --out_dtype bfloat16 \ --refcheck \ -vv \ --generate_repro_command ``` **Timing behavior:** - ✅ If CUPTI installed: Uses CUPTI (most accurate) - ⚠️ If CUPTI not installed: Automatically falls back to CUDA events with warning - 🔧 To force CUDA events: Add `--use_cuda_events` flag ### Step 3: Understand the Output ``` [INFO] FlashInfer version: 0.6.0 [VVERBOSE] gpu_name = 'NVIDIA_H100_PCIe' [PERF] fa2 :: median time 0.145 ms; std 0.002 ms; achieved tflops 125.3 TFLOPs/sec; achieved tb_per_sec 1.87 TB/sec [PERF] fa2_tc :: median time 0.138 ms; std 0.001 ms; achieved tflops 131.5 TFLOPs/sec; achieved tb_per_sec 1.96 TB/sec [PERF] cudnn :: median time 0.142 ms; std 0.001 ms; achieved tflops 127.8 TFLOPs/sec; achieved tb_per_sec 1.91 TB/sec ``` **Key metrics:** - **median time**: Median kernel execution time (lower is better) - **std**: Standard deviation (lower means more consistent) - **achieved tflops**: Effective TFLOPS throughput - **achieved tb_per_sec**: Memory bandwidth utilization ### Step 4: Run Batch Benchmarks Create a test list file `my_benchmarks.txt`: ```bash --routine BatchDecodeWithPagedKVCacheWrapper --backends fa2 cudnn --page_size 16 --batch_size 32 --s_kv 2048 --num_qo_heads 32 --num_kv_heads 8 --head_dim_qk 128 --head_dim_vo 128 --routine BatchDecodeWithPagedKVCacheWrapper --backends fa2 cudnn --page_size 16 --batch_size 64 --s_kv 4096 --num_qo_heads 32 --num_kv_heads 8 --head_dim_qk 128 --head_dim_vo 128 --routine bmm_fp8 --backends cudnn cutlass --batch_size 256 --m 1 --n 1024 --k 7168 --input_dtype fp8_e4m3 --mat2_dtype fp8_e4m3 --out_dtype bfloat16 ``` Run all tests: ```bash python benchmarks/flashinfer_benchmark.py \ --testlist my_benchmarks.txt \ --output_path results.csv \ --generate_repro_command \ --refcheck ``` Results are saved to `results.csv` with all metrics and reproducer commands. ### Step 5: Common Flags | Flag | Description | Default | |------|-------------|---------| | `--num_iters` | Measurement iterations | 30 | | `--dry_run_iters` | Warmup iterations | 5 | | `--refcheck` | Verify output correctness | False | | `--allow_output_mismatch` | Continue on mismatch | False | | `--use_cuda_events` | Force CUDA events (skip CUPTI) | False | | `--no_cuda_graph` | Disable CUDA graph | False | | `-vv` | Very verbose output | - | | `--generate_repro_command` | Print reproducer command | False | | `--case_tag` | Tag for CSV output | None | ## Method 2: Using bench_gpu_time() in Python For custom benchmarking in your own code: ### Step 1: Write Your Benchmark Script ```python import torch from flashinfer.testing import bench_gpu_time # Setup your kernel def my_kernel_wrapper(q, k, v): # Your kernel call here return output # Create test inputs device = torch.device("cuda") q = torch.randn(32, 8, 128, dtype=torch.bfloat16, device=device) k = torch.randn(2048, 8, 128, dtype=torch.bfloat16, device=device) v = torch.randn(2048, 8, 128, dtype=torch.bfloat16, device=device) # Benchmark - CUPTI preferred, CUDA events if CUPTI unavailable median_time, std_time = bench_gpu_time( my_kernel_wrapper, args=(q, k, v), enable_cupti=True, # Prefer CUPTI, fallback to CUDA events num_iters=30, # Number of iterations dry_run_iters=5, # Warmup iterations ) print(f"Kernel time: {median_time:.3f} ms ± {std_time:.3f} ms") # Calculate FLOPS if you know the operation count flops = ... # Your FLOP count tflops = (flops / 1e12) / (median_time / 1000) print(f"Achieved: {tflops:.2f} TFLOPS/sec") ``` **Note**: If CUPTI is not installed, you'll see a warning and the function will automatically use CUDA events instead. ### Step 2: Run Your Benchmark ```bash python my_benchmark.py ``` Output with CUPTI: ``` Kernel time: 0.145 ms ± 0.002 ms Achieved: 125.3 TFLOPS/sec ``` Output without CUPTI (automatic fallback): ``` [WARNING] CUPTI is not installed. Try 'pip install -U cupti-python'. Falling back to CUDA events. Kernel time: 0.147 ms ± 0.003 ms Achieved: 124.1 TFLOPS/sec ``` ### Step 3: Advanced Options ```python # Cold L2 cache benchmarking (optional) median_time, std_time = bench_gpu_time( my_kernel, args=(x, y), enable_cupti=True, # Will use CUDA events if CUPTI unavailable cold_l2_cache=True, # Flush L2 or rotate buffers automatically num_iters=30 ) # Force CUDA events (skip CUPTI even if installed) median_time, std_time = bench_gpu_time( my_kernel, args=(x, y), enable_cupti=False, # Explicitly use CUDA events num_iters=30 ) ``` ## Troubleshooting ### CUPTI Warning Message **Warning**: `CUPTI is not installed. Falling back to CUDA events.` **What it means**: CUPTI is not available, using CUDA events instead **Impact**: Less accurate for very fast kernels (5-50 us) due to synchronization overhead, but becomes negligible for longer-running kernels **Solution (optional)**: Install CUPTI for best accuracy: ```bash pip install -U cupti-python ``` If installation fails, check: - CUDA version >= 13 - Compatible `cupti-python` version **You can still run benchmarks without CUPTI** - the framework handles this automatically. ### Inconsistent Results **Problem**: Large standard deviation or varying results **Solutions**: 1. **Increase warmup iterations**: ```bash --dry_run_iters 10 ``` 2. **Increase measurement iterations**: ```bash --num_iters 50 ``` 3. **Use cold L2 cache** (in Python): ```python bench_gpu_time(..., rotate_buffers=True) ``` 4. **Disable GPU boost** (advanced): ```bash sudo nvidia-smi -lgc <base_clock> ``` ### Reference Check Failures **Error**: `[ERROR] Output mismatch between backends` **What it means**: Different backends produce different results **Solutions**: 1. **Allow mismatch and continue**: ```bash --allow_output_mismatch ``` 2. **Check numerical tolerance**: Some backends use different precisions (FP32 vs FP16) 3. **Investigate the difference**: ```bash -vv # Very verbose mode shows tensor statistics ``` ### Backend Not Supported **Error**: `[WARNING] fa3 for routine ... is not supported on compute capability X.X` **Solution**: Check the backend support matrix in `benchmarks/README.md` or remove that backend from `--backends` list ## Best Practices 1. **Install CUPTI for best accuracy** (but not required): ```bash pip install -U cupti-python ``` 2. **Use reference checking** to verify correctness: ```bash --refcheck ``` 3. **Use verbose mode** to see input shapes and dtypes: ```bash -vv ``` 4. **Generate reproducer commands** for sharing results: ```bash --generate_repro_command ``` 5. **Run multiple iterations** for statistical significance: ```bash --num_iters 30 --dry_run_iters 5 ``` 6. **Save results to CSV** for later analysis: ```bash --output_path results.csv ``` 7. **Compare multiple backends** to find the best: ```bash --backends fa2 fa3 cudnn cutlass ``` ## Quick Examples ### Decode Attention (H100) ```bash python benchmarks/flashinfer_benchmark.py \ --routine BatchDecodeWithPagedKVCacheWrapper \ --backends fa2 fa2_tc cudnn trtllm-gen \ --page_size 16 --batch_size 128 --s_kv 8192 \ --num_qo_heads 64 --num_kv_heads 8 \ --head_dim_qk 128 --head_dim_vo 128 \ --refcheck -vv --generate_repro_command ``` ### Prefill Attention (Multi-head) ```bash python benchmarks/flashinfer_benchmark.py \ --routine BatchPrefillWithRaggedKVCacheWrapper \ --backends fa2 fa3 cudnn cutlass \ --batch_size 16 --s_qo 1024 --s_kv 1024 \ --num_qo_heads 128 --num_kv_heads 128 \ --head_dim_qk 192 --head_dim_vo 128 \ --causal --random_actual_seq_len \ --q_dtype bfloat16 --kv_dtype bfloat16 \ --refcheck -vv ``` ### FP8 GEMM (Batched) ```bash python benchmarks/flashinfer_benchmark.py \ --routine bmm_fp8 \ --backends cudnn cublas cutlass \ --batch_size 256 --m 1 --n 1024 --k 7168 \ --input_dtype fp8_e4m3 --mat2_dtype fp8_e4m3 \ --out_dtype bfloat16 \ --refcheck -vv ``` ### MOE (DeepSeek-style routing) ```bash python benchmarks/flashinfer_benchmark.py \ --routine trtllm_fp8_block_scale_moe \ --backends trtllm \ --num_tokens 1024 --hidden_size 5120 \ --intermediate_size 13824 --num_experts 256 \