Provides guidance for writing, optimizing, and benchmarking C++ CPU kernels with SIMD intrinsics (AVX2/AVX512) for the Hugging Face kernels ecosystem. Includes a two-phase workflow: Phase 1 correctness (generic → AVX2) and Phase 2 performance exploration (AVX512 with branching trial loop), runtime CPU dispatch, OpenMP threading, and brgemm integration for GEMM-heavy kernels.
Provides guidance for writing, optimizing, and benchmarking C++ CPU kernels with SIMD intrinsics (AVX2/AVX512) for the Hugging Face kernels ecosystem. Includes a two-phase workflow: Phase 1 correctness (generic → AVX2) and Phase 2 performance exploration (AVX512 with branching trial loop), runtime CPU dispatch, OpenMP threading, and brgemm integration for GEMM-heavy kernels.
This skill provides patterns and guidance for developing optimized C++ kernels targeting x86 CPUs (Intel Xeon and compatible processors) with AVX2 and AVX512 intrinsics. Kernels are compiled via kernel-builder and distributed through the Hugging Face kernels ecosystem.
Who runs these commands? , the agent — not a human. This is an autonomous loop: you write/edit the C++ kernel, build it, then run the scripts below as tools (via Bash) to check correctness, benchmark, and profile. You read each result, record it with , decide the next change from the Phase 2 decision tree, and repeat until you hit or run all .
You
trial_manager.py
early_stop_speedup
max_trials
Key Concepts (read before the Quick Start)
The commands use a few names that mean different things. They are not interchangeable:
Name (example)
What it is
Used by
baseline.py
The PyTorch reference implementation you optimize against. It is the ground truth for correctness and the speed reference for speedup. It must define get_inputs() and eitherget_reference_output()or a Model class (plus optional get_init_inputs()). You write this file (or it is given) before starting.
every script
my_rmsnorm
A trial-tree label — an arbitrary name you pick for this optimization task. trial_manager.py stores all attempts under trials/my_rmsnorm/. It is only a tracking ID.
trial_manager.py only
my_kernel
The installed Python package name — the build artifact produced by kernel-builder build + pip install. This is the importable module that contains your compiled kernel.
--kernel-package
my_kernel.rms_norm
An <package>.<function> path — the actual callable inside the installed package. Passed to --op to tell the benchmark/profiler which function to run.
--op
⚠️ --op means two different things depending on the script. In analyze_op.py, --op is a plain operation name (e.g. "rms_norm") used to look up compute/memory characteristics. In benchmark_cpu.py and cpu_profiler.py, --op is a package.function path (e.g. my_kernel.rms_norm) used to import and call your kernel. Same flag, different meaning — read each command below carefully.
Quick Start
Write a New CPU Kernel
The example below optimizes an RMSNorm kernel. The trial label is my_rmsnorm, the built package is my_kernel, and its function is my_kernel.rms_norm — keep these consistent across all six steps.
# 1. Analyze the target op. Here --op is an OPERATION NAME (looked up in the# knowledge base), not a package path.
python scripts/analyze_op.py --op "rms_norm" --shapes "1024x4096,2048x8192"# 2. Initialize trial tracking. Args: <trial-label> <baseline-file>.# Creates trials/my_rmsnorm/ and records baseline.py as the reference.
python scripts/trial_manager.py init my_rmsnorm baseline.py
# 3. Build the kernel package (produces the installable 'my_kernel' wheel).cd /path/to/my-kernel && kernel-builder build --release && pip install dist/*.whl --force-reinstall
# 4. Benchmark correctness + performance. Here --op is a PACKAGE.FUNCTION path.# Compares my_kernel.rms_norm against baseline.py (correctness + speedup).
python scripts/benchmark_cpu.py baseline.py --kernel-package my_kernel --op my_kernel.rms_norm
# 5. Profile with perf stat (same package.function path as step 4).
python scripts/cpu_profiler.py --kernel-package my_kernel --op my_kernel.rms_norm
# 6. Finalize: promote the best trial in trials/my_rmsnorm/ into output/.
python scripts/trial_manager.py finalize my_rmsnorm output/
Supported Hardware
ISA
Extensions
Key Instructions
Typical CPUs
AVX2
FMA, F16C
_mm256_fmadd_ps, _mm256_cvtph_ps
Most x86 CPUs (2013+)
AVX512
F, BF16, VL, DQ, BW, VBMI
_mm512_dpbf16_ps, _mm512_permutexvar_epi16
Intel Xeon
GEMM Acceleration: brgemm
For kernels that involve matrix multiplication (quantized GEMM, Flash Attention, MoE), large-M cases use at::native::cpublas::brgemm() — a PyTorch wrapper around oneDNN brgemm, which internally dispatches to AMX tile instructions on Intel Xeon (4th Gen+). Small-M cases (M ≤ 4 for bf16) fall back to hand-written tinygemm using AVX512 _mm512_dpbf16_ps. See brgemm_patterns.yaml for details.
Note: brgemm is NOT used in element-wise kernels (RMSNorm, activations, reductions). Those use AVX512 intrinsics directly.
When This Skill Applies
Use this skill when:
Writing C++ CPU kernels with SIMD intrinsics for the HF kernels ecosystem
Optimizing existing CPU kernels (e.g., adding AVX512 to a generic implementation)
Implementing Flash Attention or other attention kernels for CPU
Building kernels with kernel-builder that target backend = "cpu"
Two-Phase Optimization Workflow
CPU kernel development has two distinct phases with different strategies.
Configuration — Read config.yaml first
At the start of every session, read scripts/config.yaml. It controls:
max_trials — hard cap on Phase 2 optimization trials
early_stop_speedup — speedup vs PyTorch baseline to trigger early stop (default: 3.0)
perf_stat_enabled — if true, use perf stat for profiling (default)
vtune_enabled — if true, use VTune for detailed microarchitecture analysis
build_command — command to build the kernel package
Rules — Never Violate
ONLY modify C++ kernel files (.cpp, .hpp), torch_binding.cpp, and build.toml. Do NOT create benchmark or test scripts.
NEVER write custom timing code — ONLY use scripts/benchmark_cpu.py.
If a tool fails, STOP and report the error. Do NOT work around it with custom scripts.
Generated kernels must follow the runtime dispatch pattern with cpu_features.hpp — see references/runtime_dispatch.yaml.
Every kernel should have a generic ATen fallback that works on any CPU. If a specific path cannot have a meaningful fallback, use TORCH_CHECK(false, ...) with a clear error message.
Each SIMD tier (AVX2, AVX512) must be in a separate translation unit (.cpp file) with its own compiler flags in build.toml. Do NOT mix intrinsics from different ISA levels in the same file.
All SIMD implementations must handle edge cases (hidden_size not divisible by vector width).
AVX2 tier is optional — most CPU kernels go directly from generic fallback to AVX512. Only add AVX2 when it provides meaningful benefit for element-wise ops.
You MUST run all max_trials trials in Phase 2. Do NOT stop early due to plateau — the only valid early stop is speedup > early_stop_speedup.
perf stat hardware counters + optimization recommendations
Trial Manager
python scripts/trial_manager.py <command> ...
Trial tree management (init/save/result/status/best/finalize)
Benchmark discipline: Pin to a single NUMA node — numactl --cpunodebind=0 --membind=0 python scripts/benchmark_cpu.py .... See threading_patterns.yaml.
Phase 1: Correctness (Linear, No Branching)
Build the kernel tier by tier. Each tier must be correct before moving on.
Tier 0: Generic Fallback
Implement using PyTorch ATen ops only (no intrinsics).
This serves as the portable baseline that runs on any CPU.
Must produce results matching the PyTorch reference within tolerance.
Do NOT keep tuning the same knobs. Change the approach: switch algorithm path (tinygemm ↔ brgemm), change the fusion/blocking/data-layout strategy, or reconsider the dispatch heuristic. A different structure beats endless parameter sweeps.
Max trials reached
Stop — must run all max_trials from config.yaml
Optimization Search Space (Phase 2)
These tables are a starting menu of values seen in existing kernels, not an exhaustive recipe. Use them to seed trials, but when a branch plateaus, prefer a structurally different idea (algorithm, fusion, memory strategy) over sweeping these knobs further. See the try-harder tree in optimization_levels.yaml.
brgemm API usage, VNNI packing, tinygemm vs brgemm selection (GEMM kernels only)
references/memory_patterns.yaml
Prefetch, alignment, cache blocking
references/threading_patterns.yaml
OpenMP parallel patterns
references/dtype_optimizations.yaml
bf16/fp8/int8 handling and conversion on CPU
references/optimization_levels.yaml
Progressive L1→L5 optimization checklist + try-harder tree
references/optimization_strategies.md
Strategy reference, decision tree, checklist
references/workflow_details.md
Detailed trial loop workflow
references/huggingface-kernels-integration.md
Hub integration for CPU kernels
Core CPU Kernel Patterns
Runtime Dispatch (Required for All Kernels)
Every CPU kernel has its own cpu_features.hpp (in its own namespace) and dispatches at runtime. Most kernels dispatch as AVX512 → fallback (no AVX2 tier):
// my_kernel_cpu/cpu_features.hpp — each kernel has its OWN copynamespace my_kernel_cpu {
classCPUFeatures {
public:
staticboolhasAVX512BF16(){ /* CPUID + XCR0 checks */ }
staticboolhasAVX2(){ /* CPUID check */ }
// GEMM kernels also check: static bool hasAMX() { ... }
};
}
// my_kernel_cpu/my_kernel_cpu.cpp — dispatcher#include"cpu_features.hpp"#include"my_kernel_avx512.hpp"voidmy_kernel(torch::Tensor& out, const torch::Tensor& input, ...){
if (CPUFeatures::hasAVX512BF16()) {
avx512::my_kernel_impl(out, input, ...);
} else {
// ATen fallback — inline or in a separate _fallback.cpp
out = torch::some_aten_op(input, ...);
}
}
Note: Only rmsnorm has a three-tier dispatch (AVX512 → AVX2 → ATen). GEMM kernels skip AVX2. Flash-attn2 additionally requires AMX via hasAllRequiredFeatures().
Each SIMD tier is a separate [kernel.*] section with its own compiler flags. The include directive is required for header resolution:
[kernel.my_kernel_cpu]backend = "cpu"depends = ["torch"]
include = ["my_kernel_cpu"]
src = [
"my_kernel_cpu/my_kernel_cpu.cpp",
"my_kernel_cpu/my_kernel_cpu_torch.cpp",
"my_kernel_cpu/my_kernel_cpu.hpp",
"my_kernel_cpu/cpu_features.hpp",
]
[kernel.my_kernel_cpu_avx512]backend = "cpu"# Note: For GEMM kernels (e.g., flash-attn2, megablocks), you must also include "-mamx-tile", "-mamx-bf16", "-mamx-int8"cxx-flags = ["-mavx512f", "-mavx512bf16", "-mavx512vl", "-mavx512dq", "-mavx512bw", "-mavx512vbmi", "-mfma", "-mf16c", "-fopenmp"]
depends = ["torch"]
include = ["my_kernel_cpu"]
src = [
"my_kernel_cpu/my_kernel_avx512.cpp",
"my_kernel_cpu/my_kernel_avx512.hpp",
]
Note: Every section needs include = ["<kernel_dir>"] for header resolution. The _torch.cpp file bridges Python-facing declarations to the C++ dispatcher. AVX2 section is optional (only rmsnorm has one).
Zero-point: per-group (GPTQ), none/encoded in LUT (BnB), per-block (FP8)
Algorithm: tinygemm (small M, fused) vs brgemm (large M, unpack+BLAS)
Weight conversion: The C++ kernel expects a specific block-interleaved format, NOT raw checkpoint format. Each framework converts in its own repo:
GPTQ: transform_cpu() unpacks int32→uint8, reorders by g_idx, transposes to [N,K]; then convert_weight_packed_zp() repacks to [N,K/2] block-interleaved (BLOCK_N=32). Zeros unpacked to [groups,N] uint8. Scales to bf16. Done at first forward in GPTQModel repo.
BnB: _convert_weight_packed_for_cpu() unpacks uint8 nibbles→[N,K], repacks to [N,K/2] block-interleaved (same algo as GPTQ). Denests nested absmax. Transposes scales to [K/blocksize,N] bf16. Done at first forward in bitsandbytes repo.
Megablocks MoE: ops.convert_weight_packed() does transpose+VNNI pack. ops.convert_scale_packed() reorders scales. Cached via packed_weight=True.
VNNI Conversion (K/V Activations):
Flash Attention: pack_vnni() per tile per forward (K/V change every call, so caching is not possible).
xpu-kernels skill — the Intel XPU Triton skill this workflow was adapted from
Xe-Forge — the LLM-driven optimization framework the skill methodology originates from
Acknowledgments
The methodology of this skill — the YAML knowledge base, the benchmark/validation harnesses, and the branching trial-manager optimization loop — was adapted from the xpu-kernels skill built by a group of Intel AI researchers, the IntelLabs team behind Xe-Forge, where the methodology originates. Thanks to the original authors for a solid foundation to build on.
