A model rewrites PyTorch workloads as GPU kernels; fast_p scores the share that are both correct and at least p times faster than the PyTorch baseline.
unassessed
| Category | coding |
|---|---|
| Subcategory | GPU kernel generation from PyTorch reference code, scored on both functional correctness and runtime speedup |
| Page status | active |
| Metric | fast_p (share of tasks both functionally correct and at least p times faster than the PyTorch reference); fast_1 and fast_2 are the thresholds most commonly reported, fast_0 reduces to plain correctness |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 270 |
| Dataset licence | MIT (GitHub repository LICENSE file and the Hugging Face dataset listing); the arXiv paper text itself is separately licensed CC BY 4.0. |
| Publisher | Stanford University (Scaling Intelligence Lab); Princeton University |
KernelBench asks a model to replace a PyTorch reference implementation -- a single operator, a small fused pattern, or a full neural network architecture -- with a custom GPU kernel that computes the same result, faster. Problems are organized into four difficulty levels: Level 1 (100 single-kernel operators such as convolutions, matrix multiplies and normalization layers), Level 2 (100 simple fusion patterns, such as a convolution followed by a bias add and a ReLU, where fusing operations into one kernel should beat running them separately), Level 3 (50 full model architectures such as MobileNet, VGG, MiniGPT and Mamba, requiring end-to-end kernel optimization), and Level 4 (20 Hugging Face model architectures, added in a July 2025 update). The default backend is raw CUDA, but the harness also supports Triton, CUTE, TileLang, ThunderKittens and HIP (for AMD GPUs). The benchmark exercises both a model's ability to write functionally correct low-level GPU code and its ability to make that code fast -- these are graded separately, not folded into one pass/fail check.
A PyTorch nn.Module reference implementation in; a replacement implementation that calls a custom GPU kernel out. The generated kernel is compiled and executed, then checked for correctness against the reference on randomized inputs and timed against it for speed, inside a Docker sandbox.
No model card in ModelSpec reports this benchmark yet.