NVIDIA's benchmark of CUDA programming challenges - a model must write CUDA code that compiles with nvcc and passes a held-out test harness, spanning kernels, runtime APIs and GPU libraries.
unassessed
| Category | coding |
|---|---|
| Subcategory | CUDA / GPU kernel code generation |
| Page status | active |
| Metric | pass@1 (compiles and all hidden test cases pass) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 566 |
| Dataset licence | NVIDIA Evaluation Dataset License Agreement (data, custom licence, not OSI-standard); Apache 2.0 (evaluation-harness code) |
| Publisher | NVIDIA (NVIDIA Research) |
ComputeEval tests whether a model can write correct CUDA code from a natural-language specification. Each problem gives the model a prompt describing a GPU programming task, plus visible context files (headers defining the interface to implement, optional helper utilities) and the exact compiler command that will be used. Problems span seven domain groups: CUDA runtime (memory management, streams, kernel launch), CUDA kernels (shared memory, warp intrinsics, reductions, tensor cores), CCCL (Thrust, CUB, libcu++), cuBLAS, math libraries (cuSPARSE, cuSOLVER, cuFFT, cuRAND), cuDNN, and cuTile (Python tile-based kernels). Almost all current problems are C++ CUDA; the Python problem type exists in the schema but the shipped dataset contains none as of this page's research.
Given a prompt, interface header(s) and a fixed build command, the model must generate a CUDA source file. The solution is compiled with `nvcc` and run against a held-out test harness; a solution is scored correct only if compilation succeeds and every test case passes (exit code 0). Some problems also carry a `source_references` requirement (specific API calls or symbols that must appear in the solution) and an optional performance-benchmarking mode that times passing solutions against a reference implementation.
No model card in ModelSpec reports this benchmark yet.