{
 "body": "\n## What it measures\n\nComputeEval tests CUDA programming ability directly: given a natural-language description of a GPU\ntask, visible interface headers, and the exact compiler invocation that will be used, a model must\nwrite CUDA source code that actually compiles and works. Problems are organised into seven domain\ngroups covering the practical range of CUDA development -- raw kernel and runtime work (memory\nmanagement, streams, launch configuration, warp intrinsics, tensor cores), Thrust/CUB/libcu++ via\nCCCL, and the major GPU libraries (cuBLAS, cuSPARSE, cuSOLVER, cuFFT, cuRAND, cuDNN), plus an\nemerging Python tile-based track (cuTile) that, as of this page's research, has no problems shipped\nyet despite existing in the schema. This makes ComputeEval closer in spirit to HumanEval or MBPP\nthan to a general reasoning benchmark, but targeting a systems-programming domain (CUDA C++, mostly)\nthat general code benchmarks do not exercise.\n\n## How it is scored\n\nA submitted solution is compiled with `nvcc` against the problem's fixed build command and then run\nagainst a held-out test harness; it counts as correct only if compilation succeeds with no errors\nand every test case exits 0. Some problems add a `source_references` check requiring specific CUDA\nAPI calls or symbols to appear in the solution, so a solution that compiles and passes tests by using\na different approach than intended can still be marked incorrect. The default reported metric is\nsingle-sample pass@1; the harness also supports generating multiple samples per problem at\ntemperature above zero and reducing with a `pass_at_k` estimator for an unbiased pass@k figure,\nplus an optional mode that benchmarks the execution time of passing solutions against a reference\nimplementation. There is no official NVIDIA-published leaderboard or paper reporting comparative\nmodel scores; the numbers available (noted below) come from a third-party research harness, not\nNVIDIA's own reporting.\n\n## Dataset and licence\n\nThe dataset is explicitly described as under active, frequent development rather than fixed at\nrelease. NVIDIA's own versioned releases went from 127 problems (2025.1) to 232 (2025.2) to 406\n(2025.3); counting the current default problem file directly gives 566 problems as of this page's\nresearch date, and individual problem metadata already tags a further \"2026-1\" release not yet\nreflected in the version table this page could confirm. Data is released under a custom \"NVIDIA\nEvaluation Dataset License Agreement,\" distinct from the Apache 2.0 licence covering the separate\nevaluation-harness code; problems are additionally organised into seven Hugging-Face-loadable domain\ngroups (cccl, cublas, cuda-kernels, cuda-runtime, cudnn, cutile, mathlibs). Problems were authored by\nNVIDIA engineers, in some cases with generative assistance from frontier models, and reviewed by\nhuman engineers before release.\n\n## Who publishes it\n\nComputeEval is published by NVIDIA (NVIDIA Corporation, credited via NVIDIA Research), created\nstarting April 2025. No accompanying research paper was found -- inspect_evals' own documentation\nstates explicitly that no paper or leaderboard exists for comparison -- so this page leaves the\n`paper` fields empty rather than inferring one. NVIDIA maintains the reference repository at\n`NVIDIA/compute-eval` on GitHub (147 stars, actively pushed to as recently as August 2026) and the\ndataset on Hugging Face, along with periodic \"Engineering Diaries\" technical write-ups documenting\ndevelopment decisions.\n\n## Lineage\n\nComputeEval has no predecessor or successor tracked in this repository. The name is generic enough\nthat this page checked specifically for other benchmarks sharing it: a GitHub and Hugging Face search\nfound no other project of comparable adoption using \"ComputeEval\" or \"compute-eval\" as its name --\nNVIDIA's repository (147 stars) is far ahead of any similarly named project found, and this\nrepository's own census sources (the inspect_evals task directory and the `nvidia/compute-eval`\ndataset) point at it unambiguously. No genuine naming collision was confirmed, unlike `r_bench`\nelsewhere in this batch.\n\n## Saturation and contamination\n\nComputeEval is open, not saturated: the only scores this page could confirm -- from inspect_evals'\nown evaluation report rather than an NVIDIA leaderboard -- are 45.3% (gpt-5-nano-2025-08-07) and\n60.3% (gpt-5.2-2025-12-11), both well below a ceiling. Because the problem set itself grows across\nversions (127 to 566-plus problems observed), scores against different dataset versions are not\ndirectly comparable even for an unchanged model.\n\nContamination risk is medium: a sample problem inspected directly for this page shows that the\npublic dataset row bundles the prompt, the complete hidden test harness, and the reference\n(baseline) solution together -- \"hidden\" describes what the evaluation harness withholds from the\nmodel's context during a run, not what is withheld from the published data itself, so reference\nsolutions are as exposed to web/code-scraping as any other public GitHub content. This is partly\noffset by the benchmark's continual growth, since newer problems have had less time to enter any\ngiven model's pretraining window than the original 2025.1 release.\n\n## How to run it\n\nThe reference implementation and Dockerfile live in `NVIDIA/compute-eval` on GitHub, requiring an\nNVIDIA GPU, CUDA Toolkit 12.0+, and `nvcc` in `PATH`. inspect_evals packages an independent\nimplementation as the `compute_eval` task, explicitly built to match NVIDIA's own prompt and system-\nmessage format for comparability, configurable via `dataset_version` (a specific dated release or\nthe latest default), `num_epochs`/`epoch_reducer` for pass@k, `temperature`, and an optional Docker\nsandbox for isolating generated-code execution (recommended, since the harness compiles and runs\nmachine-generated code). No lm-evaluation-harness, HELM, OpenCompass or BIG-bench implementation was\nconfirmed. Because the dataset version affects both problem count and difficulty mix, a reported\nComputeEval score should always be checked against which `dataset_version` produced it.\n\n## Reading the numbers\n\nA high ComputeEval score means a model can produce CUDA code that actually compiles with `nvcc` and\npasses a real test harness across a range of GPU programming domains -- a considerably stronger bar\nthan passing a general Python coding benchmark, since it requires correct memory management,\nsynchronization and (for library-backed problems) correct API usage in a domain with far less public\ntraining data than mainstream languages. It says nothing about code outside CUDA/GPU programming, and\na pass@1 figure alone does not capture solution quality or performance -- the harness's separate,\nopt-in performance-benchmarking mode is needed to check whether a passing solution is also fast.\nBecause both the dataset version and the sampling/pass@k configuration materially change the\nreported number, and because no official leaderboard exists to cross-check a claimed score against,\ntreat any single ComputeEval figure as provisional until you know which version and configuration\nproduced it.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "compute_eval",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "NVIDIA ComputeEval",
   "compute-eval"
  ],
  "category": "coding",
  "contamination": {
   "note": "Each publicly downloadable problem record bundles the prompt, the full hidden test harness, and a baseline (reference) solution together in the same JSON row -- confirmed by inspecting a sample row directly. \"Hidden\" in the documentation means withheld from the model's context during a normal evaluation run by harness convention, not withheld from anyone who downloads the dataset, so the reference solutions are exposed to the same web/code corpora a model's training data is drawn from. This is partially offset by the benchmark's continual growth: newer problems (dated as recently as October 2025 in the sample reviewed) have had less time to be scraped into any given model's training cutoff than the original 2025.1 release.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "NVIDIA Evaluation Dataset License Agreement (data, custom licence, not OSI-standard); Apache 2.0 (evaluation-harness code)",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 566,
   "size_note": "The dataset is explicitly under active, versioned development. Per NVIDIA's own release table: version 2025.1 shipped 127 problems, 2025.2 shipped 232, and 2025.3 shipped 406. Counting the current default/latest problems.jsonl directly (accessed 2026-09-08) gives 566 problems, confirming continued growth past the 406 documented in inspect_evals' release table; individual problem metadata (e.g. task CUDA/146, dated 2025-10-31) tags membership in a \"2026-1\" release not yet reflected in that table. Problems are also grouped by domain: cccl, cublas, cuda-kernels, cuda-runtime, cudnn, cutile and mathlibs, each independently loadable as a Hugging Face config.\n",
   "splits": "single 'eval' split per version/group config (default, 2025-1, 2025-2, 2025-3, 2026-1, and per-group configs); no train split",
   "url": "https://huggingface.co/datasets/nvidia/compute-eval"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 2, slice D",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "compute_eval",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "compute_eval",
  "last_updated": "2026-08",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "ComputeEval tests whether a model can write correct CUDA code from a natural-language specification. Each problem gives the model a prompt describing a GPU programming task, plus visible context files (headers defining the interface to implement, optional helper utilities) and the exact compiler command that will be used. Problems span seven domain groups: CUDA runtime (memory management, streams, kernel launch), CUDA kernels (shared memory, warp intrinsics, reductions, tensor cores), CCCL (Thrust, CUB, libcu++), cuBLAS, math libraries (cuSPARSE, cuSOLVER, cuFFT, cuRAND), cuDNN, and cuTile (Python tile-based kernels). Almost all current problems are C++ CUDA; the Python problem type exists in the schema but the shipped dataset contains none as of this page's research.\n",
  "metric": {
   "baseline_note": "The reference harness supports pass@k more generally via a num_epochs / epoch_reducer configuration (generate multiple solutions per problem at temperature > 0 and reduce with \"pass_at_k\"), but the default and most commonly reported configuration is single-sample pass@1. No official NVIDIA leaderboard or paper-reported score was found; the closest published numbers this page located are from inspect_evals' own evaluation report (not an NVIDIA-run number): gpt-5-nano-2025-08-07 scored 45.3% (stderr +/-2.5%) and gpt-5.2-2025-12-11 scored 60.3% (stderr +/-2.4%), both against the then-current dataset version. inspect_evals' own README states plainly \"no paper or leaderboard available for comparison.\"\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100.0,
   "name": "pass@1 (compiles and all hidden test cases pass)",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "ComputeEval",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "NVIDIA (NVIDIA Research)",
   "url": "https://github.com/NVIDIA/compute-eval"
  },
  "released": "2025-04",
  "repo_url": "https://github.com/NVIDIA/compute-eval",
  "saturation": {
   "as_of": "2025-12",
   "note": "60.3% (gpt-5.2-2025-12-11, per inspect_evals' own evaluation report) leaves clear headroom and is a third-party research-harness run, not an NVIDIA-published leaderboard figure -- no independent, maintained leaderboard was found. Because the problem set itself keeps growing (127 to 232 to 406 to 566-plus problems across versions), scores from different dataset versions are not directly comparable even for the same model.\n",
   "status": "open",
   "top_score": 60.3
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "NVIDIA/compute-eval GitHub repository (README, problem structure, licence)",
    "url": "https://github.com/NVIDIA/compute-eval"
   },
   {
    "accessed": "2026-09-08",
    "title": "nvidia/compute-eval dataset card, Hugging Face",
    "url": "https://huggingface.co/datasets/nvidia/compute-eval"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals compute_eval task README (parameters, scoring, evaluation report)",
    "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/compute_eval"
   }
  ],
  "status": "active",
  "subcategory": "CUDA / GPU kernel code generation",
  "summary": "NVIDIA's benchmark of CUDA programming challenges - a model must write CUDA code that compiles with nvcc and passes a held-out test harness, spanning kernels, runtime APIs and GPU libraries.",
  "tags": [
   "coding",
   "cuda",
   "gpu",
   "code-generation",
   "nvidia"
  ],
  "task_format": "Given a prompt, interface header(s) and a fixed build command, the model must generate a CUDA source file. The solution is compiled with `nvcc` and run against a held-out test harness; a solution is scored correct only if compilation succeeds and every test case passes (exit code 0). Some problems also carry a `source_references` requirement (specific API calls or symbols that must appear in the solution) and an optional performance-benchmarking mode that times passing solutions against a reference implementation.\n"
 }
}