Given a short Python function and either its input or its output, predict the other by reasoning about execution -- not by writing new code.
unassessed
| Category | coding |
|---|---|
| Subcategory | code execution reasoning (input/output prediction; no code generation involved) |
| Page status | active |
| Metric | pass@1 (also pass@5), reported separately for CRUXEval-I and CRUXEval-O |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 800 |
| Dataset licence | MIT |
| Publisher | MIT CSAIL and Meta AI |
CRUXEval tests whether a model can reason about what a piece of code actually does when it runs, rather than whether it can write new code from a description. Each of its 800 short Python functions (3-13 lines) comes with a verified input-output pair, producing two distinct tasks: CRUXEval-I (input prediction), where the model sees the function and its output and must produce an input that would generate it, and CRUXEval-O (output prediction), where the model sees the function and an input and must predict what it returns. Functions were synthetically generated and then filtered to need only simple, low-memory execution -- the kind a competent human programmer could trace by hand in about a minute -- so failures reflect gaps in code understanding rather than raw computational load.
Given Python source for a function `f`, plus either an input or the corresponding output, generate the missing side of the pair. Graded by execution: an input-prediction answer passes if `assert f(generated_input) == output` runs without error, and an output-prediction answer passes if `assert f(input) == generated_output` does.
No model card in ModelSpec reports this benchmark yet.