LiveCodeBench

LiveCodeBench scores code generation, self-repair, test-output prediction and code execution on dated competitive-programming problems, filterable by a model's training cutoff.

Also known as: LCB, livecodebench

unassessed

What it measures

LiveCodeBench evaluates a model on competitive-programming problems collected continuously from LeetCode, AtCoder and Codeforces, going beyond plain code generation to also test self-repair (fixing a wrong solution given feedback), test output prediction (predicting what a given piece of code outputs) and code execution (simulating running code by hand). Because every problem is tagged with its original release date, a reader can score a model only on problems released after that model's training cutoff, directly addressing whether high scores reflect memorised solutions rather than genuine problem-solving.

Task format

For code generation, the model is given a natural-language problem statement (as posed on the source contest site) and must produce a working solution, evaluated against the contest's own or reconstructed test cases. The other three scenarios reuse the same problem pool but change what the model is asked to produce: a corrected solution given a failing one and error feedback (self-repair), the printed output of a given program on given input (test output prediction), or the result of executing a given snippet by reasoning about it directly (code execution).

Models reporting this benchmark

Data

This page as JSON · Edit on GitHub