OpenCompass's LCBench2023: 581 LeetCode weekly-contest problems (2022-2023) in English and Chinese, scored pass@1 by executing generated code against each problem's tests.
unassessed
| Category | coding |
|---|---|
| Subcategory | bilingual (English/Chinese) code generation from LeetCode weekly-contest problems, scored by test execution |
| Page status | active |
| Metric | pass@1 (share of problems solved on a single sampled generation, by direct test execution); pass@10 and pass@100 are also defined via a repeated-sampling config |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1162 |
| Dataset licence | Not stated separately for the problem data. The OpenCompass repository itself is Apache-2.0 (confirmed from its LICENSE file); the underlying problem statements and test cases originate from LeetCode's weekly programming contests, a proprietary third-party source OpenCompass does not separately license. |
| Publisher | OpenCompass (Shanghai AI Laboratory) |
The census hint for this id names an OpenCompass source directory, and opening it shows "lcbench" here is LCBench2023, a code-generation benchmark built from LeetCode weekly-contest problems -- not a long-context benchmark, and not the unrelated AutoML "LCBench" learning-curve dataset that also exists under the same short name (see Lineage). LCBench2023 gives a model a competitive-programming problem statement, drawn from LeetCode weekly contests held in 2022 and 2023, plus a set of example test assertions the solution must satisfy, and asks for a Python function that solves it. The same 581 problems ship in two parallel versions, one with the problem statement in English and one in Chinese, so the same underlying coding task can be compared across the two prompt languages.
A LeetCode-style problem statement plus example assert test cases in (English or Chinese version); the model returns Python code in a fenced code block, which is executed against the problem's tests in a sandboxed subprocess with a per-attempt timeout, and classified as pass, timeout, wrong_answer or failed.
No model card in ModelSpec reports this benchmark yet.