OpenCompass C++ pass@1 on CodeCompass: 270 recent AtCoder, Codeforces and Nowcoder problems with SAGA-generated tests.
unassessed
| Category | coding |
|---|---|
| Subcategory | OpenCompass C++ pass@1 on 270 SAGA-tested contest problems |
| Page status | active |
| Metric | pass@1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 270 |
| Dataset licence | Apache-2.0 |
| Publisher | Shanghai AI Laboratory (OpenCompass); Xi'an Jiaotong University |
CodeCompass is a code-generation benchmark from Shanghai AI Laboratory's OpenCompass line. The model reads an English online-judge statement and must emit a self-contained C++ program that reads stdin and writes stdout. OpenCompass extracts a ```cpp fence, compiles it, and runs the problem's test cases. The problems are 270 AtCoder, Codeforces, and Nowcoder tasks from June 2024 onward, with SAGA-generated tests (paper: 50.54 cases per problem on average). The same paper also reports Python pass@1; the shipped OpenCompass config only builds a C++ dataset (abbr codecompass_gen_cpp). It is not CompassBench and not Ericsson's CodeCompass static-analysis tool.
Zero-shot generation. OpenCompass PromptTemplate round is the processed {prompt}: an expert-C++ system block plus [[Problem begin]]...[[Problem end]]. GenInferencer max_out_len 2048. Evaluator CodeCompassEvaluator, k_list=[1], timeout 15s (or the problem's Time Limit), 16 worker processes.
No model card in ModelSpec reports this benchmark yet.