1,140 Python tasks that require chaining calls across 139 real libraries, testing whether a model can use diverse tools correctly rather than write self-contained algorithmic code.
unassessed
| Category | coding |
|---|---|
| Subcategory | function-calling / library-use code generation |
| Page status | active |
| Metric | Pass@1 (including a "calibrated" variant that accounts for omitted setup code) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1140 |
| Dataset licence | Apache-2.0 |
| Publisher | BigCode project (multi-institution community collaboration) |
BigCodeBench asks a model to write a Python function that correctly uses one or more calls into real, popular libraries (data processing, visualization, networking, cryptography and more) to satisfy either a structured docstring (the Complete split) or a condensed natural-language instruction (the Instruct split). This targets a different skill than algorithmic benchmarks like HumanEval or MBPP: knowing which library function to call, with which arguments, and how to combine several such calls correctly, rather than implementing logic from scratch in the standard library alone.
Given a function signature with a docstring (Complete) or a natural-language instruction (Instruct), the model generates a Python function body; the generated code is executed against an average of 5.6 unit test cases per task with about 99% branch coverage.
No model card in ModelSpec reports this benchmark yet.