A 32-challenge BIG-bench task that asks a model to complete a Python function, then compiles and unit-tests the body in a RestrictedPython sandbox.
unassessed
| Category | coding |
|---|---|
| Subcategory | BIG-bench programmatic Python function-completion with sandboxed unit tests (32 challenges) |
| Page status | unknown |
| Metric | correct (mean fraction of unit tests passed); compiled is also reported |
| Direction | higher_is_better |
| Dataset size | 32 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
python_programming_challenge shows a comment plus a function signature and asks the model to write the indented body. The harness concatenates signature and body, then runs the result against generated test cases in a RestrictedPython sandbox with a one-second timeout. Authors Laria Reynolds, Kyle McDonell, Nick Cameron, Joseph Jones, Michael Ivantitskiy and John Balis wrote 32 challenges: 7 very easy, 14 easy, 4 medium and 7 hard. The README intro lists medium (14); the later challenge list and challenge_definitions.py both give 4 medium. The auto-generated header only reports 32 free-text queries. The task is English plus Python, zero-shot, and programmatic rather than a fixed JSON list.
Free-response code generation. The model is queried once per challenge (max 500 tokens). Scoring uses compiled-success rate and mean test-case accuracy. A canary GUID is embedded.
No model card in ModelSpec reports this benchmark yet.