A diagnostic coding test that renames NumPy into a fake library so a score reflects whether an agent internalized new APIs, not old knowledge or hard reasoning.
unassessed
| Category | coding |
|---|---|
| Subcategory | knowledge internalization / self-evolution diagnostic |
| Page status | active |
| Metric | accuracy (test-case + AST constraints) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1417 |
| Dataset licence | MIT |
| Publisher | THUNLP, Tsinghua University |
SE-Bench checks whether an agent can absorb a new library and later use it with no docs. The authors wrap 268 common NumPy functions as a package named zwc, with nonsense identifiers such as zwc.kocito. Inputs and outputs are ZWCArray objects so the model cannot call NumPy methods on arrays. Training items include the relevant docstring. Test items are ordinary coding problems without that docstring. Single-function tests check recall. Multi-function tests require composing at least three APIs. A base model scores 0% without docs, so failures are meant to be memory failures.
English problem statement in; the model writes Python that must call zwc APIs, pass hidden tests, and must not import numpy. Training may include API docs; the official test run does not.
No model card in ModelSpec reports this benchmark yet.