100 hand-written Python class-generation tasks testing whether a model can implement a whole class correctly, not just a single function like HumanEval.
unassessed
| Category | coding |
|---|---|
| Subcategory | class-level code generation |
| Page status | active |
| Metric | class-level Pass@k and method-level Pass@k |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 100 |
| Dataset licence | MIT (stated on both the GitHub repository and the Hugging Face dataset card) |
| Publisher | Fudan University, Shanghai, China |
ClassEval asks a model to implement an entire Python class from a skeleton, rather than a single function as HumanEval and MBPP do. Each of the 100 hand-written tasks gives a class name, description, constructor, fields and per-method signatures with docstrings, and the model must generate the bodies of several methods per class (412 methods total per the dataset card, 410 per the GitHub README) that may depend on each other, on class fields, or on library imports -- not just produce isolated, self-contained code. The benchmark evaluates three distinct generation strategies for producing the class: Holistic (generate the whole class at once), Incremental (method-by-method, each conditioned on previously generated methods), and Compositional (method-by-method, generated independently and then assembled).
Given a class skeleton (imports, class and method signatures, docstrings, and, depending on strategy, a natural-language instruction), the model generates the class body. Correctness is checked by executing the completed class against the item's own unit tests (33.1 test cases per class on average); Pass@k is computed separately at class granularity (all tests for the whole class must pass) and method granularity (only that method's own tests must pass).
No model card in ModelSpec reports this benchmark yet.