1,899 tasks testing whether a model can learn new domain knowledge, rules or procedures from its own prompt and apply them; the best of ten frontier models solved only 23.7%.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | context learning: applying novel in-prompt knowledge, rule systems and procedures |
| Page status | active |
| Metric | rubric pass rate (binary, LLM-judged, all-or-nothing per task) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1899 |
| Dataset licence | Custom "evaluation-only" licence (Hugging Face lists it as "other"): the LICENSE file permits use, copying and redistribution of the dataset solely for evaluation, testing and benchmarking of models, and explicitly prohibits using it for training, fine-tuning, calibrating, distilling, adapting or any other form of parameter updating. |
| Publisher | Hunyuan Team, Tencent, and Fudan University |
CL-bench (OpenCompass folder name "CLBench") tests what its authors call context learning: whether a model can absorb new, task-specific material supplied directly in its prompt -- new domain-specific knowledge, rule systems, complex procedures, or laws derived from empirical data, all absent from pretraining -- and then apply that material correctly to solve a task grounded in it. Every one of its 1,899 tasks is built so the information needed to solve it sits entirely inside its accompanying context; the authors explicitly distinguish this from long-context benchmarks, which mainly test retrieval or reading comprehension over a large volume of text, and from ordinary in-context learning, where a model learns a simple pattern from a handful of demonstrations. Contexts span four main categories and 18 sub-categories (for example "Rule System Application" / "Game Mechanics"), and tasks are presented as multi-turn conversations in OpenAI chat format.
Each sample supplies a multi-turn conversation (a context, in system or user turns, followed by a task) and a list of rubric criteria the response must satisfy. The model produces a free-text response; there are no answer options.
No model card in ModelSpec reports this benchmark yet.