{
 "body": "\n## What it measures\n\nSE-Bench asks whether an agent can learn a new Python API and later use it with no documentation. The library is a NumPy wrapper named zwc. Each function has a nonsense name. Arrays are wrapped so `.mean()`-style shortcuts fail. Training problems include the matching docstring. Official tests hide that docstring.\n\nTasks are easy if the mapping is known. The same items are solved at high pass@64 with plain NumPy. They score 0% on zwc with no docs. Multi-function tests need at least three APIs, so the score is not only one-function memorization.\n\n## How it is scored\n\nA run is correct only if tests pass, AST analysis ties the return value to zwc, and numpy is not imported. The headline number is accuracy on that binary rule. Table 1 uses pass@64 to check the design. Table 2 uses the mean of five rollouts.\n\nOn Qwen3-8B, Closed-SFT-RL reached 54.4% single-function and 17.9% multi-function. Closed-SFT alone reached 39.6% and 11.6%. Open-SFT, Open-RL, Closed-RL, and Absolute-Zero scored 0.0% on both splits. Expel memory search reached 47.1% and 15.5%, the best memory baseline. With docs in context and no training, Qwen3-8B pass@64 was 85.3% / 70.5%, below the NumPy upper bound of 97.4% / 93.6% because models still hallucinate NumPy names.\n\n## Dataset and licence\n\nAfter filtering, there are 1,417 tasks: 718 train, 259 single-function test, 440 multi-function test. Hugging Face row counts match. Claude-4.5-sonnet wrote NumPy-form problems. Gemini-2.5-Pro rewrote docs into zwc. A task is kept only if Qwen3-Coder-480B, Gemini-2.5-Pro, and GPT-OSS-120B all solve the NumPy form. Humans checked a 10% sample; all sampled items were valid. The GitHub file `LICENCE.md` and the Hub tag are MIT. Test JSONL includes test cases, so the eval set is public.\n\n## Who publishes it\n\nJiarui Yuan, Tailin Jin, Weize Chen, and Zeyuan Liu (Tsinghua / THUNLP) posted v1 on 4 February 2026 and v2 on 9 May 2026. The repo citation also lists Zhiyuan Liu and Maosong Sun. Code is thunlp/SE-Bench. Data is jintailin/SE-Bench. No public leaderboard was found.\n\n## Lineage\n\nThis is not a general coding contest. [HumanEval](humaneval.md) and [SWE-bench](swe_bench.md) measure writing or patching known languages. SE-Bench hides the language of the library on purpose. It is also not Princeton SAgE. Use it as a diagnostic for self-evolution methods, not as a drop-in coding score.\n\n## Saturation and contamination\n\nThe best trained Qwen3-8B result is still 54.4% / 17.9%. Multi-function accuracy is low. The design zero-shot floor is 0%. Names are random, so classic web-scrape contamination of NumPy docs should not help unless the model saw this release. Once the JSONL is in training data, the diagnostic is spent.\n\n## How to run it\n\nInstall thunlp/SE-Bench, load `jintailin/SE-Bench`, and follow the README: train only on `datasets/train/`, then evaluate `single_test` and `multiple_test` without docs. Serve the model, roll out with `query_only.py`, start the Docker sandbox, and run `filter_correct_trajectory.py`. Custom rollouts need `query`, `response`, `test_cases`, and `right_exe_result`. Open versus closed training is a real protocol split: docs may be present when collecting traces but must be stripped for Closed-SFT. No standard harness task was found.\n\n## Reading the numbers\n\nA high closed-book score means the mapping is in the weights, not that the model is a better programmer. Open-SFT at 0% with the same traces is the paper's Open-Book Paradox, not a broken eval. Memory methods can look strong by storing the NumPy map. Do not mix pass@64 Table 1 figures with Table 2 five-run means. Report single-function and multi-function accuracy separately, and say whether docs were visible at train and at test.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "se_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization"
  ],
  "category": "coding",
  "contamination": {
   "note": "Function names are random and the package is synthetic, so pretraining should not contain the mapping. Tasks and test cases are public on Hugging Face. The diagnostic still holds only if models are not trained on this release before the reported run.\n",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "MIT",
   "modalities": [
    "text",
    "code"
   ],
   "public_test_set": true,
   "size": 1417,
   "size_note": "1,417 tasks after consensus filtering: 718 single-function training items, 259 single-function test items, and 440 multi-function test items. Hugging Face configs train / single_test / multiple_test have 718, 259, and 440 rows. The library covers 268 NumPy functions. Training covers every function at least once.\n",
   "splits": "train 718 (single-function); test 699 (259 single-function + 440 multi-function)",
   "url": "https://huggingface.co/datasets/jintailin/SE-Bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-080 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Reference code is thunlp/SE-Bench. Roll out with query_only.py (no docs) or query_doc.py, then score in a Docker sandbox via filter_correct_trajectory.py. No lm-eval, inspect_evals, HELM, OpenCompass, or BIG-bench task was found.\n"
  },
  "id": "se_bench",
  "last_updated": "2026-05",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "SE-Bench checks whether an agent can absorb a new library and later use it with no docs. The authors wrap 268 common NumPy functions as a package named zwc, with nonsense identifiers such as zwc.kocito. Inputs and outputs are ZWCArray objects so the model cannot call NumPy methods on arrays. Training items include the relevant docstring. Test items are ordinary coding problems without that docstring. Single-function tests check recall. Multi-function tests require composing at least three APIs. A base model scores 0% without docs, so failures are meant to be memory failures.\n",
  "metric": {
   "baseline_note": "A solution counts only if every test case passes, AST checks show the return value depends on zwc, and numpy is never imported. Qwen3-8B pass@64 is 0.0% zero-shot on zwc, 97.4%/93.6% on the same tasks in NumPy, and 85.3%/70.5% with zwc docs in context (paper Table 1). No human baseline was reported.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy (test-case + AST constraints)",
   "random_baseline": 0,
   "unit": "%"
  },
  "name": "SE-Bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2602.04811",
   "title": "SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization",
   "url": "https://arxiv.org/abs/2602.04811",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Jiarui Yuan",
    "Tailin Jin",
    "Weize Chen",
    "Zeyuan Liu"
   ],
   "org": "THUNLP, Tsinghua University",
   "url": "https://github.com/thunlp/SE-Bench"
  },
  "released": "2026-02",
  "repo_url": "https://github.com/thunlp/SE-Bench",
  "saturation": {
   "as_of": "2026-05",
   "note": "Table 2 (v2, 2026-05) reports mean accuracy over five rollouts. The best parameter-update result is Closed-SFT-RL on Qwen3-8B: 54.4% single-function and 17.9% multi-function. Expel memory search reached 47.1% and 15.5% on the same splits. In-context docs still leave a gap to the NumPy upper bound. The ceiling is not reached.\n",
   "status": "open",
   "top_score": 54.4
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SE-Bench arXiv abstract (v2, 9 May 2026)",
    "url": "https://arxiv.org/abs/2602.04811"
   },
   {
    "accessed": "2026-09-08",
    "title": "SE-Bench HTML full text, arXiv 2602.04811v2",
    "url": "https://arxiv.org/html/2602.04811v2"
   },
   {
    "accessed": "2026-09-08",
    "title": "thunlp/SE-Bench GitHub repository",
    "url": "https://github.com/thunlp/SE-Bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "SE-Bench README (protocol, Hugging Face configs, citation)",
    "url": "https://raw.githubusercontent.com/thunlp/SE-Bench/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "SE-Bench MIT licence (LICENCE.md)",
    "url": "https://raw.githubusercontent.com/thunlp/SE-Bench/main/LICENCE.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "jintailin/SE-Bench dataset card",
    "url": "https://huggingface.co/datasets/jintailin/SE-Bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face Hub API (license:mit, lastModified 2026-05-07)",
    "url": "https://huggingface.co/api/datasets/jintailin/SE-Bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face datasets-server row counts (718 / 259 / 440)",
    "url": "https://datasets-server.huggingface.co/size?dataset=jintailin/SE-Bench"
   }
  ],
  "status": "active",
  "subcategory": "knowledge internalization / self-evolution diagnostic",
  "summary": "A diagnostic coding test that renames NumPy into a fake library so a score reflects whether an agent internalized new APIs, not old knowledge or hard reasoning.\n",
  "tags": [
   "coding",
   "self-evolution",
   "knowledge-internalization",
   "synthetic",
   "python"
  ],
  "task_format": "English problem statement in; the model writes Python that must call zwc APIs, pass hidden tests, and must not import numpy. Training may include API docs; the official test run does not.\n"
 }
}