{
 "body": "## What it measures\n\nSWE-AGI tests whether an LLM-based agent can build a substantial, production-scale program from a\nwritten specification rather than edit an existing codebase, which is what most SWE-bench style\nbenchmarks require. Tasks ask the agent to implement systems such as parsers, interpreters, binary\ndecoders, and SAT solvers strictly from authoritative standards and RFCs, against a fixed API\nscaffold, with each task expected to need on the order of 1,000 to 10,000 lines of core logic.\n\nTasks are written against MoonBit, a language with little presence in typical pretraining corpora.\nThe paper's stated reason is to reduce the chance that an agent solves a task by retrieving a similar\nimplementation from memory rather than reasoning through the specification, so the benchmark leans on\narchitectural and long-horizon reasoning rather than familiarity with a popular language's ecosystem.\n\n## How it is scored\n\nTasks are scored as completed or not against a held test suite for that task's specification; the\npaper reports results as a raw count out of 22 tasks rather than a normalized percentage metric with\nan established baseline. No random or human baseline is given in the source reviewed.\n\n## Dataset and licence\n\nSWE-AGI comprises 22 tasks of varying difficulty. The reviewed source does not state a dataset\nlicence, so this is left unknown rather than assumed. No separate dataset repository beyond the paper\nitself was found in the material reviewed.\n\n## Who publishes it\n\nSWE-AGI was introduced in the paper \"SWE-AGI: Benchmarking Specification-Driven Software Construction\nwith MoonBit in the Era of Autonomous Agents,\" authored by Zhirui Zhang, Hongbo Zhang, Haoxiang Fei,\nZhiyuan Bao, Yubin Chen, Zhengyu Lei, Ziyue Liu, Yixuan Sun, Mingkun Xiao, Zihang Ye, Yu Zhang,\nHongcheng Zhu, Yuxiang Wen, and Heung-Yeung Shum, posted to arXiv in February 2026. The publishing\norganisation was not stated in the source reviewed. No separate public leaderboard was found.\n\n## Lineage\n\nSWE-AGI is framed against the broader SWE-bench-style tradition of agent coding benchmarks but tests\nconstruction from a specification rather than modification of an existing repository, and uses its\nown MoonBit-based tasks rather than any shared dataset. No predecessor or successor benchmark within\nthis repository was established from the reviewed source, so lineage fields are left empty rather than\nforced into the SWE-bench family.\n\n## Saturation and contamination\n\nThe benchmark is not saturated but shows a wide spread: the paper reports GPT-5.3-codex completing 19\nof 22 tasks (86.4%) and Claude-Opus-4.6 completing 15 of 22 (68.2%), with performance degrading\nsharply on the hardest, most specification-intensive tasks. The paper states that \"code reading,\nrather than writing, becomes the dominant bottleneck\" as codebases scale within a task. Contamination\nrisk reads as low by design: the choice of MoonBit, a language with little presence in typical\npretraining corpora, is presented explicitly as a way to prevent agents from relying on web-retrieved\nreference implementations, though this is a design choice rather than an empirical contamination\nstudy.\n\n## How to run it\n\nNo lm-evaluation-harness, HELM, or OpenCompass task name was found in the sources reviewed, and no\npublic reference harness repository was identified. Because tasks require an agent to build software\nfrom scratch against a fixed API scaffold, reported results likely depend on the agent framework, tool\naccess, and step or token budget used, none of which are standardized in the material reviewed.\n\n## Reading the numbers\n\nA high SWE-AGI score means an agent can turn a written specification into a large, working program\nwithout a reference implementation to copy from or modify, a different and arguably harder skill than\npatching an existing codebase. With only 22 tasks, each additional or missed task moves the score by\nabout 4.5 percentage points, so small differences between models are not necessarily meaningful. The\npaper's finding that reading code becomes the bottleneck as size grows suggests scores should be read\nalongside task difficulty and size, not as a single uniform measure of coding skill. Because the\nlanguage (MoonBit) is unfamiliar to most models by design, scores here should not be assumed to\npredict performance on more common languages, and vice versa.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swe_agi",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "coding",
  "contamination": {
   "note": "Tasks are implemented in MoonBit, a language with little presence in typical pretraining data, specifically to reduce the chance an agent can retrieve a working solution rather than construct one; the paper presents this as a deliberate contamination-reduction design rather than a study of an existing corpus.",
   "risk": "low"
  },
  "dataset": {
   "languages": [
    "MoonBit"
   ],
   "license": "",
   "modalities": [
    "code",
    "text"
   ],
   "public_test_set": null,
   "size": 22,
   "size_note": "22 specification-driven software construction tasks of varying difficulty, each requiring an estimated 1,000-10,000 lines of core logic.",
   "splits": "single evaluation set",
   "url": "https://arxiv.org/abs/2602.09447"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-stream-b-003 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-stream-b-003"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "swe_agi",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "SWE-AGI tests whether an LLM-based agent can autonomously construct substantial software from an explicit specification rather than modify an existing codebase. Tasks require implementing systems such as parsers, interpreters, binary decoders, and SAT solvers strictly from authoritative standards and RFCs, against a fixed API scaffold, with each task expected to take 1,000-10,000 lines of core logic. Tasks are written in MoonBit, a language with little presence in typical pretraining corpora, specifically to reduce the chance that an agent can retrieve a working solution instead of building one.\n",
  "metric": {
   "baseline_note": "The paper reports completion as a count out of 22 tasks rather than a normalized percentage baseline.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 22,
   "name": "Tasks completed",
   "random_baseline": null,
   "unit": "tasks (of 22)"
  },
  "name": "SWE-AGI",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2602.09447",
   "title": "SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents",
   "url": "https://arxiv.org/abs/2602.09447",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Zhirui Zhang",
    "Hongbo Zhang",
    "Haoxiang Fei",
    "Zhiyuan Bao",
    "Yubin Chen",
    "Zhengyu Lei",
    "Ziyue Liu",
    "Yixuan Sun",
    "Mingkun Xiao",
    "Zihang Ye",
    "Yu Zhang",
    "Hongcheng Zhu",
    "Yuxiang Wen",
    "Heung-Yeung Shum"
   ],
   "org": "",
   "url": "https://arxiv.org/abs/2602.09447"
  },
  "released": "2026-02",
  "repo_url": "",
  "saturation": {
   "as_of": "2026-02",
   "note": "The paper reports GPT-5.3-codex completing 19 of 22 tasks (86.4%) and Claude-Opus-4.6 completing 15 of 22 (68.2%); performance drops sharply on the hardest, most specification-intensive tasks, which the authors read as evidence the benchmark still separates models.",
   "status": "open",
   "top_score": 86.4
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "SWE-AGI: Benchmarking Specification-Driven Software Construction with MoonBit in the Era of Autonomous Agents",
    "url": "https://arxiv.org/abs/2602.09447"
   }
  ],
  "status": "active",
  "subcategory": "specification-driven software construction",
  "summary": "SWE-AGI asks an agent to build a production-scale program, such as a parser or SAT solver, from a written specification in the little-known MoonBit language.",
  "tags": [
   "benchmark",
   "coding",
   "agentic",
   "long-horizon",
   "specification-driven"
  ],
  "task_format": "The agent is given a written specification (an RFC or standard), a fixed API scaffold to implement against, and no access to a pre-existing reference implementation. It must produce a working MoonBit program satisfying the specification, graded against a held test suite for that task.\n"
 }
}