SWE-AGI asks an agent to build a production-scale program, such as a parser or SAT solver, from a written specification in the little-known MoonBit language.
unassessed
| Category | coding |
|---|---|
| Subcategory | specification-driven software construction |
| Page status | active |
| Metric | Tasks completed |
| Direction | higher_is_better |
| Unit | tasks (of 22) |
| Dataset size | 22 |
SWE-AGI tests whether an LLM-based agent can autonomously construct substantial software from an explicit specification rather than modify an existing codebase. Tasks require implementing systems such as parsers, interpreters, binary decoders, and SAT solvers strictly from authoritative standards and RFCs, against a fixed API scaffold, with each task expected to take 1,000-10,000 lines of core logic. Tasks are written in MoonBit, a language with little presence in typical pretraining corpora, specifically to reduce the chance that an agent can retrieve a working solution instead of building one.
The agent is given a written specification (an RFC or standard), a fixed API scaffold to implement against, and no access to a pre-existing reference implementation. It must produce a working MoonBit program satisfying the specification, graded against a held test suite for that task.
No model card in ModelSpec reports this benchmark yet.