Single-turn inspect_evals task: emit a Vim keystroke string that turns each of 612 public VimGolf inputs into the target buffer.
unassessed
| Category | coding |
|---|---|
| Subcategory | single-turn Vim keystroke sequences on public VimGolf challenges |
| Page status | active |
| Metric | accuracy (binary correct/incorrect per challenge) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 612 |
| Dataset licence | MIT on inspect_evals (UK AI Security Institute, 2024). James4Ever0/vimgolf_challenges_and_solutions is Unlicense. Challenge text originates from vimgolf.com. |
| Publisher | UK AI Security Institute (inspect_evals packaging); challenges from vimgolf.com; inspect port by james4ever0 |
The model sees a VimGolf challenge (description, start buffer, target buffer) and must write one line of Vim keycodes that transforms the start text into the target. It does not get an interactive Vim session. A solution is correct only if it is non-empty, uses fewer keystrokes than the target's character length (so dumping the output file is illegal), and a Dockerised Vim run reproduces the target (checksum or formatted text). The skill is Vim editing, not GUI computer use.
Single-turn generate. A system prompt explains VimGolf notation and shows two example solutions. The last non-empty line of the completion is the candidate. Scoring runs vimgolf-verifier.py inside Docker (image built from the task Dockerfile, 15s timeout). inspect eval id is vimgolf_single_turn; the directory and extra are vimgolf_challenges.
No model card in ModelSpec reports this benchmark yet.