Twenty-five elementary English tasks that humans solve perfectly, scored as accuracy multiplied by robustness to trivial input changes.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | elementary English language unit tests (25 tasks, accuracy × robustness) |
| Page status | active |
| Metric | LMentry score (mean accuracy × mean robustness); HELM reports quasi_exact_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 110703 |
| Publisher | Tel Aviv University |
LMentry checks skills an elementary-school student is expected to have: write a sentence that contains a given word, pick the longer of two words, name the first letter, say which of two numbers is bigger, pick a rhyme, and similar. Avia Efrat, Or Honovich, and Omer Levy built 25 such tasks as a compact zero-shot unit test, not as a hard reasoning contest. Each task has three prompt templates. The suite also measures brittleness to argument order, argument content, template wording, and adjacent-task flips. HELM's scenario id is lm_entry; the evaluation itself is LMentry.
Zero-shot generation (official suite) or HELM generation / multiple-choice joint. Official scoring uses about ten regexes per task. HELM generation uses quasi-exact match with a one-word instruction and max_tokens 10. HELM multiple-choice joint is offered for tasks that can be cast as options; letter/word extraction tasks cannot.
No model card in ModelSpec reports this benchmark yet.