2,305 elementary English math word problems with annotated type and grade; lm-eval runs a log-likelihood task and an 8-shot GSM8K-style CoT variant.
unassessed
| Category | math |
|---|---|
| Subcategory | elementary English math word problems with type and grade tags |
| Page status | unknown |
| Metric | accuracy (lm-eval asdiv: acc on log-likelihood; asdiv_cot_llama: exact_match) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 2305 |
| Dataset licence | CC-BY-NC-4.0 |
| Publisher | Natural Language Understanding laboratory, Institute of Information Science, Academia Sinica |
ASDiv (Academia Sinica Diverse MWP Dataset) tests whether a solver can answer short English math word problems taught in elementary school. Each item has a story body, a question, a numeric answer that may include a unit, an annotated formula, a solution type (24 types in the authors' tag set), and a grade level. The authors built it because earlier MWP corpora were narrow in wording or in operation mix. It is one-unknown school arithmetic and related elementary types, not contest math.
Free-response English word problem. lm-eval `asdiv` concatenates body and question, then scores log-likelihood of the gold answer with the unit stripped. `asdiv_cot_llama` is 8-shot generate-until with the GSM8K CoT wording from Wei et al. 2022, exact_match after "The final answer is". Formulas are ignored in both harness tasks.
No model card in ModelSpec reports this benchmark yet.