401 constructed arithmetic expressions used to test LLM calculation; OpenCompass runs a four-shot English cloze with 1e-3 tolerance.
unassessed
| Category | math |
|---|---|
| Subcategory | 401 arithmetic expressions, scored by numeric tolerance |
| Page status | unknown |
| Metric | accuracy (absolute error < 1e-3); paper also RE and non-number ratio |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 401 |
| Publisher | Alibaba Group / Tsinghua University |
MATH 401 gives a model a bare arithmetic expression and asks for the numeric value. Yuan et al. built 401 items: Euler's identity plus 16 groups of 25 (small integers, large integers, decimals, negatives, multiplication, division, integer and decimal exponents, irrationals, long bracketed expressions, trigonometry, logarithms). English operator text in OpenCompass; original queries look like `78*64=`. This is calculation, not word problems ([gsm8k](gsm8k.md)) and not contest proofs ([math](math.md)).
OpenCompass generation with four baked-in few-shot lines ("Let's think step by step ... The answer is"), ZeroRetriever, GenInferencer max_out_len 512. mathbench_postprocess(name='en') pulls a number after "The answer is". Math401Evaluator marks correct if abs(pred − gold) < 1e-3. Original paper also reports relative error (capped at 10) and non-number ratio.
No model card in ModelSpec reports this benchmark yet.