A 211-item BIG-bench Lite task: read an English definition of a novel operator and return the numeric result, zero-shot.
unassessed
| Category | math |
|---|---|
| Subcategory | zero-shot application of a natural-language arithmetic operator |
| Page status | unknown |
| Metric | exact_str_match |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 211 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration); task author at Naver Labs |
operators defines an arithmetic operator in one or two English sentences, then applies it to one or more numbers. The model must output the result without seeing worked examples. Author Jos Rozen (Naver Labs) wrote items so a young person could compute them, with a hidden lambda used only to verify gold values. The probe is whether the model uses the stated rule, not whether it recalls times-tables. Keywords mark it as mathematics, numerical response, and zero-shot. It is one of the 24 BIG-bench Lite JSON tasks.
Free-text number. preferred_score exact_str_match. task_prefix is "Given the definition of the op operator, compute the result." example_output_prefix is " = ". output_regex keeps the last integer. stop_string is a newline. Canary GUID embedded. Dummy-model header reports 210 free-text queries; task.json holds 211 examples.
No model card in ModelSpec reports this benchmark yet.