MMLU subject subset: Macroeconomic concepts -- GDP, inflation, fiscal and monetary policy -- at high-school/introductory level.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | economics |
| Page status | active |
| Metric | accuracy |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 390 |
| Dataset licence | MIT |
| Publisher | UC Berkeley (original); Center for AI Safety (current host) |
Macroeconomic concepts -- GDP, inflation, fiscal and monetary policy -- at high-school/introductory level. Questions are four-option multiple-choice, drawn from the MMLU test set's "economics" subcategory within the benchmark's "social sciences" top-level group, and are graded on the single correct labelled option.
Four-option multiple-choice questions, graded on the single correct labelled option; commonly evaluated 5-shot, consistent with the rest of MMLU.
| Model | Provider | Score | Card as of |
|---|---|---|---|
| Meta Llama 3 70B | Meta | 83.8 | 2024-07 |
| Meta Llama 3 70B Instruct | Meta | 83.8 | 2026-04 |
| Meta Llama 3 70B Instruct | Nous Research | 83.8 | 2026-04 |
| Yi 1.5 34B Chat | 01.AI | 83.8 | 2026-04 |
| Yi 1.5 34B Chat 16K | 01.AI | 83.8 | 2024-07 |
| Yi 1.5 34B 32K | 01.AI | 83.6 | 2024-07 |
| Yi 1.5 34B | 01.AI | 83.1 | 2026-04 |
| Nous Hermes 2 Yi 34B | Nous Research | 82.8 | 2024-07 |
| Mixtral 8x22B Instruct v0.1 | Mistral AI | 81.3 | 2026-04 |
| Yi 34B Chat | 01.AI | 78.5 | 2024-07 |
| Yi 34B 200K | 01.AI | 78.2 | 2024-07 |
| Yi 1.5 9B Chat | 01.AI | 77.9 | 2024-07 |
| Yi 1.5 9B Chat 16K | 01.AI | 76.2 | 2024-07 |
| Yi 9B | 01.AI | 75.9 | 2024-07 |
| Phi 3 mini 4K instruct | Microsoft | 75.4 | 2024-07 |
| Yi 1.5 9B | 01.AI | 75.4 | 2024-07 |
| Yi 1.5 9B 32K | 01.AI | 73.6 | 2024-07 |
| Phi 3 mini 128K instruct | Microsoft | 73.3 | 2024-07 |
| Yi 1.5 6B | 01.AI | 71.0 | 2024-07 |
| Mixtral 8x7B v0.1 | Mistral AI | 70.5 | 2026-04 |
| Mixtral 8x7B Instruct v0.1 | Mistral AI | 69.7 | 2026-04 |
| Nous Hermes 2 Mixtral 8x7B DPO | Nous Research | 69.0 | 2024-07 |
| Nous Hermes 2 SOLAR 10.7B | Nous Research | 67.4 | 2024-07 |
| Yi 1.5 6B Chat | 01.AI | 66.9 | 2024-07 |
| Meta Llama 3 8B Instruct | Meta | 65.9 | 2024-07 |
| Meta Llama 3 8B Instruct | Nous Research | 65.9 | 2024-07 |
| Hermes 2 Theta Llama 3 8B | Nous Research | 65.1 | 2024-07 |
| Meta Llama 3 8B | Meta | 65.1 | 2024-07 |
| Meta Llama 3 8B | Nous Research | 65.1 | 2024-07 |
| Yi 6B | 01.AI | 64.9 | 2024-07 |
| Yi 6B Chat | 01.AI | 64.9 | 2024-07 |
| gemma 7B it | Google DeepMind | 64.6 | 2024-07 |
| Hermes 2 Pro Llama 3 8B | Nous Research | 62.1 | 2024-07 |
| Mistral 7B v0.3 | Mistral AI | 60.5 | 2024-07 |
| mistral 7B v0.3 bnb 4bit | Unsloth | 60.5 | 2024-07 |
| Qwen2 1.5B Instruct | Alibaba / Qwen Team | 58.5 | 2024-07 |
| phi 2 | Microsoft | 58.2 | 2024-07 |
| Mistral 7B Instruct v0.2 | Mistral AI | 55.6 | 2024-07 |
| falcon 40B | TII | 55.4 | 2024-07 |
| deepseek llm 7B base | DeepSeek | 46.7 | 2024-07 |
| deepseek llm 7B chat | DeepSeek | 46.7 | 2024-07 |
| Qwen2 0.5B Instruct | Alibaba / Qwen Team | 45.9 | 2024-07 |
| chatglm2 6B | Zhipu AI | 44.9 | 2024-07 |
| gemma 2B | Google DeepMind | 41.0 | 2024-07 |
| deepseek coder 6.7B instruct | DeepSeek | 35.6 | 2024-07 |
| deepseek coder 6.7B base | DeepSeek | 34.6 | 2024-07 |
| gemma 2B it | Google DeepMind | 32.8 | 2024-07 |
| OLMo 1B hf | Allen AI | 27.2 | 2024-07 |
| deepseek coder 1.3B base | DeepSeek | 26.9 | 2024-07 |
| deepseek coder 1.3B instruct | DeepSeek | 26.9 | 2024-07 |