466 real-world assistant questions needing tools and browsing; 300 test answers are held out and scoring is quasi-exact match.
unassessed
| Category | agentic |
|---|---|
| Subcategory | real-world assistant questions requiring tools, browsing, and files |
| Page status | active |
| Metric | accuracy (quasi-exact match after type-specific normalisation) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 466 |
| Publisher | FAIR, Meta and Hugging Face (paper affiliations); AutoGPT and Meta GenAI also listed |
GAIA asks conceptually simple assistant questions that still need reasoning, web browsing, file reading, and other tools. Answers are a number, a short string, or a comma-separated list, so scoring can be automatic. The authors wrote 466 questions: 146 Level 1 (at most one tool, few steps), 245 Level 2 (roughly 5–10 steps, mixed tools), and 75 Level 3 (long action sequences). English questions; some items attach a spreadsheet, image, audio, or other file. Humans scored 92% overall in the paper. GPT-4 with plugins scored 15% in the abstract (Table 4: 30.3 / 9.7 / 0 by level).
Zero-shot assistant prompt, optional attached file. Inspect default solver is a ReAct agent with bash, Python, and a web browser inside Docker. Default split is validation (answers present). Test split has no scores in-harness; those answers go to the public leaderboard. Tasks: gaia (2023_all), gaia_level1, gaia_level2, gaia_level3.
No model card in ModelSpec reports this benchmark yet.