MASBench varies synthetic task graphs on five axes so a run can show when a multi-agent system beats a single agent, rather than quoting one headline score.
unassessed
| Category | agentic |
|---|---|
| Subcategory | controlled multi-agent versus single-agent task graphs |
| Page status | active |
| Metric | Avg@8 accuracy |
| Direction | higher_is_better |
| Dataset size | 4583 |
| Dataset licence | Apache-2.0 |
| Publisher | Salesforce AI Research, with MIT and University of Wisconsin-Madison |
Each item is an English question plus a dependency graph of subtasks. Five axes set the graph: Depth (longest answer-containing chain), Horizon (intermediate answers that must be carried forward), Breadth (maximum in-degree), Parallel (independent components), and Robustness (subtasks with an adversarial note). Most items come from the iGSM math generator. Robustness mixes iGSM steps with RULER needle-in-a-haystack notes. The question is whether a multi-agent system (MAS) outperforms a single-agent system (SAS) as those axes grow. MAS-Orchestra, in the same paper, is the training method that uses this suite; this page documents the evaluation.
Text question with an explicit or implicit subtask graph. Axis-specific train and test splits. Scoring is Avg@8 accuracy. Horizon and Robustness also check intermediate answers; Depth, Breadth, and Parallel score the final answer only (Parallel allows multiple finals).
No model card in ModelSpec reports this benchmark yet.