{
 "body": "## What it measures\n\nMTR-Bench evaluates reasoning in interactive, multi-turn settings. It was designed because most reasoning evaluations use a single prompt and do not test whether a model can update its plan as an environment responds.\n\nThe benchmark contains four classes, 40 tasks, and 3,600 instances. It provides fine-grained difficulty and requires interaction with task environments.\n\n## How it is scored\n\nThe benchmark uses a fully automated evaluation framework for task construction and model assessment. The paper reports task performance and finds large gaps even for cutting-edge reasoning models. The abstract does not define one universal metric, maximum, or human baseline, so those details remain unknown.\n\n## Dataset and licence\n\nThe paper establishes 3,600 instances across 40 tasks and four classes. It does not state a licence or public test policy in the abstract. Record task class, difficulty, interaction budget, and environment state when reproducing results.\n\n## Who publishes it\n\nXiaoyuan Li and eight coauthors introduced MTR-Bench in a paper accepted to ACL 2026. No independent leaderboard is established.\n\n## Lineage\n\nMTR-Bench is a standalone multi-turn reasoning benchmark motivated by single-turn evaluation limits. The paper names no successor.\n\n## Saturation and contamination\n\nReported failures on interactive reasoning indicate an open benchmark. Training-data exposure is unknown.\n\n## How to run it\n\nUse the automated evaluation framework and preserve task class, difficulty, turn limit, and environment responses. Report whether tools or external knowledge are available. Single-turn adaptations are not comparable to the published protocol.\n\n## Reading the numbers\n\nA high score indicates successful reasoning under the selected interaction sequence. It does not guarantee planning robustness beyond the tested environments. Inspect performance by task class and turn depth. Interactive success and final answer correctness can diverge.\n\nThe benchmark is useful when a model must ask for missing information, revise an intermediate answer, or act after observing a changed state. A report that gives only a final aggregate loses the distinction between a model that reasons correctly from the first turn and one that recovers after feedback. Include failure traces or at least turn-level success when the implementation permits it.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mtr_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "category": "reasoning",
  "contamination": {
   "note": "Training exposure is not established by the paper abstract.",
   "risk": "unknown"
  },
  "dataset": {
   "modalities": [
    "text",
    "actions"
   ],
   "public_test_set": null,
   "size": 3600,
   "size_note": "3,600 instances across 4 classes and 40 tasks."
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-stream-a-008 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "other": "Fully automated MTR-Bench construction and evaluation framework."
  },
  "id": "mtr_bench",
  "measures": "MTR-Bench tests reasoning models in interactive environments rather than isolated single-turn prompts. It covers four classes, 40 tasks, and 3,600 instances with fine-grained difficulty and multi-turn interaction.",
  "metric": {
   "direction": "higher_is_better",
   "name": "task success",
   "unit": "score"
  },
  "name": "MTR-Bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2505.17123",
   "title": "MTR-Bench: A Comprehensive Benchmark for Multi-Turn Reasoning Evaluation",
   "url": "https://arxiv.org/abs/2505.17123",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Xiaoyuan Li",
    "Keqin Bao",
    "Yubo Ma",
    "Moxin Li",
    "Wenjie Wang",
    "Rui Men",
    "Yichang Zhang",
    "Fuli Feng",
    "Dayiheng Liu"
   ],
   "org": "MTR-Bench authors",
   "url": "https://arxiv.org/abs/2505.17123"
  },
  "released": "2025-05",
  "saturation": {
   "note": "The paper reports that even leading reasoning models fall short on interactive tasks.",
   "status": "open"
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MTR-Bench paper",
    "url": "https://arxiv.org/abs/2505.17123"
   }
  ],
  "summary": "MTR-Bench evaluates multi-turn interactive reasoning across 40 tasks and 3,600 instances in four task classes.",
  "tags": [
   "multi-turn",
   "interactive-reasoning",
   "agents"
  ],
  "task_format": "Multi-turn text interaction with task environments."
 }
}