{
 "body": "## What it measures\n\nMTR-Suite is a family of tools and evaluations for conversational retrieval. MTR-Eval audits alignment gaps in existing benchmarks, MTR-Pipeline synthesizes dialogues, and MTR-Bench supplies a general-domain test.\n\nThe suite targets production-style conversations with hard topic switches and verbosity. It is intended to make retrieval evaluation more scalable and discriminative.\n\n## How it is scored\n\nThe paper presents MTR-Eval as an LLM-based auditor and MTR-Bench as the resulting benchmark. The abstract does not establish a single metric, maximum, or human baseline. Report the component, retrieval setup, dialogue generator, and evaluator with every number.\n\n## Dataset and licence\n\nThe authors state that code and data are publicly available. The abstract does not specify a consolidated item count or licence. Component benchmark sizes and public test boundaries should be taken from the release.\n\n## Who publishes it\n\nJunhao Ruan and ten coauthors introduced MTR-Suite in an ACL 2026 main-conference paper. The authors link code and data from the arXiv record.\n\n## Lineage\n\nMTR-Suite audits previous conversational retrieval benchmarks and includes the MTR-Bench variant. It is a family page rather than a single fixed dataset. No successor is named.\n\n## Saturation and contamination\n\nThe project reports alignment gaps in prior benchmarks and greater discrimination from production-style challenges. The evaluation remains open. Public data imply possible contamination, but model-specific exposure is unknown.\n\n## How to run it\n\nUse MTR-Eval for audit, MTR-Pipeline for synthesis, or MTR-Bench for standardized scoring. Record retriever, corpus, dialogue history, topic switches, verbosity, and evaluator. Do not compare synthetic and human-authored conversations without labeling the source.\n\n## Reading the numbers\n\nA high score means strong retrieval under the selected conversational conditions. It does not prove robust RAG in every domain. Read audit alignment and benchmark performance together. Topic-switch and verbosity slices are particularly important for production use.\n\nThe suite also changes the economics of benchmark construction: its synthesis pipeline is presented as a way to create high-fidelity dialogues at much lower human annotation cost. That claim concerns the construction process, not a guarantee that every generated dialogue is equally realistic. Validate dialogue quality and retrieval labels before extending the benchmark to a new domain.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "mtr_suite",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "category": "composite",
  "contamination": {
   "note": "Training exposure is not established by the paper abstract.",
   "risk": "unknown"
  },
  "dataset": {
   "modalities": [
    "text"
   ],
   "public_test_set": true
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-stream-a-008 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "other": "MTR-Eval auditor and MTR-Pipeline synthesis framework."
  },
  "id": "mtr_suite",
  "lineage": {
   "variants": [
    "mtr_bench"
   ]
  },
  "measures": "MTR-Suite evaluates conversational retrieval through an LLM auditor, a multi-agent dialogue synthesis pipeline, and a general-domain benchmark. It targets hard topic switches, verbosity, and alignment gaps in existing retrieval tests.",
  "metric": {
   "direction": "higher_is_better",
   "name": "retrieval quality",
   "unit": "score"
  },
  "name": "MTR-Suite",
  "page_kind": "family",
  "paper": {
   "arxiv": "2605.20729",
   "title": "MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks",
   "url": "https://arxiv.org/abs/2605.20729",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Junhao Ruan",
    "Abudukeyumu Abudula",
    "Bei Li",
    "Yongjing Yin",
    "Xinyu Liu",
    "Kechen Jiao",
    "Xin Chen",
    "Jingang Wang",
    "Xunliang Cai",
    "Tong Xiao",
    "Jingbo Zhu"
   ],
   "org": "MTR-Suite authors",
   "url": "https://arxiv.org/abs/2605.20729"
  },
  "released": "2026-05",
  "saturation": {
   "note": "The project identifies alignment gaps and production-style challenges in existing conversational retrieval benchmarks.",
   "status": "open"
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "MTR-Suite paper",
    "url": "https://arxiv.org/abs/2605.20729"
   }
  ],
  "summary": "MTR-Suite audits and synthesizes conversational retrieval benchmarks and introduces a production-style MTR-Bench.",
  "tags": [
   "retrieval",
   "conversational",
   "RAG",
   "benchmark-synthesis"
  ],
  "task_format": "Conversational retrieval with synthetic and audited multi-turn dialogues."
 }
}