{
 "body": "## What it measures\n\n\u03c4-bench tests an agent in a simulated customer-service interaction. A language-model user presents a goal, the agent receives domain policy, and the agent can call APIs that change a database. The published domains are retail and airline operations.\n\nThe task therefore combines conversation, tool selection, state mutation, and policy compliance. It is an agentic text benchmark rather than a test of isolated question answering.\n\n## How it is scored\n\nThe reference evaluation compares the database state after a conversation with an annotated goal state. A completed goal is a pass. The paper introduces pass^k, the probability that all k independent trials pass, to expose reliability that one successful run hides. Results depend on the user simulator, agent prompt, available tools, and sampling procedure, so scores from different configurations should not be treated as interchangeable.\n\n## Dataset and licence\n\nThe benchmark repository contains the task environments, policies, tools, and evaluation code. The paper describes retail and airline domains and stateful conversations, but the sources consulted here do not establish a single total item count or a separately stated dataset licence. Those fields remain unknown. The test interactions are generated or executed through the environment; the repository should be checked for the exact fixture and answer visibility before claiming a split size.\n\n## Who publishes it\n\nThe benchmark was introduced by Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan in a 2024 arXiv paper. The official repository is published under the Sierra Research GitHub organisation. The paper is the authoritative description consulted for the protocol.\n\n## Lineage\n\n\u03c4-bench is a standalone benchmark. The later \u03c4\u00b2-bench project is listed as a successor in this catalogue because it extends the tool-agent evaluation setting. No predecessor is established by the paper.\n\n## Saturation and contamination\n\nThe paper reports that strong function-calling agents succeed on fewer than half of tasks in some settings and that repeated-trial pass^8 can be below one quarter in retail. That result indicates an open evaluation space. The benchmark is public, but the paper does not establish whether its test data entered model training; contamination risk is therefore unknown.\n\n## How to run it\n\nUse the official repository and reproduce its environment, tool definitions, policy prompts, user simulator, and database-state evaluator. Report the agent model, sampling settings, number of trials, domain, and whether pass rate or pass^k is used. Small prompt or simulator changes can alter results.\n\n## Reading the numbers\n\nA high single-trial pass rate suggests that an agent can reach the requested state in the tested domain. A high pass^k additionally indicates repeatability. Neither score proves safe behavior outside the supplied APIs or policies. Compare domain-level results and inspect policy violations, tool traces, and repeated-trial reliability alongside the aggregate.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "bench_bench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "tau-bench"
  ],
  "category": "agentic",
  "contamination": {
   "note": "The paper does not establish a training-data contamination rate in the abstract.",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "modalities": [
    "text",
    "actions"
   ],
   "public_test_set": true,
   "url": "https://github.com/sierra-research/tau-bench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-stream-a-003 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "other": "The reference implementation in the tau-bench repository."
  },
  "id": "bench_bench",
  "lineage": {
   "successors": [
    "tau2_bench"
   ]
  },
  "measures": "\u03c4-bench evaluates an agent that talks with a simulated user and calls domain-specific APIs. The benchmark checks both task completion and adherence to written policies in dynamic retail and airline domains.",
  "metric": {
   "baseline_note": "The paper also reports single-trial pass rates; exact benchmark-wide baselines are not established here.",
   "direction": "higher_is_better",
   "name": "pass^k",
   "unit": "fraction"
  },
  "name": "\u03c4-bench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2406.12045",
   "title": "\u03c4-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains",
   "url": "https://arxiv.org/abs/2406.12045",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Shunyu Yao",
    "Noah Shinn",
    "Pedram Razavi",
    "Karthik Narasimhan"
   ],
   "org": "Sierra",
   "url": "https://arxiv.org/abs/2406.12045"
  },
  "released": "2024-06",
  "repo_url": "https://github.com/sierra-research/tau-bench",
  "saturation": {
   "note": "The authors report substantial room for improvement and low repeated-trial reliability.",
   "status": "open"
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "\u03c4-bench paper and abstract",
    "url": "https://arxiv.org/abs/2406.12045"
   },
   {
    "accessed": "2026-09-08",
    "title": "Official tau-bench repository",
    "url": "https://github.com/sierra-research/tau-bench"
   }
  ],
  "summary": "\u03c4-bench evaluates whether tool-using agents complete user goals while following domain policies across stateful conversations.",
  "tags": [
   "tool-use",
   "agents",
   "policy-following"
  ],
  "task_format": "Multi-turn text conversations with tool calls and a final database state."
 }
}