{
 "body": "\n## What it measures\n\nTopical-Chat evaluates open-domain response generation. A model receives a multi-turn English conversation and generates the next response, with the dialogue grounded in topical information.\n\nThe public examples span everyday and factual topics. The task measures similarity to reference responses, not unrestricted conversational quality.\n\n## How it is scored\n\nThe BIG-bench task metadata (`preferred_score: bleu`) names BLEU as the headline metric, but the task file also computes ROUGE, BLEURT, and log-likelihood for every response. BLEU rewards n-gram overlap with references and does not fully capture relevance, factuality, or naturalness; BLEURT is a learned metric intended to correlate better with human judgment, so the two can disagree on the same response. The task file provides no human baseline.\n\n## Dataset and licence\n\nThe public task file contains 22,295 examples. It does not state a separate split or licence in the inspected metadata. Dialogue and reference text are public, creating substantial exposure risk.\n\n## Who publishes it\n\nTopical-Chat is distributed through Google\u2019s BIG-bench collection and covered by the broader BIG-bench paper. No current standalone leaderboard or separate paper was established.\n\n## Lineage\n\nThis is a standalone BIG-bench task. The inspected sources do not establish a predecessor, successor, or formal variant.\n\n## Saturation and contamination\n\nSaturation is unknown. Public dialogue references can appear in training corpora, and BLEU can reward memorization. A high overlap score should not be treated as evidence of safe, useful conversation.\n\n## How to run it\n\nRun BIG-bench task `topical_chat` with its generation metric. Preserve conversation order, reference handling, task revision, and decoding settings. Report BLEU implementation details when comparing runs.\n\n## Reading the numbers\n\nA higher BLEU score indicates more lexical overlap with the task\u2019s reference responses. It does not establish engaging dialogue, factual grounding, long-context tracking, or resistance to repetition. Pair automatic scores with human judgments and factuality checks.\n\nThe task\u2019s open-domain framing makes topic distribution and reference diversity important. Inspect responses across topics rather than relying only on one corpus-level number.\n\nReference-based generation metrics can penalize valid responses that use different wording. Human relevance, coherence, and factuality checks are needed to complement BLEU.\n\nConversation length and topic transitions can also affect results. Preserve the full context and report truncation and decoding settings when comparing systems.\n\nThese controls are necessary for fair generation comparisons.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "topical_chat",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "generation",
  "contamination": {
   "note": "The public dialogue examples and references may have entered training corpora; actual exposure is unknown.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "English"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 22295,
   "size_note": "The public BIG-bench task file contains 22,295 examples.",
   "splits": "",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/topical_chat"
  },
  "freshness": {
   "luna-new-001 (Codex coordinated)": null,
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, luna-new-001"
  },
  "harness": {
   "bigbench": "topical_chat",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "topical_chat",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "bigbench",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The task asks a model to generate a response in an open-domain conversation. Its examples contain multi-turn dialogue about topics such as people, science, sports, and culture.",
  "metric": {
   "baseline_note": "The task metadata names BLEU and provides no human baseline.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "BLEU",
   "random_baseline": null,
   "unit": "score"
  },
  "name": "Topical-Chat",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [],
   "org": "Google BIG-bench",
   "url": "https://github.com/google/BIG-bench"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench",
  "saturation": {
   "as_of": "",
   "note": "No current task-specific leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench Topical-Chat task",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/topical_chat/task.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench paper",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "active",
  "subcategory": "open-domain dialogue",
  "summary": "BIG-bench Topical-Chat evaluates open-domain response generation in conversations grounded in topical information.",
  "tags": [
   "dialogue",
   "generation",
   "open-domain"
  ],
  "task_format": "Dialogue context followed by a generated response."
 }
}