{
 "body": "\n## What it measures\n\nSuperCLUE-Agent asks Chinese chat models to behave as agents on Chinese-native tasks. The publisher contrasts this with English agent suites that do not cover Chinese APIs, documents or dialogue habits.\n\nEach item is an open-ended user request. Tool-use tasks ask the model to pick, read and call APIs, or to use general tools such as search, browsing, files and databases. Planning tasks ask it to split a large goal, reflect on a failed step, or write a chain of thought. Memory tasks ask it to combine several documents, track topics in a long conversation, or learn from in-prompt examples.\n\n## How it is scored\n\nThe README and official page publish a total score, three capability scores and ten task scores, all on a 0-100 scale. GPT-4 is listed at 80.56 total. ChatGLM3-Turbo is listed at 73.09.\n\nThe exact per-item rubric, judge model, shot count and aggregation (mean of ten tasks versus a weighted blend of the three skills) are not stated in those pages. Treat two tables as comparable only when they come from the same SuperCLUE-Agent snapshot.\n\n## Dataset and licence\n\nNo item count, split table or licence file is in the GitHub repository. The GitHub API reports `license: null`. The tree is README plus images; there is no downloadable test set.\n\nThe official page and README give task definitions and screenshots only. Whether answers are held out cannot be checked from public files. Do not assume a reproducible local run from the repo alone.\n\n## Who publishes it\n\nCLUE / CLUEbenchmark publishes the page at cluebenchmarks.com and the GitHub repo. A named author list for this agent split is not on the README or the official page.\n\nThe GitHub repo was created 2023-10-19. The README is dated 2023-10-24. The last recorded push is 2023-11-09. The same organisation also runs SuperCLUE, SuperCLUE-Safety and the SuperCLUEAI.com leaderboard hub.\n\n## Lineage\n\nThis is a SuperCLUE track, not a CLUE 1.1 NLU task and not [AgentBench](agent_bench.md). SuperCLUE (arXiv:2307.15020) is the parent Chinese LLM suite of open-ended and closed-ended user tasks. SuperCLUE-Agent is the later agent-skill slice.\n\n[superclue_safety](superclue_safety.md) is a sibling SuperCLUE safety track, not a rename of this eval. There is no SuperCLUE family page in this repository yet.\n\n## Saturation and contamination\n\nOn the 2023 table, GPT-4 sits at 80.56 and the next Chinese-listed model at 73.09, so the published slice was not at ceiling then. Several task cells hit 100 (GPT-4 on general tool use; ChatGLM3-Turbo on task decomposition), so a single task score can saturate while the total does not.\n\nNo later dated agent table was opened for this page. Contamination risk is unknown because items are not public.\n\n## How to run it\n\nThere is no confirmed lm-evaluation-harness, inspect_evals, HELM, OpenCompass or BIG-bench task name. The GitHub repo has no scoring script.\n\nReported numbers come from CLUE's own leaderboard process. Prompt format, tools, and judge choice are not documented enough to reproduce a row from the README.\n\n## Reading the numbers\n\nA high total means the model scored well on CLUE's 2023 Chinese agent tasks across tools, plans and memory. It does not mean the model can operate a live tool sandbox, and it is not AgentBench.\n\nDo not compare a SuperCLUE-Agent total to English tool-use leaderboards. Look at the three skill columns: GPT-4's published gap was largest on tool use, while Claude-2-100K led the listed memory column at 73.97, which the official page links to long context.\n\nBecause n, licence and rubric are unpublished, treat the 80.56 GPT-4 figure as a historical CLUE table, not as a current frontier ranking.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "superclue_agent",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "SuperCLUE Agent"
  ],
  "category": "agentic",
  "contamination": {
   "note": "Items are not in the public GitHub tree, so training-set leakage cannot be checked from the repository. The publisher does not describe a held-out private split.\n",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": false,
   "size": null,
   "size_note": "Item count is not established. The GitHub repository contains a README and images only; no JSON/CSV split is published there. The official page likewise gives task definitions without n.\n",
   "splits": "",
   "url": "https://github.com/CLUEbenchmark/SuperCLUE-Agent"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-082 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "superclue_agent",
  "last_updated": "2023-11",
  "leaderboard_url": "https://www.cluebenchmarks.com/superclue_agent.html",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "SuperCLUE-Agent tests whether a Chinese LLM can act as an agent on native Chinese tasks rather than translated English agent suites. The publisher groups ten tasks into three skills: tool use (call, retrieve and plan APIs, plus general tools such as search, browsing, files and databases), task planning (decomposition, self-reflection and chain-of-thought), and long/short-term memory (multi-document QA, long-turn dialogue and in-context example learning). Prompts are Chinese user requests that require API choice, multi-step plans or recall across documents and dialogue turns.\n",
  "metric": {
   "baseline_note": "The GitHub README and cluebenchmarks.com table report a 0-100 total and per-task percentages. GPT-4 is listed at 80.56 total (tool use 90.23, task planning 81.88, memory 66.67). How those figures are aggregated, and whether a human or random baseline exists, is not stated in the README or the official page.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "published total score plus three capability scores and ten task scores",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "SuperCLUE-Agent",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "CLUE / CLUEbenchmark",
   "url": "https://www.cluebenchmarks.com/superclue_agent.html"
  },
  "released": "2023-10",
  "repo_url": "https://github.com/CLUEbenchmark/SuperCLUE-Agent",
  "saturation": {
   "as_of": "2023-11",
   "note": "GPT-4 led the first published table at 80.56; ChatGLM3-Turbo was the strongest Chinese-listed model at 73.09. GitHub last pushed 2023-11-09. Whether later SuperCLUEAI.com rows still use this ten-task agent split is not established.\n",
   "status": "unknown",
   "top_score": 80.56
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "CLUEbenchmark/SuperCLUE-Agent GitHub repository",
    "url": "https://github.com/CLUEbenchmark/SuperCLUE-Agent"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperCLUE-Agent README",
    "url": "https://raw.githubusercontent.com/CLUEbenchmark/SuperCLUE-Agent/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "SuperCLUE-Agent official page",
    "url": "https://www.cluebenchmarks.com/superclue_agent.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "GitHub API metadata for SuperCLUE-Agent",
    "url": "https://api.github.com/repos/CLUEbenchmark/SuperCLUE-Agent"
   }
  ],
  "status": "unknown",
  "subcategory": "Chinese-native agent skills: tool use, task planning, and long/short-term memory",
  "summary": "Chinese-native agent eval of tool use, planning and memory across ten tasks; GPT-4 led the 2023 table at 80.56, with no published item count or licence.",
  "tags": [
   "chinese",
   "agent",
   "tool-use",
   "planning",
   "memory"
  ],
  "task_format": "Open-ended Chinese agent prompts spanning the ten tasks above. The official page and GitHub README show worked examples as screenshots; they do not publish a machine-readable item schema, shot count or tool-sandbox specification.\n"
 }
}