{
 "body": "\n## What it measures\n\nFewCLUE does not test one skill; it bundles nine Chinese NLU tasks -- sentiment classification,\ntopic classification, natural language inference, dialogue-intent matching, idiom cloze, scientific\nkeyword verification, and pronoun coreference -- under one shared constraint: every task is learned\nfrom a handful of labelled examples, not the thousands a standard supervised split gives. A FewCLUE\nscore answers \"how much can this model learn from 32 labelled reviews,\" not \"can this model do\nsentiment analysis\" -- the same task on CLUE's full-size training set answers a different question,\nwhich is why FewCLUE and CLUE keep separate ids in this wiki for the six tasks they share (Lineage).\n\nThe protocol has three deliberate parts. Sampling is stingy and label-count-aware: three or fewer\nclasses gets 32 training examples total (five of the nine tasks); 4-20 classes gets 16 per class\n(TNEWS); more than 20 gets 8 per class (CSLDCP, IFLYTEK); CHID, sampled by answer position rather\nthan semantic class, gets six examples per position, 42 total. Because a set this small causes\n\"severe fluctuation\" in results -- the paper's own TNEWS ablation found accuracy swinging several\npoints across otherwise-identical resamplings -- every task ships five independent train/dev splits\n(train_0..train_4) plus a deduplicated merge, so a score can be averaged across draws instead of\nresting on one lucky split. Each task also carries up to 20,000 unlabelled examples, unused in the\nscored metric but there for semi-supervised research.\n\n## How it is scored\n\nAll nine tasks are scored by accuracy against a labelled public test set (test_public.json), one\nlabel per item, no partial credit. Each method's headline \"Score\" is the unweighted mean of the nine\nper-task accuracies -- confirmed here by recomputing it: P-tuning-on-RoBERTa's reported 59.91 is\nexactly that mean. A second, larger test set was held out unlabelled, purely for the original\nclosed-leaderboard contest (NLPCC 2021 Task 2); most third-party evaluations, including OpenCompass,\nscore against the labelled set instead, which matters for contamination (see below).\n\nBaselines span a majority-class heuristic (0.8% to 50%, depending on the task's label count) through\nfive prompt-based few-shot methods (PET, ADAPET, LM-BFF, P-tuning, EFL) compared against plain\nfine-tuning, zero-shot prompting, and a trained human evaluation (82.5% overall, annotators trained\nthe same way SuperGLUE trains its raters). The best method tested, P-tuning on RoBERTa, reached\n59.91 -- about 23 points below the human figure -- with EFL notably stronger specifically on the two\nsentence-pair tasks (OCNLI, BUSTM), and CHID behaving close to zero-shot learning under PET because\nits native format is already cloze-shaped.\n\n## Dataset and licence\n\nThe labelled public test set totals 16,251 questions across the nine tasks, from 610 (EPRSTMT) to\n2,828 (CSL); training data follows the small, label-count-scaled sizes set out above, repeated over\nfive splits plus a merged train_few_all/dev_few_all. Six tasks (TNEWS, CHID, IFLYTEK, OCNLI, CSL,\nCLUEWSC) are existing CLUE-benchmark tasks resampled into this few-shot shape, in places with fresh\nannotation; three (EPRSTMT, CSLDCP, BUSTM) are new, built from e-commerce reviews, academic abstracts\nand a voice assistant's intent-matching logs respectively. All text is Chinese.\n\nNo licence is published for the data: the GitHub README marks its licence section as still being\nwritten, and the repository (confirmed via the GitHub API) has no LICENSE file -- treat the licence\nas not established. No official Hugging Face dataset card exists either (only a stale 2022 community\nmirror), so this page treats the GitHub repository as authoritative.\n\n## Who publishes it\n\nFewCLUE was built by the CLUE team -- Liang Xu, Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huilin Xu,\nHu Yuan, Guoao Wei, Xiang Pan, Xin Tian, Libo Qin and Hu Hai -- released as arXiv:2107.07498\n(submitted July 2021, revised September 2021) and used as the shared task for NLPCC 2021's few-shot\nlearning contest. The GitHub repository CLUEbenchmark/FewCLUE is the reference implementation and\ndata host; a leaderboard is hosted at cluebenchmarks.com/fewclue.html.\n\n## Lineage\n\nFewCLUE's parent project is CLUE (`clue` in this wiki): six of its nine tasks -- TNEWS, CHID,\nIFLYTEK, OCNLI, CSL, CLUEWSC -- are CLUE tasks resampled into few-shot splits; full-size OCNLI is\ntracked separately here as `clue_ocnli`. The other three (EPRSTMT, CSLDCP, BUSTM) are new to\nFewCLUE. This page's seven subset pages (`fewclue_bustm`, `fewclue_chid`, `fewclue_cluewsc`,\n`fewclue_csl`, `fewclue_eprstmt`, `fewclue_tnews`, `fewclue_ocnli_fc`) match exactly the seven of\nnine tasks OpenCompass implements as harness configs; `fewclue_csldcp` and `fewclue_iflytek` have\nneither a page here nor a confirmed OpenCompass config. No confirmed successor exists: the\nrepository has had no code changes since September 2022, and Chinese LLM evaluation has since\nlargely moved to broad knowledge-exam suites such as `ceval` and `cmmlu` -- a different question\n(breadth of static knowledge) from FewCLUE's own (how much a model learns from a handful of\nexamples).\n\n## Saturation and contamination\n\nSaturation status is not established for current models: no model card in this repository reports\nFewCLUE, and the cluebenchmarks.com leaderboard shows no visible activity beyond the initial 2021\nlaunch notice. The roughly 23-point gap between the best method and human performance (see How it is\nscored) describes only 2021-era BERT/RoBERTa/ERNIE/GPT-scale models under classic prompt-tuning;\nwhether it still holds for current frontier LLMs is unknown.\n\nContamination risk is high for the numbers actually in circulation: the labelled public test set has\nsat unchanged in the open GitHub repository since 2021 and is what OpenCompass and most external\nreproductions score against; the genuinely held-out split mattered only for the original leaderboard's\nprivate grading.\n\n## How to run it\n\nOpenCompass ships seven of the nine tasks as individual `FewCLUE_<task>` dataset configs (confirmed\nfor bustm, chid, cluewsc, csl, eprstmt, ocnli_fc and tnews; csldcp and iflytek are absent), each\nscored with an accuracy evaluator against test_public.json; no task-specific entry was found in\nlm-evaluation-harness or HELM. The GitHub repository itself is the reference implementation, with\nbaseline scripts for fine-tuning, PET, P-tuning, ADAPET, LM-BFF, EFL and zero-shot prompting under\n`baselines/`, keyed by task and split. Because five splits exist per task, treat a single-split score\nas noisy -- the paper's own ablation shows one small split can swing a result by several points, and\nreporters are not guaranteed to average the same way.\n\n## Reading the numbers\n\nA FewCLUE score answers \"how much can this model learn from a handful of examples,\" not \"how good is\nthis model at the task\" -- the same nine tasks at full CLUE scale produce different numbers. A high\nscore suggests strong few-shot adaptation, not necessarily strong task competence on its own; the\npaper's own baselines never got within 20 points of human performance under any method tested.\nBecause the test set is small and public since 2021, treat a modern score with caution on both\nvariance (multiple splits, not one) and contamination.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "fewclue",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Few-shot CLUE",
   "Chinese Few-shot Learning Evaluation Benchmark"
  ],
  "category": "composite",
  "contamination": {
   "note": "The labelled public test set (test_public.json, 16,251 items total) has sat unchanged in the open GitHub repository since 2021 and is what OpenCompass and most third-party reproductions score against. The genuinely held-out split (test.json) was used only for the original 2021 leaderboard's private grading and is not what a third-party harness run today typically uses.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "zh"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 16251,
   "size_note": "Sum of the nine tasks' labelled public test sets (test_public.json), the split most third-party harnesses score against: from 610 (EPRSTMT) to 2,828 (CSL) per task. Training data is the actual point of the benchmark and is deliberately tiny: 32 examples total for the five tasks with three or fewer labels (EPRSTMT, OCNLI, BUSTM, CSL, CLUEWSC), 240 for TNEWS (16/class x 15 classes), 536 for CSLDCP and 928 for IFLYTEK (approx. 8/class), and 42 for CHID (6 examples for each of 7 blank positions). Every task repeats its training/validation pair across five independent splits (train_0..train_4, dev_0..dev_4) plus a deduplicated train_few_all/dev_few_all merge, and separately ships up to 20,000 unlabelled examples per task (not used for the accuracy metric) and a private, unlabelled test set (test.json) used only for the original 2021 leaderboard. Figures are from the paper's Table 1, cross-checked against the GitHub README's own statistics table (identical).\n",
   "splits": "train_0..train_4 + train_few_all; dev_0..dev_4 + dev_few_all; test_public (16,251 total, labelled, used for scoring); test (private, unlabelled, original leaderboard only); unlabeled (up to 20,000/task, not scored)",
   "url": "https://github.com/CLUEbenchmark/FewCLUE"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "sonnet-5 agent, batch 5, slice B"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "lm_eval": "",
   "opencompass": "Per-task FewCLUE_<task> dataset configs, confirmed for bustm, chid, cluewsc, csl, eprstmt, ocnli_fc and tnews; no combined umbrella task name found, and csldcp/iflytek are absent from the registry",
   "other": ""
  },
  "id": "fewclue",
  "last_updated": "2022-09",
  "leaderboard_url": "https://www.cluebenchmarks.com/fewclue.html",
  "lineage": {
   "family": "",
   "predecessor": "clue",
   "successors": [],
   "variants": []
  },
  "measures": "FewCLUE bundles nine separate Chinese NLU tasks -- sentiment classification, long- and short-text topic classification, natural language inference, dialogue-intent matching, idiom cloze, scientific keyword verification, and pronoun coreference -- under one shared constraint: every task is learned from a handful of labelled examples (8 to 32 per class, depending on the task's number of labels) rather than a full-size training set. Five independent training/validation splits are provided per task specifically to average out the instability that small samples cause, plus up to 20,000 unlabelled examples per task for semi-supervised research. A FewCLUE score measures how much a model can learn from a handful of examples, which is a different quantity from a full-data score on the same underlying task.\n",
  "metric": {
   "baseline_note": "No single random-guess rate applies across tasks with such different label counts (2 to 119 classes); the paper's own naive \"majority class\" baseline ranges from 0.8% (IFLYTEK, 119 classes) to 50% (the five roughly-balanced binary/ternary tasks) -- see each subset page for its own figure. 82.5% is the paper's aggregate human score (recomputed here as the mean of its nine per-task human scores, which lands on 82.49-82.50; the paper's own prose separately rounds this to 82.40% elsewhere in the same document, a minor internal inconsistency in the source).\n",
   "direction": "higher_is_better",
   "human_baseline": 82.5,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": null,
   "unit": "%"
  },
  "name": "FewCLUE (Chinese Few-shot Learning Evaluation Benchmark)",
  "page_kind": "family",
  "paper": {
   "arxiv": "2107.07498",
   "title": "FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark",
   "url": "https://arxiv.org/abs/2107.07498",
   "year": 2021
  },
  "publisher": {
   "authors": [
    "Liang Xu",
    "Xiaojing Lu",
    "Chenyang Yuan",
    "Xuanwei Zhang",
    "Huilin Xu",
    "Hu Yuan",
    "Guoao Wei",
    "Xiang Pan",
    "Xin Tian",
    "Libo Qin",
    "Hu Hai"
   ],
   "org": "CLUE team",
   "url": "https://github.com/CLUEbenchmark/FewCLUE"
  },
  "released": "2021-04",
  "repo_url": "https://github.com/CLUEbenchmark/FewCLUE",
  "saturation": {
   "as_of": "2021-09",
   "note": "The paper's own baseline sweep found its best method (P-tuning on RoBERTa) scoring 59.91 -- the unweighted mean of its nine per-task accuracies -- against a human score of 82.5, roughly 23 points of headroom, but only for 2021-era BERT/RoBERTa/ERNIE/GPT-scale models under classic prompt-tuning methods (PET, P-tuning, ADAPET, LM-BFF, EFL). No model card in this repository reports FewCLUE and the public leaderboard shows no visible activity beyond the original 2021 launch notice, so whether current frontier models have closed this gap is not established.\n",
   "status": "unknown",
   "top_score": 59.91
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark (Xu et al., arXiv:2107.07498)",
    "url": "https://arxiv.org/abs/2107.07498"
   },
   {
    "accessed": "2026-09-08",
    "title": "CLUEbenchmark/FewCLUE GitHub repository (README, task descriptions, dataset statistics, licence section)",
    "url": "https://github.com/CLUEbenchmark/FewCLUE"
   },
   {
    "accessed": "2026-09-08",
    "title": "CLUEbenchmark/FewCLUE repository metadata via the GitHub REST API (creation/push dates, licence field)",
    "url": "https://api.github.com/repos/CLUEbenchmark/FewCLUE"
   },
   {
    "accessed": "2026-09-08",
    "title": "FewCLUE leaderboard, cluebenchmarks.com",
    "url": "https://www.cluebenchmarks.com/fewclue.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass datasets config directory (FewCLUE_* task configs and absence of csldcp/iflytek)",
    "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets"
   },
   {
    "accessed": "2026-09-08",
    "title": "clue dataset card, Hugging Face API (full-size CLUE benchmark mirror, for Lineage comparison)",
    "url": "https://huggingface.co/api/datasets/clue"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset search results for \"fewclue\" (no official CLUE-team dataset card found)",
    "url": "https://huggingface.co/datasets?search=fewclue"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness tasks directory listing via the GitHub REST API (no clue/fewclue task)",
    "url": "https://api.github.com/repos/EleutherAI/lm-evaluation-harness/contents/lm_eval/tasks"
   },
   {
    "accessed": "2026-09-08",
    "title": "HELM scenarios directory listing via the GitHub REST API (no clue/fewclue scenario)",
    "url": "https://api.github.com/repos/stanford-crfm/helm/contents/src/helm/benchmark/scenarios"
   }
  ],
  "status": "unknown",
  "subcategory": "Chinese few-shot NLU suite: sentiment, matching, classification, cloze and coreference tasks",
  "summary": "A Chinese few-shot NLU benchmark: nine tasks learned from 8-32 labelled examples per class across five parallel splits, so a score measures few-shot learning rather than full-data task competence.",
  "tags": [
   "chinese",
   "few-shot",
   "multi-task",
   "composite",
   "classification",
   "nli"
  ],
  "task_format": "Nine tasks: five single-sentence or sentence-pair classification tasks (2, 3, 15, 67 or 119 classes), one dialogue short-text matching task, and two reading-comprehension/cloze tasks (idiom cloze, keyword verification), all graded by accuracy against one correct label. Training draws 8-32 labelled examples per class from one of five parallel splits (train_0..train_4); a merged split and up to 20,000 unlabelled examples per task are also provided, unused for the scored metric.\n"
 }
}