{
 "body": "\n## What it measures\n\nOpenCompass `qabench` is a free-text QA run over a CSV of `prompt` and `reference` rows. The model sees only the prompt. There is no article field, no multiple-choice template, and no stated subject. The public GitHub config does not say who wrote the items or which language they use. Hugging Face had no dataset named qabench when searched on 2026-09-08. The same OpenCompass release zip that carries `nq` and `race` also lists `qabench`, so the CSV is part of the packaged data drop, not a Hugging Face auto-download.\n\nThis id is not [Natural Questions](nq_open.md), not Q-Bench (visual quality assessment), and not a documented alias of another page in this repository. Treat a reported \"qabench\" score as an OpenCompass CSV run until a paper or dataset card appears.\n\n## How it is scored\n\nThe config does not name a metric. `eval_cfg` only sets `ds_column` to `reference`. There is no AccEvaluator, RougeEvaluator, or postprocessor in the file. Whether OpenCompass then applies exact match, BLEU, or another default is not established from that source. Do not compare a qabench number with F1 on Natural Questions or accuracy on a multiple-choice QA set.\n\n## Dataset and licence\n\nSize, licence, and language are not established. The runnable path is `./data/qabench/qabench-test.qa.csv` via `HFDataset` with `path='csv'`. `datasets_info.py` maps `opencompass/qabench` to that folder and leaves `hf_id` and `ms_id` empty. A ModelScope page URL for `opencompass/qabench` exists as a shell, but the API returned 404 and the HTML card did not state counts. The OpenCompassData-complete zip dated 2024-02-07 lists the name; this session did not unpack that archive.\n\n## Who publishes it\n\nOpenCompass hosts the config. No authors, organisation, or paper are named in the config, the dataset map, or the dataset-statistics table. An arXiv search for QABench returned zero hits. The original publisher is not established.\n\n## Lineage\n\nNot a family page. It is not an alias of [nq_open](nq_open.md), [natural_qa](natural_qa.md), or [triviaqa](triviaqa.md), even though OpenCompass Natural Questions uses a similar `*.qa.csv` filename pattern. No successor was found.\n\n## Saturation and contamination\n\nNo public scores were found, so saturation is unknown. Contamination risk is unknown: the CSV is local to OpenCompass data drops, and whether those references appear in pretraining corpora was not shown.\n\n## How to run it\n\nIn OpenCompass, import `qabench_datasets` from `opencompass/configs/datasets/qabench/qabench_gen.py` (which loads `qabench_gen_353ae7.py`). Place `qabench-test.qa.csv` under `./data/qabench/` or unpack OpenCompassData-complete. The run is zero-shot generation with `ZeroRetriever` and `GenInferencer`. No lm-eval, HELM, or inspect_evals task was found. Numbers from a custom CSV that happens to use the same filename are not this task.\n\n## Reading the numbers\n\nA qabench figure from OpenCompass means the model produced text against whatever references sit in that CSV, under an unnamed scorer. It does not tell you domain, language, or difficulty. Ask for the CSV hash, the evaluator class, and the item count before comparing two runs. If those are missing, ignore the number and use a documented QA set instead.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "qabench",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "knowledge",
  "contamination": {
   "note": "The CSV is packaged for local use. Whether answers are public, held out, or derived from another named QA set (for example Natural Questions, which OpenCompass loads from nq-{split}.qa.csv) was not established. Do not treat the filename as evidence of a Natural Questions alias.\n",
   "risk": "unknown"
  },
  "dataset": {
   "languages": [],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": null,
   "size": null,
   "size_note": "Item count was not established. The config points at ./data/qabench/qabench-test.qa.csv. OpenCompass datasets_info maps the key opencompass/qabench to that local directory with empty Hugging Face and ModelScope ids. The name appears in the OpenCompassData-complete 0.2.2.rc1 zip listing (2024-02-07) next to nq and race, so a copy is distributed with that release, but the zip was not unpacked here. OpenCompass dataset_statistics does not list qabench.\n",
   "splits": "OpenCompass loads the CSV as HuggingFace split 'train'; no separate official test split is named",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/qabench"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-020 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-020"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "qabench (abbr qabench; dataset class HFDataset; config qabench_gen_353ae7.py)",
   "other": "No lm-eval, HELM or inspect_evals task with this name was found."
  },
  "id": "qabench",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "OpenCompass qabench is a generation task that feeds each row's `prompt` field to the model and compares the reply with a `reference` string. The shipped config uses HuggingFace `load_dataset` on a local CSV at `./data/qabench/qabench-test.qa.csv`, loaded as split `train` even though the filename says test. No README in the config directory describes the questions, the language, or how the references were written. Hugging Face dataset search for \"qabench\" returned an empty list on 2026-09-08. This page documents that OpenCompass task, not Q-Bench (low-level vision) or other QA-Bench names.\n",
  "metric": {
   "baseline_note": "qabench_gen_353ae7.py sets eval_cfg to `{ds_column: reference}` and does not name an evaluator class (no AccEvaluator, RougeEvaluator, or BLEU). How OpenCompass turns the reference column into a score is not established from the public config.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "",
   "random_baseline": null,
   "unit": ""
  },
  "name": "qabench",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "OpenCompass (config host); original dataset publisher not established",
   "url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/qabench"
  },
  "released": "",
  "repo_url": "https://github.com/open-compass/opencompass/tree/main/opencompass/configs/datasets/qabench",
  "saturation": {
   "as_of": "",
   "note": "No public leaderboard or published model table was found for this OpenCompass id.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass qabench_gen.py re-exports qabench_datasets",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/qabench/qabench_gen.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "qabench_gen_353ae7.py: HFDataset CSV path, prompt/reference columns",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/configs/datasets/qabench/qabench_gen_353ae7.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "DATASETS_MAPPING opencompass/qabench -> ./data/qabench, empty hf_id",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/utils/datasets_info.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "HFDataset wraps datasets.load_dataset and rewrites local data_files",
    "url": "https://raw.githubusercontent.com/open-compass/opencompass/main/opencompass/datasets/huggingface.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompassData-complete zip listing includes qabench",
    "url": "https://github.com/open-compass/opencompass/releases/tag/0.2.2.rc1"
   },
   {
    "accessed": "2026-09-08",
    "title": "OpenCompass dataset statistics (qabench not listed)",
    "url": "https://opencompass.readthedocs.io/en/latest/dataset_statistics.html"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face dataset search for qabench returned []",
    "url": "https://huggingface.co/api/datasets?search=qabench&limit=20"
   }
  ],
  "status": "unknown",
  "subcategory": "OpenCompass local-CSV question answering (prompt/reference pairs; origin not established)",
  "summary": "An OpenCompass generation config over a local qabench-test.qa.csv of prompt/reference pairs; the public tree does not identify a paper, licence, or item count.",
  "tags": [
   "question-answering",
   "opencompass",
   "origin-unestablished"
  ],
  "task_format": "Zero-shot free-text generation. The human turn is the raw `{prompt}` string. The config names no shot count and no option letters.\n"
 }
}