QA WikiData

BIG-bench JSON task of 20,442 English cloze prompts built from Wikidata triples, scored with ROUGE-Lsum.

Also known as: qa_wikidata, QA Wikidata

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
SubcategoryBIG-bench cloze prompts from Wikidata triples
Page statusunknown
MetricrougeLsum
Directionhigher_is_better
Unit%
Dataset size20442
Dataset licenceApache-2.0 (BIG-bench task files); Wikidata facts are CC0
PublisherGoogle (BIG-bench collaboration)

What it measures

QA WikiData tests whether a language model can fill in a one-token object from a Wikidata fact, written as an English cloze. An example is "The country of 2000 Summer Olympics is" with target "Australia". The author sampled triples through the Wikidata API with stratification on the Property (relation), then kept objects that are a single token. Some prompts list several valid objects. The task is open-domain: there is no passage, only parametric knowledge of the web-scale facts Wikidata records.

Task format

Free-text completion. example_output_prefix is a space. Preferred metric rougeLsum; also bleu, bleurt, and exact_str_match. Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub