Contextual Parametric Knowledge Conflicts

A 17,528-item BIG-bench QA task that swaps person names in Natural Questions passages so the context contradicts memorized facts.

Also known as: BIG-bench contextual_parametric_knowledge_conflicts

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categoryknowledge
SubcategoryBIG-bench closed-book QA with substituted PERSON answers from Natural Questions (17,528 items)
Page statusunknown
Metricexact_str_match
Directionhigher_is_better
Unit%
Dataset size17528
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration)

What it measures

contextual_parametric_knowledge_conflicts gives an English question and a Wikipedia-style passage whose PERSON mention has been replaced with another Wikidata person. The prompt asks for the answer assuming the context is true. The original Natural Questions answer is the distractor. Built with the Popularity Substitution framework from Longpre et al., EMNLP 2021. Authors listed on the task: Shayne Longpre, Kartik Perisetla, Anthony Chen. Not the original NQ span-selection task and not HELM [natural_qa](natural_qa.md).

Task format

Each JSON example has a generative target and two-way target_scores (substituted name 1, original name 0), except one malformed item with a single incorrect choice. Preferred metric exact_str_match; also rouge and multiple_choice_grade. append_choices_to_input is false. Canary GUID embedded.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub