Programmatic BIG-bench task that asks a model to write new COPA-style multiple-choice items and then answer them; the score is validity times self-consistency.
unassessed
| Category | generation |
|---|---|
| Subcategory | BIG-bench programmatic task: invent COPA-style items then answer them |
| Page status | unknown |
| Metric | creativity_and_consistency_score (created_valid_percentage × multiple_choice_grade) |
| Direction | higher_is_better |
| Dataset size | 100 |
| Dataset licence | Apache-2.0 |
| Publisher | Google (BIG-bench collaboration) |
Question-Answer Creation tests two skills at once. First the model must continue from COPA-style demonstrations and emit a new multiple-choice item in BIG-bench JSON layout: a question, two choices, and a target that is one of those choices. Second, the same model is scored as a multiple-choice solver on the items it just wrote. The product of those two rates is the preferred score. The seed data is SuperGLUE COPA validation, used with the COPA authors' consent. There is no human quality filter on the invented questions.
Programmatic task (not a static JSON list). One-shot generation from two COPA examples, then log-probability multiple-choice on parsed inventions. Preferred metric creativity_and_consistency_score. Canary GUID embedded.
No model card in ModelSpec reports this benchmark yet.