204,000 information-seeking question-answer pairs across 11 typologically diverse languages, collected without translation, testing extractive QA beyond English.
unassessed
| Category | knowledge |
|---|---|
| Subcategory | multilingual extractive question answering |
| Page status | active |
| Metric | F1 and exact match, averaged across languages |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 204000 |
| Dataset licence | Apache-2.0 |
| Publisher | Google Research |
TyDi QA gives a model a real information-seeking question in one of 11 typologically diverse languages and asks it to find the answer in Wikipedia text in that same language. Questions were written by people who wanted to know the answer and did not already know it, and the data was collected directly in each language rather than by translating English questions, which the authors built in specifically to avoid the priming and translation artefacts that shape datasets like SQuAD, XQuAD or MLQA. "Typologically diverse" means the 11 languages were chosen to span a wide range of grammatical and orthographic features - among them Arabic, Bengali, Finnish, Indonesian, Japanese, Swahili, Korean, Russian, Telugu and Thai alongside English - so that strong performance is harder to achieve by a system tuned only to English-like structure. It is a question-answering and information-retrieval task rather than a machine-translation task - no language pair is translated, and each question is answered from text in its own language.
Two primary tasks over a full Wikipedia article - Passage selection (SelectP): return the index of the passage that answers the question, or NULL if none does; Minimal answer span (MinSpan): return the start/end byte span of the minimal answering text, YES/NO for yes-no questions, or NULL if unanswerable. A secondary Gold passage task (GoldP) hands the model a single passage guaranteed to contain the answer and asks for the extractive span, matching the SQuAD 1.1 format; GoldP drops unanswerable questions and excludes Japanese and Thai.
No model card in ModelSpec reports this benchmark yet.