A 126-item BIG-bench task that asks for entailment, contradiction or neutral on MultiNLI sentence pairs written in IPA.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | BIG-bench MultiNLI pairs transliterated into IPA |
| Page status | unknown |
| Metric | multiple_choice_grade |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 126 |
| Dataset licence | OANC End User License |
| Publisher | Google (BIG-bench collaboration) |
international_phonetic_alphabet_nli presents two English sentences that have been converted to International Phonetic Alphabet transcriptions and asks whether the pair is entailment, contradiction or neutral. The authors, a Stanford student group, built it so that word-by-word IPA lookup is not enough: the model has to recover meaning in a rare symbol set. Source pairs are MultiNLI items restricted to author-generated hypotheses and OANC-licensed genres (fiction excluded). Transliteration uses the CMU Pronouncing Dictionary via the eng-to-ipa library.
Three-way multiple choice over entailment, contradiction and neutral. task.json sets preferred_score to multiple_choice_grade, append_choices_to_input false, and a task_prefix that defines the three labels. Canary GUID embedded. Keywords include zero-shot.
No model card in ModelSpec reports this benchmark yet.