{
 "body": "\n## What it measures\n\nPRiSM (Phone Realization in Speech Models) is a speech benchmark, not a text exam. A system hears an utterance and must emit an IPA phone string, or offer a frozen representation for a probe. Intrinsic tests compare that string to gold phones with phonetic feature error rate (PFER): edit distance over articulatory features such as voicing, not token-level phone error rate.\n\nExtrinsic tests ask whether those transcripts or hidden states help on clinical and sociolinguistic jobs: dysarthria intelligibility (EasyCall), atypical child speech (UltraSuite), L1 classification, L2 assessment, FLEURS language id, Hindi dialect geolocation, and phone-inventory induction. Audio is the input; English is not the only language.\n\n## How it is scored\n\nTable 1 splits metrics. Core recognition uses PFER (lower is better). Downstream utility uses F1, Recall@1, or F1-PI (higher is better). There is no official macro average. The authors run transcription probes (a small GRU on predicted IPA) and representation probes (MLP on encoder states). They compare encoder-CTC, attention encoder-decoder, and large audio language models with fixed prompts.\n\nThe paper's headline comparison is qualitative: diverse training languages help; encoder-CTC is more stable; specialised PR models still beat LALMs on these probes. POWSM-CTC is called out as strongest on several representation tasks. No single top score is published as a leaderboard number.\n\n## Dataset and licence\n\nData are a union of existing corpora, not one new dump. The README lists Kaldi-style PR sets (DoReCo, Speech Accent Archive, L2-ARCTIC Perceived, TIMIT, Tusom2021, VoxAngeles) and Hugging Face probe repos under `changelinglab/prism`. Appendix A of the paper lists per-source licences (TIMIT LDC; DoReCo CC0; several CC BY or CC BY-NC; Tusom2021 MIT). There is no one SPDX id for the suite. GEO-v (Vaani) was still marked unreleased in the README retrieved 2026-09-08. Gold transcripts for the listed PR sets are public if you hold each source licence.\n\n## Who publishes it\n\nShikhar Bharadwaj and co-authors including Shinji Watanabe and David R. Mortensen. arXiv 2601.14046, submitted 20 January 2026, revised 13 July 2026, presented at ACL 2026. Code: `github.com/changelinglab/prism`. The README notes acceptance on 7 April 2026.\n\n## Lineage\n\nThis page is PRiSM the phone-recognition suite. It is not [PRISM-Bench](prism_bench.md) (text-to-audio-video, arXiv 2609.04867). It is not PRISM the LLM peer-review scorer (arXiv 2605.26730). It is not the PRISM alignment preference dataset. Those share an acronym only. PRiSM sits next to SUPERB-style speech probes rather than replacing them.\n\n## Saturation and contamination\n\nLALMs still lag specialised PR models on the paper's probes, so the suite is open. Several source sets are old and public, so models trained on TIMIT or Common Voice may have seen related audio. Feature-level gold for PFER is newer packaging. Risk is medium.\n\n## How to run it\n\nClone `changelinglab/prism`, install from `requirements.txt`, download `*-pr` repos into one tree, then `python src/main.py experiment=inference/transcribe_powsm data=powsmeval data.data_dir=...`. Probing uses `experiment=probing/lid_fleurs_powsm` and siblings. No lm-eval or HELM task was found. Compare PFER to F1 only with the paper's split of lower-vs-higher-is-better tasks.\n\n## Reading the numbers\n\nA low PFER means the model is close in articulatory features, not that it would pass a clinical listening test. Downstream F1 can rise from non-phonetic cues in the representation probe. LALM numbers in the paper use the authors' IPA prompts; a different prompt is a different exam. Report intrinsic and extrinsic columns separately, and name the source corpus.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "prism",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Phone Realization in Speech Models"
  ],
  "category": "multimodal",
  "contamination": {
   "note": "Several test sets are long-public (TIMIT is LDC-licensed; DoReCo CC0; others CC BY / CC BY-NC). Gold IPA and probe labels are in the collection. Risk is medium for models trained on common speech corpora, unknown for LALM pretraining.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [],
   "license": "",
   "modalities": [
    "audio",
    "text"
   ],
   "public_test_set": true,
   "size": null,
   "size_note": "No single item count. Intrinsic PR configs include DoReCo, Speech Accent Archive, L2-ARCTIC Perceived, TIMIT, Tusom2021, and VoxAngeles (Appendix A lists e.g. TIMIT 6,300; DoReCo 18,734; VoxAngeles 5,445). Extrinsic HF repos in collection changelinglab/prism include EasyCall, UltraSuite, EdAcc L1, CMU/L2 Arctic L1, SpeechOcean L2, FLEURS-24 LID; Vaani geolocation marked unreleased in the README.\n",
   "splits": "per-source train/dev/test where the upstream corpus defines them",
   "url": "https://huggingface.co/collections/changelinglab/prism"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-079 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": "Official Hydra entrypoint: python src/main.py experiment=inference/transcribe_powsm or experiment=probing/... on github.com/changelinglab/prism.\n"
  },
  "id": "prism",
  "last_updated": "2026-07",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "PRiSM tests whether a speech model hears phones, not words. Intrinsic tasks ask for an IPA transcript of an utterance and score articulatory-feature edits against gold phones (PFER), rather than token-level phone error rate. Extrinsic tasks reuse those transcripts or hidden states on dysarthria intelligibility, atypical child speech, L1 classification, L2 assessment, language id, geolocation, and phone-inventory induction. The paper argues that transcription error alone hides failures on clinical and sociophonetic work.\n",
  "metric": {
   "baseline_note": "Table 1 marks core PFER tasks as lower-is-better and downstream utility as higher-is-better. PFER is feature-level edit distance, not PER. F1-PI is defined in the appendix. No single official average. The paper reports encoder-CTC models as more stable than AED, and specialised PR models ahead of LALMs (Gemini 2.5 Flash, Qwen3-Omni) on phonetic probes.\n",
   "direction": "lower_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "PFER (intrinsic); task F1 / Recall@1 (extrinsic)",
   "random_baseline": null,
   "unit": ""
  },
  "name": "PRiSM",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2601.14046",
   "title": "PRiSM: Benchmarking Phone Realization in Speech Models",
   "url": "https://arxiv.org/abs/2601.14046",
   "year": 2026
  },
  "publisher": {
   "authors": [
    "Shikhar Bharadwaj",
    "Chin-Jou Li",
    "Yoonjae Kim",
    "Kwanghee Choi",
    "Eunjung Yeo",
    "Ryan Soh-Eun Shim",
    "Hanyu Zhou",
    "Brendon Boldt",
    "Karen Rosero Jacome",
    "Kalvin Chang",
    "Darsh Agrawal",
    "Keer Xu",
    "Chao-Han Huck Yang",
    "Jian Zhu",
    "Shinji Watanabe",
    "David R. Mortensen"
   ],
   "org": "Changeling Lab / Carnegie Mellon and collaborators",
   "url": "https://github.com/changelinglab/prism"
  },
  "released": "2026-01",
  "repo_url": "https://github.com/changelinglab/prism",
  "saturation": {
   "as_of": "",
   "note": "No live leaderboard. The paper's qualitative ranking is that specialised PR models beat large audio language models, with POWSM-CTC strongest on several representation probes. PFER on seen-language variation remains high enough that the suite still separates systems.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "PRiSM arXiv abs (v1 20 Jan 2026; v2 13 Jul 2026; ACL 2026)",
    "url": "https://arxiv.org/abs/2601.14046"
   },
   {
    "accessed": "2026-09-08",
    "title": "PRiSM full text (PFER, Table 1, Appendix A licences)",
    "url": "https://ar5iv.labs.arxiv.org/html/2601.14046"
   },
   {
    "accessed": "2026-09-08",
    "title": "changelinglab/prism README (HF collection, task table)",
    "url": "https://raw.githubusercontent.com/changelinglab/prism/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face collection changelinglab/prism",
    "url": "https://huggingface.co/collections/changelinglab/prism"
   }
  ],
  "status": "active",
  "subcategory": "phone recognition and phonetic downstream probes",
  "summary": "Open suite for phone recognition: phonetic-feature error on IPA transcripts plus clinical, L2, and multilingual probes of speech models.\n",
  "tags": [
   "speech",
   "phonetics",
   "audio",
   "multilingual",
   "asr"
  ],
  "task_format": "Audio in, IPA transcript and/or frozen representation out. Hydra configs under github.com/changelinglab/prism. Kaldi-style test sets for PR; Hugging Face repos for downstream probes.\n"
 }
}