{
 "body": "\n## What it measures\n\nTwitterAAE evaluates language modeling on tweets aligned with African-American English and White dialect proportions. HELM downloads two files and evaluates them separately. The labels describe corpus alignment, not demographic ground truth for every author.\n\nIt measures text prediction under a dialectal distribution shift. It does not measure speaker identity, language ability or fairness in isolation.\n\n## How it is scored\n\nHELM reports bits per byte, with lower values better. Run the aa and white demographic arguments separately; an aggregate can hide the intended comparison. Tokenization and byte encoding affect the number.\n\nThe scenario says it selects 830,000 tweets with the highest African-American proportions and 7.3 million with the highest White proportions, then samples 50,000 from each. Those selections and the test files are part of HELM's protocol.\n\n## Dataset and licence\n\nThe HELM scenario exposes 50,000 tweets in each test file, for 100,000 rows in the scenario. It downloads aa_tweets.csv and white_tweets.csv from CodaLab. The opened scenario and paper do not establish a redistribution licence, so the field remains unknown.\n\nHELM's derived selection is not the entire source corpus in the 2016 paper. Twitter terms and source restrictions may apply.\n\n## Who publishes it\n\nThe source paper is by Su Lin Blodgett, Lisa Green and Brendan O'Connor and appeared at EMNLP 2016. Stanford's CRFM maintains the HELM integration. The scenario notes that its two datasets differ from the paper's aligned corpora.\n\n## Lineage\n\nTwitterAAE derives from the demographic dialectal variation corpus described by Blodgett et al. It is not a sentiment or toxicity benchmark, and no successor page was established.\n\n## Saturation and contamination\n\nNo current saturation result was established. The corpus is public and old, so contamination risk is high. A lower bits-per-byte score can reflect tokenizer or domain differences rather than a social improvement.\n\n## How to run it\n\nUse HELM's twitter_aae scenario with demographic=aa and demographic=white. Record tokenizer, byte encoding, context handling and exact files. Do not compare directly with token-level perplexity from another harness.\n\n## Reading the numbers\n\nLower bits per byte means better compression of the selected text under that model and tokenizer. The useful comparison is within a controlled run across subsets. It does not establish fair representation of either dialect. Pair it with qualitative dialect analysis and other social-bias tests.\n\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "twitter_aae",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Twitter African-American English",
   "Twitter AAE corpus"
  ],
  "category": "domain",
  "contamination": {
   "note": "The tweets and download endpoints are public and the source corpus predates current model training.",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 100000,
   "size_note": "HELM samples 50,000 tweets from each demographic subset.",
   "splits": "test: aa 50,000 / white 50,000",
   "url": "https://worksheets.codalab.org/rest/bundles/0x31485f8c37ad481fb9f4e9bf7ccff6e5/contents/blob/aa_tweets.csv"
  },
  "freshness": {
   "researched": "2026-09-09",
   "researched_by": "GPT-5.6 Luna, luna-new-003 (Codex coordinated)",
   "reviewed": "",
   "reviewed_by": ""
  },
  "harness": {
   "bigbench": "",
   "helm": "twitter_aae",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "twitter_aae",
  "last_updated": "",
  "leaderboard_url": "https://crfm.stanford.edu/helm/",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "TwitterAAE compares language-model performance on tweets selected for high estimated African-American English or White alignment. These are corpus-alignment labels, not identity claims about individual authors.",
  "metric": {
   "baseline_note": "HELM names bits_per_byte as the main metric and gives no human or random baseline.",
   "direction": "lower_is_better",
   "human_baseline": null,
   "max_score": null,
   "name": "bits per byte",
   "random_baseline": null,
   "unit": "bits/byte"
  },
  "name": "TwitterAAE",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Demographic Dialectal Variation in Social Media: A Case Study of African-American English",
   "url": "https://aclanthology.org/D16-1120/",
   "year": 2016
  },
  "publisher": {
   "authors": [
    "Su Lin Blodgett",
    "Lisa Green",
    "Brendan O'Connor"
   ],
   "org": "Stanford Center for Research on Foundation Models (HELM scenario)",
   "url": "https://crfm.stanford.edu/helm/"
  },
  "released": "2016",
  "repo_url": "https://github.com/stanford-crfm/helm",
  "saturation": {
   "as_of": "",
   "note": "No current ceiling was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-09",
    "title": "HELM TwitterAAE scenario",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/twitter_aae_scenario.py"
   },
   {
    "accessed": "2026-09-09",
    "title": "Blodgett, Green and O'Connor, EMNLP 2016",
    "url": "https://aclanthology.org/D16-1120/"
   }
  ],
  "status": "active",
  "subcategory": "language modeling across dialect-aligned tweet subsets",
  "summary": "HELM evaluates language-model bits per byte on 50,000 AAE-aligned and 50,000 White-aligned tweets.",
  "tags": [
   "helm",
   "language-modeling",
   "dialect",
   "social-bias",
   "twitter"
  ],
  "task_format": "Autoregressive language modeling on raw tweet text, evaluated separately for aa and white subsets."
 }
}