{
 "body": "\n## What it measures\n\nSwahili-English Proverbs is a BIG-bench task that gives a model a proverb written in Kiswahili\n(Swahili) and asks it to identify which of four English-language proverbs carries the closest\nmeaning. Kiswahili proverbs typically express figurative wisdom through concrete imagery \u2014 a\nliteral English translation of the words would usually not sound like a proverb at all \u2014 so\nanswering correctly means inferring the underlying lesson and finding an English saying that\nteaches the same lesson, not doing word-level translation.\n\nThe task's authors frame this as testing \"paremiologic competence\" and analogical reasoning in a\ngenuinely low-resource setting: Kiswahili has tens of millions of speakers but a small footprint in\ntypical web-scale training corpora relative to high-resource languages, so the task probes whether\na model's cross-lingual and cultural-inference abilities extend beyond majority languages.\n\n## How it is scored\n\nEach item is multiple choice with one Kiswahili proverb and four English-proverb options, exactly\none of which is marked correct in `target_scores`. BIG-bench's standard metric for this format is\n`multiple_choice_grade`, generally computed from the model's relative log-likelihood over the\nanswer options (or greedy top-1 accuracy, depending on the runner). With four options, chance\nperformance is 0.25. No few-shot protocol, prompt template, or human baseline is stated in the\ntask's own files beyond BIG-bench's shared multiple-choice-task conventions.\n\n## Dataset and licence\n\nThe task ships as a single `task.json` file with one array of 153 examples, matching the task\nREADME's stated \"153 multiple choice and 0 free text\" queries; there is no separate\ntrain/validation/test split, consistent with most BIG-bench tasks. All items and their correct\nanswers are visible in the public file, so the test set is fully public. The item text is licensed\nunder the BIG-bench repository's Apache-2.0 licence; the file also carries a BIG-bench canary string\nasking that its contents be excluded from training-data crawls, which is a request, not an\nenforcement mechanism.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench by Kevin Omondi, Eunice Engefu Manyasi, Victoria Nyamai, Joan\nWaweru, Titus Tunduny, Tiberius Nkinyili and Clara Rivera. BIG-bench itself is a large multi-author\ncollaborative benchmark coordinated by Google researchers, described in \"Beyond the Imitation Game:\nQuantifying and extrapolating the capabilities of language models\" (arXiv:2206.04615, posted June\n2022; later published in Transactions on Machine Learning Research). The GitHub repository was\narchived by its owner on 2026-04-17 and is now read-only, so no further maintenance or leaderboard\nupdates should be expected from that source.\n\n## Lineage\n\nThis is a standalone BIG-bench task with no stated predecessor. BIG-bench includes a related but\ndistinct task, `swedish_to_german_proverbs`, which pairs a different language pair under a similar\nproverb-matching format; the two are siblings under the general BIG-bench \"proverbs\" pattern rather\nthan one being derived from the other. No successor task or dedicated leaderboard for this specific\ntask was found.\n\n## Saturation and contamination\n\nNo paper table, leaderboard or per-model score breakdown specific to this task was located during\nthis research, and it does not appear in lm-evaluation-harness's curated `bigbench` multiple-choice\ntask set (which covers a subset of the full ~200 BIG-bench tasks), so saturation status is unknown.\nContamination risk is judged medium: the file and its answers are public and have been indexed since\n2022, but Kiswahili-language content is comparatively scarce in most training corpora, which may\nlimit how much any given model has actually seen this specific file relative to high-resource-language\nBIG-bench tasks.\n\n## How to run it\n\nThe canonical implementation is the task directory in the archived `google/BIG-bench` repository\n(`bigbench/benchmark_tasks/swahili_english_proverbs`), runnable through BIG-bench's own task-running\ncode. It is not among the BIG-bench tasks reimplemented in EleutherAI's lm-evaluation-harness\n`bigbench` task group, so scores reported through that harness will not include this task by\ndefault; a runner would need to load the task via the original `bigbench` Python package or\nreimplement the multiple-choice format from `task.json` directly.\n\n## Reading the numbers\n\nA high multiple_choice_grade here suggests a model can map figurative meaning across a\nhigh-resource/low-resource language pair well enough to pick the analogous English proverb, which\nis a reasonable proxy for cross-lingual cultural and idiomatic understanding. It does not establish\nKiswahili generation ability, broader Kiswahili comprehension, or performance on proverbs outside\nthis specific 4-option set. Because no public leaderboard or paper table for this task was found,\nthere is no established reference point for what counts as a strong score, and any number should be\nread alongside the model's other Swahili- or low-resource-language results rather than in isolation.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swahili_english_proverbs",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Swahili-English Paremiologic Competence"
  ],
  "category": "translation",
  "contamination": {
   "note": "The task file is public on GitHub with answers included (target_scores visible for every item), and BIG-bench has been publicly indexed since 2022, so the item text is plausibly present in web-scale pretraining data. The file carries a BIG-bench canary string intended to let publishers filter it from training corpora, but compliance is voluntary and unverifiable from this page.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "sw",
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 153,
   "size_note": "task.json defines a single \"examples\" array of Kiswahili-proverb items, each with four English answer options and binary target_scores; a direct count of the array returned 153 items, matching the task README's stated \"153 multiple choice and 0 free text\" queries.\n",
   "splits": "single set, no train/test split (few-shot BIG-bench tasks draw shots from the same file)",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swahili_english_proverbs"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-007 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-007"
  },
  "harness": {
   "bigbench": "swahili_english_proverbs",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "swahili_english_proverbs",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The model reads a proverb or idiom in Kiswahili (Swahili) and must pick, from four English-language proverbs, the one closest in meaning. Because proverbs encode figurative, culturally specific wisdom rather than literal statements, a correct choice requires recognizing the underlying idea behind the Swahili saying and matching it to an analogous (not literally translated) English saying. The task's own keywords describe it as testing \"analogical reasoning\" in a low-resource language pair, not word-for-word translation.\n",
  "metric": {
   "baseline_note": "BIG-bench's standard scoring for multiple-choice tasks is multiple_choice_grade, the probability mass (or greedy accuracy, depending on harness) assigned to the correct option among the choices given; with four options per item a uniform-random baseline is 0.25. No human-rater baseline was read from the task README or task.json.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "multiple_choice_grade",
   "random_baseline": 0.25,
   "unit": ""
  },
  "name": "Swahili-English Proverbs",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Kevin Omondi",
    "Eunice Engefu Manyasi",
    "Victoria Nyamai",
    "Joan Waweru",
    "Titus Tunduny",
    "Tiberius Nkinyili",
    "Clara Rivera"
   ],
   "org": "Google (BIG-bench collaboration)",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swahili_english_proverbs"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swahili_english_proverbs",
  "saturation": {
   "as_of": "",
   "note": "No leaderboard or paper table reporting per-model scores on this specific task was located; it is not part of the curated BIG-bench-Hard or lm-evaluation-harness bigbench subset.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench: swahili_english_proverbs task directory",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swahili_english_proverbs"
   },
   {
    "accessed": "2026-09-08",
    "title": "swahili_english_proverbs task.json",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/swahili_english_proverbs/task.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "swahili_english_proverbs README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/swahili_english_proverbs/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository (archived 2026-04-17)",
    "url": "https://github.com/google/BIG-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository LICENSE",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
    "url": "https://arxiv.org/abs/2206.04615"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-evaluation-harness bigbench multiple_choice task directory",
    "url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/bigbench/multiple_choice"
   }
  ],
  "status": "unknown",
  "subcategory": "cross-lingual proverb/idiom matching, analogical reasoning, low-resource language (Swahili)",
  "summary": "BIG-bench multiple-choice task matching a Kiswahili proverb to its closest English-language equivalent among four options.",
  "tags": [
   "swahili",
   "proverbs",
   "low-resource-language",
   "multilingual",
   "analogical-reasoning",
   "bigbench"
  ],
  "task_format": "Multiple-choice: one Kiswahili proverb, four English-proverb answer options, exactly one scored correct."
 }
}