{
 "body": "\n## What it measures\n\nSwedish to German Proverbs is a BIG-bench task that presents a proverb or saying in Swedish and\nasks the model to select, from four German-language options, the one closest in meaning. The\ntask's own description frames this as testing whether a model can \"identify the meaning of a\nproverb or saying in one language and select the saying most similar in meaning in another\nlanguage\" \u2014 a cross-lingual analogical-reasoning skill rather than literal sentence translation,\nsince a word-for-word rendering of a proverb rarely matches an idiomatic equivalent in another\nlanguage.\n\nThe task is small and narrowly scoped: both languages involved (Swedish and German) are\ncomparatively well-represented Germanic languages, so the exercise centers on figurative-meaning\nmatching rather than on low-resource-language coverage.\n\n## How it is scored\n\nEach item is four-way multiple choice, and BIG-bench's `multiple_choice_grade` is both the listed\nmetric and the preferred score. With four options, chance performance is 0.25. No few-shot count,\nprompt template, or human baseline beyond BIG-bench's general multiple-choice conventions is\nspecified in the task's own files.\n\n## Dataset and licence\n\nThe task's README states it comprises 72 multiple-choice items (and zero free-text items), each\nwith four German-language answer options and one correct match per Swedish proverb. All items and\ncorrect answers are visible in the public `task.json` file, so the set is fully public. The content\nis covered by the BIG-bench repository's Apache-2.0 licence.\n\n## Who publishes it\n\nThe task was contributed to BIG-bench by Marie Tolkiehn. BIG-bench itself is described in \"Beyond\nthe Imitation Game: Quantifying and extrapolating the capabilities of language models\"\n(arXiv:2206.04615, 2022; later published in Transactions on Machine Learning Research), a\nlarge multi-author collaboration coordinated by Google researchers. The `google/BIG-bench`\nrepository was archived by its owner on 2026-04-17 and is now read-only, so no further maintenance\nor leaderboard updates should be expected from that source.\n\n## Lineage\n\nThis is a standalone BIG-bench task with no stated predecessor. It shares its multiple-choice\nproverb-matching format with the sibling task `swahili_english_proverbs`, which pairs a different\nlanguage pair (Kiswahili and English) under the same general pattern; the two are independent\ncontributions rather than one deriving from the other. No successor task or dedicated leaderboard\nfor this specific task was found.\n\n## Saturation and contamination\n\nNo paper table, leaderboard, or per-model score breakdown specific to this task was located, and it\nis not part of lm-evaluation-harness's curated `bigbench` task set, so saturation status is\nunknown. Contamination risk is judged medium: the file and its answers are public and have been\nindexed since 2022, and both Swedish and German are reasonably well-represented in typical\nweb-scale training corpora, making prior exposure to this specific item set plausible for\nwidely-trained models.\n\n## How to run it\n\nThe canonical implementation is the task directory in the archived `google/BIG-bench` repository\n(`bigbench/benchmark_tasks/swedish_to_german_proverbs`), runnable through BIG-bench's own\ntask-running code. It is not among the tasks reimplemented in EleutherAI's lm-evaluation-harness\n`bigbench` group, so a runner would need to use the original `bigbench` Python package or\nreimplement the multiple-choice format directly from `task.json`.\n\n## Reading the numbers\n\nA high `multiple_choice_grade` here suggests a model can map figurative meaning between two\nGermanic languages well enough to identify an analogous proverb, a narrow proxy for cross-lingual\nidiomatic understanding. With only 72 items and no published baseline or leaderboard, a single\nscore should be treated as a small, noisy signal rather than a robust capability measurement, and\nread alongside the model's results on other BIG-bench proverb and multilingual tasks rather than in\nisolation.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "swedish_to_german_proverbs",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [],
  "category": "translation",
  "contamination": {
   "note": "The task file is public on GitHub with correct answers included in target_scores, and BIG-bench has been publicly indexed since 2022, so the item text is plausibly present in web-scale pretraining data for both Swedish and German, both comparatively well-resourced languages relative to some other BIG-bench proverb tasks.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "sv",
    "de"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 72,
   "size_note": "The task README states the benchmark comprises \"72 multiple choice and 0 free text\" queries, each with four German-proverb answer options.\n",
   "splits": "single set of 72 items, no train/test split",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swedish_to_german_proverbs"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Claude Sonnet 5, sonnet-batch-007 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Claude Sonnet 5 independent review, sonnet-batch-007"
  },
  "harness": {
   "bigbench": "swedish_to_german_proverbs",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "swedish_to_german_proverbs",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The model is shown a proverb or saying in Swedish and must choose, from four German-language options, the proverb closest in meaning. As with other BIG-bench proverb-matching tasks, a correct answer requires recognizing the figurative lesson behind the Swedish saying and matching it to an analogous (not literally translated) German saying, exercising cross-lingual analogical reasoning rather than literal translation.\n",
  "metric": {
   "baseline_note": "task.json lists \"multiple_choice_grade\" as both the metric and preferred_score. With four options per item, a uniform-random baseline is 0.25. No human-rater baseline is stated in the task's README or task.json.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1,
   "name": "multiple_choice_grade",
   "random_baseline": 0.25,
   "unit": ""
  },
  "name": "Swedish to German Proverbs",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2206.04615",
   "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
   "url": "https://arxiv.org/abs/2206.04615",
   "year": 2022
  },
  "publisher": {
   "authors": [
    "Marie Tolkiehn"
   ],
   "org": "Google (BIG-bench collaboration)",
   "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swedish_to_german_proverbs"
  },
  "released": "2022",
  "repo_url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swedish_to_german_proverbs",
  "saturation": {
   "as_of": "",
   "note": "No leaderboard or paper table reporting per-model scores on this specific task was located; it is not part of the curated BIG-bench-Hard or lm-evaluation-harness bigbench subset.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench: swedish_to_german_proverbs task directory",
    "url": "https://github.com/google/BIG-bench/tree/main/bigbench/benchmark_tasks/swedish_to_german_proverbs"
   },
   {
    "accessed": "2026-09-08",
    "title": "swedish_to_german_proverbs README",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/swedish_to_german_proverbs/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "swedish_to_german_proverbs task.json",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/bigbench/benchmark_tasks/swedish_to_german_proverbs/task.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository (archived 2026-04-17)",
    "url": "https://github.com/google/BIG-bench"
   },
   {
    "accessed": "2026-09-08",
    "title": "BIG-bench repository LICENSE",
    "url": "https://raw.githubusercontent.com/google/BIG-bench/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models",
    "url": "https://arxiv.org/abs/2206.04615"
   }
  ],
  "status": "unknown",
  "subcategory": "cross-lingual proverb/idiom matching, analogical reasoning (Swedish to German)",
  "summary": "BIG-bench multiple-choice task: pick the German proverb closest in meaning to a given Swedish proverb, from four options.",
  "tags": [
   "swedish",
   "german",
   "proverbs",
   "multilingual",
   "analogical-reasoning",
   "bigbench"
  ],
  "task_format": "Multiple-choice: one Swedish proverb, four German-proverb answer options, exactly one scored correct."
 }
}