{
 "body": "\n## What it measures\n\nPolEmo 2.0 asks a model to read a Polish consumer review and pick one of four sentiments: neutral, negative, positive, or ambiguous. Koco\u0144, Mi\u0142kowski, and Za\u015bko-Zieli\u0144ska built the corpus from hotels, medicine, products, and university (school) reviews and labelled both whole reviews and sentences. About 85% of the reviews are hotels or medicine. The KLEJ packaging that lm-eval loads keeps hotel and medicine reviews as the training pool. polemo2_in tests the same two domains. polemo2_out tests products and university instead. Prompts are Polish. This is not [imdb](imdb.md) binary polarity and not Allegro Reviews, the other KLEJ sentiment task.\n\n## How it is scored\n\nlm-eval does not score log-likelihoods over the four labels. It generates a short continuation, takes the first A\u2013D letter, maps it onto the gold class, and reports Hugging Face micro-F1 and accuracy. A run that never emits a letter is mapped to \u22121 and counted wrong. The Hub cards describe accuracy, and the OUT card's demo also prints macro-F1, so those numbers are not interchangeable with lm-eval micro-F1. Fine-tuned BERT figures from the 2019 paper use a different protocol. There is no single human accuracy; the paper's 0.91 / 0.88 figures are Positive Specific Agreement among annotators.\n\n## Dataset and licence\n\nThe CoNLL 2019 paper counts 8,216 reviews and 57,466 sentences. The two Allegro Hub dumps plus shared train reconstruct that total: 7,228 IN rows and 988 extra OUT val/test rows. lm-eval evaluates 722 IN test rows and 494 OUT test rows. The Hub column is named `sentence` but the loading examples are full reviews. Test labels are public. The Hub YAML tag is `cc-by-sa-4.0`; the same README licence section says CC BY-NC-SA 4.0; the paper only says a Creative Commons copyright licence. This page records CC-BY-NC-SA-4.0 from the Hub prose and treats the YAML tag as a disagreement. The cited CLARIN-PL handle returned 404 when opened.\n\n## Who publishes it\n\nThe corpus authors are at Wroc\u0142aw University of Science and Technology and the University of Wroc\u0142aw. CoNLL 2019 is the reference paper. Allegro published the KLEJ CSV mirrors used by lm-eval (Hub last modified 2022-08-30). EleutherAI maintains the harness YAMLs. No separate PolEmo 2.0 model-leaderboard URL was opened.\n\n## Lineage\n\nPolEmo 2.0 extends an earlier PolEmo resource described in the same authors' 2019 RANLP paper. KLEJ (Rybak et al., 2020) adopted it as two of nine Polish NLU tasks. This repository has no `klej` page. It is not [financial_phrasebank](financial_phrasebank.md) and not [legal_opinion_sentiment_classification](legal_opinion_sentiment_classification.md).\n\n## Saturation and contamination\n\nNo 2026 LLM score was read, so saturation is unknown. The source sites and the labelled CSVs are public and old enough to sit in web crawls. lm-eval's decontamination flag only helps if the training corpus is scanned.\n\n## How to run it\n\n`lm_eval --tasks polemo2` runs both tagged tasks, or run `polemo2_in` and `polemo2_out` alone. Compare only letter-generation micro-F1/accuracy on the Hub test splits. Do not mix IN with OUT, and do not mix them with KLEJ fine-tune tables.\n\n## Reading the numbers\n\nA strong IN score means the model labels hotel and medicine reviews in Polish under this four-way scheme. It does not mean the model tracks product or campus reviews: that is OUT, and OUT's class mix is different (almost no neutral in the files). Letter extraction is brittle. The Hub OUT split table is wrong; use 494/494 from datasets-server. The OUT class-distribution table labels minus as \"positive\" and plus as \"negative\"; the same card's Tasks section and the lm-eval YAML map minus to negative. Licence strings on the Hub card disagree, so reuse of the CSVs needs a human check. Look at IN and OUT together, and do not treat a KLEJ fine-tune number as an lm-eval number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "polemo2",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "PolEmo2",
   "PolEmo 2.0",
   "klej-polemo2",
   "polemo2_in",
   "polemo2_out"
  ],
  "category": "domain",
  "contamination": {
   "note": "Reviews come from public sites named in the paper (including TripAdvisor, ZnanyLekarz, PolWro, Ceneo). The KLEJ CSVs, including test labels, have been on Hugging Face since 2022-03-02. lm-eval sets should_decontaminate on the review text. The CLARIN-PL handle cited by the Hub cards returned HTTP 404 on 2026-09-08.\n",
   "risk": "high"
  },
  "dataset": {
   "languages": [
    "pl"
   ],
   "license": "CC-BY-NC-SA-4.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 1216,
   "size_note": "lm-eval scores the test splits: allegro/klej-polemo2-in test 722 rows, allegro/klej-polemo2-out test 494 rows (Hugging Face datasets-server). IN also has train 5783 and validation 723. OUT reuses the same 5783 train rows and has validation 494. Those five splits sum to 8,216 unique reviews, matching the CoNLL 2019 paper. The OUT Hub README table copies IN's 723/722 val/test counts; the parquet files do not. The same README text says the out-of-domain eval set is about 1,000 reviews (988 in the files). Paper: 8,216 reviews, 57,466 sentences, 197,046 annotations.\n",
   "splits": "IN 5783/723/722 train/val/test; OUT 5783/494/494 with shared train; lm-eval uses test",
   "url": "https://huggingface.co/datasets/allegro/klej-polemo2-in"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-066 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-066"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "",
   "lm_eval": "polemo2",
   "opencompass": "",
   "other": "Tag polemo2 runs polemo2_in (allegro/klej-polemo2-in) and polemo2_out (allegro/klej-polemo2-out). There is no group YAML, only a tag on polemo2_in.yaml. Metrics: Hugging Face micro-F1 and accuracy.\n"
  },
  "id": "polemo2",
  "last_updated": "2022-08",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "PolEmo 2.0 is a Polish consumer-review corpus. The model reads a review and must label it neutral, negative, positive, or ambiguous. KLEJ splits the same hotel and medicine training reviews into an in-domain test (hotels and medicine) and an out-of-domain test (products and university). lm-evaluation-harness follows that pair as polemo2_in and polemo2_out. It is review-level four-way sentiment, not sentence-level tagging, and not English IMDb-style polarity.\n",
  "metric": {
   "baseline_note": "Four labels, so uniform chance is 0.25 if classes were balanced. They are not: IN train is about 38% minus, 27% plus, 18% amb, 17% zero. The IN Hub card's random-demo accuracy is 0.251. Paper annotator PSA is 0.91 on texts and 0.88 on sentences, which is agreement, not a model accuracy ceiling. lm-eval reports micro-F1 and accuracy; the OUT Hub demo uses macro-F1.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "micro-F1 and accuracy (Hugging Face evaluate); KLEJ cards also quote accuracy",
   "random_baseline": 0.25,
   "unit": ""
  },
  "name": "PolEmo 2.0",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "Multi-Level Sentiment Analysis of PolEmo 2.0: Extended Corpus of Multi-Domain Consumer Reviews",
   "url": "https://aclanthology.org/K19-1092/",
   "year": 2019
  },
  "publisher": {
   "authors": [
    "Jan Koco\u0144",
    "Piotr Mi\u0142kowski",
    "Monika Za\u015bko-Zieli\u0144ska"
   ],
   "org": "Wroc\u0142aw University of Science and Technology and University of Wroc\u0142aw; KLEJ packaging by Allegro",
   "url": "https://aclanthology.org/K19-1092/"
  },
  "released": "2019-11",
  "repo_url": "https://github.com/EleutherAI/lm-evaluation-harness/tree/main/lm_eval/tasks/polemo2",
  "saturation": {
   "as_of": "",
   "note": "No current public LLM leaderboard cell was opened. The 2019 paper reports BiLSTM and BERT experiments; those are fine-tunes, not the lm-eval letter-generation protocol. KLEJ has a leaderboard that was not read here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "Koco\u0144, Mi\u0142kowski, Za\u015bko-Zieli\u0144ska, CoNLL 2019 (8,216 reviews; CC mentioned, not a SPDX id)",
    "url": "https://aclanthology.org/K19-1092/"
   },
   {
    "accessed": "2026-09-08",
    "title": "PolEmo 2.0 PDF (domains, 2+1 annotation, open-licence wording)",
    "url": "https://aclanthology.org/K19-1092.pdf"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hub card klej-polemo2-in (IN splits 5783/723/722; YAML license cc-by-sa-4.0; body CC BY-NC-SA 4.0)",
    "url": "https://huggingface.co/datasets/allegro/klej-polemo2-in"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hub card klej-polemo2-out (body table copies IN counts; OOD text ~1000 reviews)",
    "url": "https://huggingface.co/datasets/allegro/klej-polemo2-out"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server size: IN 5783/723/722",
    "url": "https://datasets-server.huggingface.co/size?dataset=allegro/klej-polemo2-in"
   },
   {
    "accessed": "2026-09-08",
    "title": "datasets-server size: OUT 5783/494/494 (not the Hub table)",
    "url": "https://datasets-server.huggingface.co/size?dataset=allegro/klej-polemo2-out"
   },
   {
    "accessed": "2026-09-08",
    "title": "lm-eval polemo2 README (group tag, in vs out domains)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/polemo2/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "polemo2_in.yaml (generate_until, micro-F1, letter map, decontaminate)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/polemo2/polemo2_in.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "polemo2_out.yaml (include in; dataset allegro/klej-polemo2-out)",
    "url": "https://raw.githubusercontent.com/EleutherAI/lm-evaluation-harness/main/lm_eval/tasks/polemo2/polemo2_out.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "KLEJ homepage (nine Polish NLU tasks; PolEmo2 is packaged as two of them)",
    "url": "https://klejbenchmark.com/"
   },
   {
    "accessed": "2026-09-08",
    "title": "KLEJ paper abs (Rybak et al., 2020); Allegro Reviews is a different KLEJ sentiment task",
    "url": "https://arxiv.org/abs/2005.00630"
   }
  ],
  "status": "unknown",
  "subcategory": "four-class Polish consumer-review sentiment, in-domain and out-of-domain",
  "summary": "Polish four-class review sentiment from PolEmo 2.0, scored in-domain and out-of-domain as two lm-evaluation-harness tasks.",
  "tags": [
   "polish",
   "sentiment",
   "classification",
   "klej",
   "reviews"
  ],
  "task_format": "lm-eval generate_until. Polish prompt \"Opinia:\" plus four lettered options (A Neutralny, B Negatywny, C Pozytywny, D Niejednoznaczny). Decoding stops at \".\" or \",\", temperature 0, max 50 tokens. A regex keeps the first A\u2013D letter. Gold labels are __label__meta_zero, __label__meta_minus_m, __label__meta_plus_m, __label__meta_amb. Tag polemo2 runs both tasks.\n"
 }
}