{
 "body": "\n## What it measures\n\nOAB Exams measures legal question answering in Brazilian Portuguese. The source describes the OAB exam as a mandatory test for people who want to practise law in Brazil. Its objective phase covers all areas of law through multiple-choice questions.\n\nThe dataset also describes a written phase, but the HELM scenario evaluates only the objective questions. It loads the dataset\u2019s `train` split and presents each question without retrieved legal material.\n\n## How it is scored\n\nHELM creates one reference for the correct choice and untagged references for the incorrect choices. Accuracy over the selected answer is the natural metric. The source does not provide a human baseline or a universal score report.\n\n## Dataset and licence\n\nThe scenario says the dataset contains exams from 2010 through 2018. The Hugging Face dataset API reports 2,210 train rows; HELM skips rows marked `nullified`, so the scored count depends on the release contents. The dataset is text, in Portuguese, and the source does not establish a licence.\n\n## Who publishes it\n\nThe runnable scenario is maintained in Stanford HELM. The data release is `eduagarcia/oab_exams` on Hugging Face. The sources read do not identify a paper, original exam-authoring committee, or current leaderboard.\n\n## Lineage\n\nOAB Exams is a legal-domain evaluation collection. The sources read do not establish a predecessor, successor, or separate repository variant.\n\n## Saturation and contamination\n\nSaturation is unknown. Historical exam questions are public in the released dataset, and model training exposure is possible, but no measurement was found. Scores should therefore be treated as a test of legal question answering under this particular historical sample.\n\n## How to run it\n\nUse HELM\u2019s `oab_exams` scenario. It loads `eduagarcia/oab_exams`, iterates over `train`, removes nullified questions, and evaluates answer text against the keyed choice. The scenario tags the task as knowledge, multiple choice, and `pt-br`; prompt and language handling should be recorded.\n\n## Reading the numbers\n\nA high score indicates familiarity with the legal concepts and wording represented in the selected exams. It does not establish legal advice quality, professional qualification, or performance on the written phase. Compare scores only when filtering of nullified questions and answer-choice formatting match. A current legal evaluation should be paired with newer law and citation checks.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "oab_exams",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Brazilian Bar Exam"
  ],
  "category": "domain",
  "contamination": {
   "note": "Historical exams are public through the dataset release; exposure in model training was not measured.",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "Portuguese"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 2210,
   "size_note": "The Hugging Face API reports 2,210 rows in the train split; HELM skips rows marked nullified at runtime, so the scored count is lower or equal.",
   "splits": "train (used as test by HELM)",
   "url": "https://huggingface.co/datasets/eduagarcia/oab_exams"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "GPT-5.6 Luna, luna-batch-017 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "GPT-5.6 Luna independent review, luna-batch-017"
  },
  "harness": {
   "bigbench": "",
   "helm": "oab_exams",
   "inspect_evals": "",
   "lm_eval": "",
   "opencompass": "",
   "other": ""
  },
  "id": "oab_exams",
  "last_updated": "",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "The benchmark uses objective multiple-choice questions from the Brazilian OAB bar examination. It tests legal knowledge across areas of Brazilian law in Brazilian Portuguese.",
  "metric": {
   "baseline_note": "No human baseline was supplied by the scenario source.",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 100,
   "name": "accuracy",
   "random_baseline": null,
   "unit": "percent"
  },
  "name": "OAB Exams",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "",
   "title": "",
   "url": "",
   "year": null
  },
  "publisher": {
   "authors": [],
   "org": "eduagarcia (dataset release)",
   "url": "https://huggingface.co/datasets/eduagarcia/oab_exams"
  },
  "released": "",
  "repo_url": "https://huggingface.co/datasets/eduagarcia/oab_exams",
  "saturation": {
   "as_of": "",
   "note": "No authoritative leaderboard was established.",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "HELM OAB Exams scenario",
    "url": "https://raw.githubusercontent.com/stanford-crfm/helm/main/src/helm/benchmark/scenarios/oab_exams_scenario.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "OAB Exams dataset card",
    "url": "https://huggingface.co/datasets/eduagarcia/oab_exams"
   }
  ],
  "status": "active",
  "subcategory": "legal knowledge",
  "summary": "OAB Exams evaluates Portuguese legal question answering on Brazilian bar examinations from 2010 through 2018.",
  "tags": [
   "law",
   "portuguese",
   "multiple-choice"
  ],
  "task_format": "Multiple-choice questions with four or more labelled choices; select the answer text."
 }
}