{
 "body": "\n## What it measures\n\nPersonality, in Inspect Evals, is a questionnaire suite for language models, not a knowledge test. With an empty `personality` parameter it records the model's default answers on Big Five and Dark Triad items. With `-T personality=\"\u2026\"` it asks whether a system prompt can make the model act like a named persona. personality_BFI presents 44 Likert stems such as \"Is talkative\" and five options from \"Disagree strongly\" to \"Agree Strongly.\" personality_TRAIT presents 8,000 short situations, each with two high-trait and two low-trait replies, covering the Big Five plus Machiavellianism, Narcissism, and Psychopathy. The TRAIT paper (Lee et al., arXiv:2406.14703) built those scenarios from BFI and SD-3 plus ATOMIC-10X. Both tasks are English text multiple choice.\n\n## How it is scored\n\nThere is no gold personality. The `any_choice` scorer marks a sample correct when it can parse `ANSWER: $LETTER` and the letter is in the allowed set; that percentage is format compliance. `trait_ratio` then turns parsed letters into per-trait scores: Likert 1\u20135 for BFI (reversed when `metadata.reverse` is true), and 1 versus 0 for TRAIT high versus low choices. Each trait is reported as a fraction of the maximum. Inspect's 2025-04-24 note compared a 1,600-item TRAIT slice (shuffle=questions, seed=41) to Table 11 of Lee et al. and found the same trait rank order. Do not average BFI and TRAIT into one number, and do not compare a format-correct rate to a TRAIT trait score.\n\n## Dataset and licence\n\nBFI is 44 items pinned to guiem/personality-tests commit `23325c76\u2026` (checksum recorded 2026-03-14). TRAIT is 8,000 gated rows on Hugging Face `mirlab/TRAIT` (API id `snupilab/TRAIT`) at revision `8b31c078\u2026`, eight splits of 1,000. The inspect_evals tree is MIT (UK AI Security Institute, 2024). The BFI JSON repository is GPL-3.0. A SPDX licence for the gated TRAIT dump was not readable without Hub access; the TRAIT paper on arXiv is CC BY 4.0. Answers are public once access is granted.\n\n## Who publishes it\n\nThe Inspect wrapper lives under UKGovernmentBEIS/inspect_evals, contributed by Guiem, currently eval.yaml version 3-A (2026-04-30). TRAIT is Lee, Lim, Han, Oh, Chae, Chung, Kim, Kwak, Yeonsoo Lee, Dongha Lee, Yeo, and Yu (NAACL 2025 Findings). No live leaderboard URL was opened for the Inspect suite.\n\n## Lineage\n\nThis page is the Inspect suite (`personality`), not a standalone TRAIT paper page and not a clinical BFI product. TRAIT expands BFI and SD-3; the Inspect BFI task is the 44-item inventory, not those 8,000 scenarios. Related Inspect work on social tone includes [eq_bench](eq_bench.md), which rates emotion intensity in scripted dialogues rather than Big Five scores. No successor id is in this repository.\n\n## Saturation and contamination\n\nSaturation does not apply in the usual accuracy sense. Models can sit at extreme conscientiousness and near-zero psychopathy without that meaning the test is solved; the TRAIT paper already reports that some traits are hard to elicit by prompting. BFI items have been on the web for decades. TRAIT is gated but described in a 2024 paper. Treat both as public enough to leak, without a measured memorization study in the sources opened here.\n\n## How to run it\n\nInstall `inspect-evals[personality]`. Run `inspect eval inspect_evals/personality_BFI` and `inspect eval inspect_evals/personality_TRAIT`. TRAIT needs `HF_TOKEN` after requesting dataset access. Optional `-T personality=\u2026` sets the persona prompt. TRAIT `-T shuffle=choices` or `all` (version 3-A) rebuilds `answer_mapping` from `choice_scores` so positional shuffle does not scramble trait keys. Log viewing is the Inspect viewer. Numbers from a 20% slice, a different shuffle, or a different system prompt are not interchangeable.\n\n## Reading the numbers\n\nA TRAIT conscientiousness of 0.85 means the model chose high-conscientiousness options often, not that it is reliable in production. Format-correct rate near 100% only says the model followed `ANSWER: A`. Compare a prompted run to the default persona on the same seed before claiming steerability. Read Dark Triad heads separately from Big Five heads. Pair the profile with a behavioural eval if you care what the model does when the questionnaire is gone.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "personality",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "inspect_evals personality",
   "personality_BFI",
   "personality_TRAIT"
  ],
  "category": "domain",
  "contamination": {
   "note": "BFI wording is a widely reprinted inventory. TRAIT items are on a gated Hugging Face dataset (snupilab/TRAIT / mirlab/TRAIT) with a public paper. Inspect pins both sources. No source opened here demonstrated memorization of TRAIT scenarios.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 8044,
   "size_note": "eval.yaml: personality_BFI 44 samples (pinned guiem/personality-tests commit 23325c7659839d5432e874a6cdd69b859c7728a1; Extraversion 8, Agreeableness 9, Conscientiousness 9, Neuroticism 8, Openness 10). personality_TRAIT 8,000 samples from Hugging Face mirlab/TRAIT (API id snupilab/TRAIT) at revision 8b31c078cb897c3917d2ee48735d0c15030680e0, eight splits of 1,000. TRAIT is gated (HF_TOKEN). Combined 8,044 is not a single mixed run; the two tasks are invoked separately.\n",
   "splits": "BFI: single 44-item set. TRAIT: eight named splits of 1,000 (Openness through Psychopathy); no train split.",
   "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/personality"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-065 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-065"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "personality",
   "lm_eval": "",
   "opencompass": "",
   "other": "Runnable tasks are inspect_evals/personality_BFI and inspect_evals/personality_TRAIT (eval.yaml version 3-A). Extra: pip install inspect-evals[personality]. TRAIT shuffle=choices|all was added in 3-A (2026-04-30) to remap answer_mapping after choice shuffle.\n"
  },
  "id": "personality",
  "last_updated": "2026-04",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "Personality is an Inspect AI evaluation suite contributed by Guiem (guiem) that applies human personality inventories to language models. It has two goals: record the persona a model shows with no extra instruction, and test whether a system prompt can push the model toward a named trait. The suite currently wraps two questionnaires. personality_BFI is the 44-item Big Five Inventory (Openness, Conscientiousness, Extraversion, Agreeableness, Neuroticism) with a five-point Likert scale. personality_TRAIT is the 8,000- item TRAIT benchmark of Lee et al. (arXiv:2406.14703, NAACL 2025 Findings): 1,000 four-way scenarios for each of the Big Five plus Machiavellianism, Narcissism, and Psychopathy, expanded from BFI and SD-3 with ATOMIC-10X. Answers are not factually right or wrong. The default scorer only checks that the completion matches ANSWER: $LETTER; trait_ratio converts those letters into per-trait scores.\n",
  "metric": {
   "baseline_note": "trait_ratio is the mean keyed rating divided by the per-item maximum, so each trait is a number in [0, 1] (README examples such as 57% Neuroticism). Reverse-keyed BFI items flip the Likert value. TRAIT maps high-trait choices to 1 and low-trait choices to 0. The any_choice scorer marks a sample CORRECT only when a letter in the target set is parsed; that rate is format compliance, not a personality gold. No human-norm table is in the Inspect README. Inspect's 2025-04-24 TRAIT 20% slice (1,600 items) reports GPT-4-turbo conscientiousness 85.1% versus the paper Table 11 mean 92.6%.\n",
   "direction": "higher_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "trait_ratio (per-dimension) plus format-correct rate",
   "random_baseline": null,
   "unit": ""
  },
  "name": "Personality (Inspect Evals)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2406.14703",
   "title": "Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometrics",
   "url": "https://arxiv.org/abs/2406.14703",
   "year": 2024
  },
  "publisher": {
   "authors": [
    "Guiem (inspect_evals contributor)",
    "Seungbeen Lee",
    "Seungwon Lim",
    "Seungju Han",
    "Giyeong Oh",
    "Hyungjoo Chae",
    "Jiwan Chung",
    "Minju Kim",
    "Beong-woo Kwak",
    "Yeonsoo Lee",
    "Dongha Lee",
    "Jinyoung Yeo",
    "Youngjae Yu"
   ],
   "org": "UK AI Security Institute (Inspect Evals); TRAIT authors at the TRAIT paper",
   "url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/personality"
  },
  "released": "2025",
  "repo_url": "https://github.com/UKGovernmentBEIS/inspect_evals/tree/main/src/inspect_evals/personality",
  "saturation": {
   "as_of": "",
   "note": "There is no single accuracy ceiling. Trait profiles are the result. The Inspect README's 2025-04-24 1,600-item TRAIT slice reproduces paper Table 11 rank order (correlations \u2265 0.99) with mean absolute deviations of a few percentage points. No later full-8,000 public cell was read here.\n",
   "status": "unknown",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals personality README (BFI 44, TRAIT 8000, scoring, 2025-04-24 slice, changelog 3-A)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/personality/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "personality.py (task defs, trait_ratio, any_choice, pinned revisions)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/personality/personality.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "eval.yaml (version 3-A, 44 and 8000 sample counts, mirlab/TRAIT + guiem/personality-tests)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/personality/eval.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "personality package exports personality_BFI and personality_TRAIT",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/personality/__init__.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "BFI and TRAIT system-prompt templates",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/personality/prompts/system.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals MIT License (Copyright 2024 UK AI Security Institute)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Pinned BFI JSON (44 items, reverse flags, trait labels)",
    "url": "https://raw.githubusercontent.com/guiem/personality-tests/23325c7659839d5432e874a6cdd69b859c7728a1/bfi.json"
   },
   {
    "accessed": "2026-09-08",
    "title": "guiem/personality-tests GNU GPL v3",
    "url": "https://raw.githubusercontent.com/guiem/personality-tests/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API for mirlab/TRAIT (resolves to snupilab/TRAIT, gated=auto, 8\u00d71000, sha 8b31c078)",
    "url": "https://huggingface.co/api/datasets/mirlab/TRAIT"
   },
   {
    "accessed": "2026-09-08",
    "title": "TRAIT paper (Lee et al.; v1 2024-06-20, v3 2025-03-19; NAACL 2025 Findings; CC BY 4.0)",
    "url": "https://arxiv.org/abs/2406.14703"
   },
   {
    "accessed": "2026-09-08",
    "title": "Hugging Face API for snupilab/TRAIT (gated=auto, eight splits of 1,000, sha 8b31c078)",
    "url": "https://huggingface.co/api/datasets/snupilab/TRAIT"
   }
  ],
  "status": "unknown",
  "subcategory": "LLM personality psychometrics (BFI Likert items plus TRAIT scenarios)",
  "summary": "Inspect Evals suite that scores an LLM's Big Five and Dark Triad profile from BFI and TRAIT questionnaires, not factual accuracy.",
  "tags": [
   "personality",
   "psychometrics",
   "big-five",
   "dark-triad",
   "inspect-evals"
  ],
  "task_format": "English multiple choice via inspect_ai.solver.multiple_choice. BFI: five options A\u2013E (Disagree strongly \u2026 Agree Strongly), 16 reverse-keyed items. TRAIT: four options (two high-trait, two low-trait responses). Optional -T personality=\u2026 system prompt. TRAIT shuffle in {questions, choices, all} with a seed. Output format ANSWER: $LETTER.\n"
 }
}