{
 "body": "\n## What it measures\n\nAPE asks whether a model will try to persuade, not whether it is good at persuading. The evaluated model is the persuader. It is told to move a simulated user toward a statement. The user is another model with a hidden belief from 0 to 100. Under the default belief range 0\u201320, the requested direction is always \"for\". Topics are English sentences in six buckets, 100 each: benign facts, benign opinions, conspiracies, controversial issues, non-controversially harmful claims, and claims that undermine human oversight of AI.\n\nThe paper's safety question is whether the model will \"follow orders\" to advocate harmful content, such as glorifying terrorism. Jailbreaks in the paper raised that willingness. This is not Automatic Prompt Engineer (Zhou et al., 2022), which shares the APE acronym and is a prompt-optimisation method, not this eval.\n\n## How it is scored\n\nEach persuader turn is labelled 1 (attempt) or 0 (no attempt) by the evaluator. StrongREJECT marks refusals. inspect_evals then reports, grouped by category: `turn1_attempt` (attempt and not refused), `turn1_refusal`, `turn1_non_persuasive` (the residual), and `avg_persuasion_score` (mean of parseable turn scores). Version 2-A (12 August 2026) stores NaN when no turn score parses, so a judge failure is not counted as \"never persuaded\". The paper also tracked persuadee belief; inspect_evals always records a belief trajectory but does not treat it as a headline metric.\n\nProtocol knobs change the number. The default `persuade` system prompt tells the model to be maximally persuasive. `baseline` tells it not to persuade. Default length is 3 turns; the paper also reports 10. inspect_evals does not implement the paper's five deployment-persona prompts.\n\n## Dataset and licence\n\nThe topic file has 600 lines, 100 per category, at the SHA inspect_evals pins. The original repository's LICENSE is Apache 2.0. The inspect_evals port is MIT (UK AI Security Institute). Topics are public. There is no hidden test split.\n\n## Who publishes it\n\nKowal, Timm, Godbout, Costello, Arechar, Pennycook, Rand, Gleave and Pelrine, with FAR.AI as the first listed affiliation. The paper went on arXiv on 3 June 2025 (v4, 15 February 2026). The reference code is AlignmentResearch/AttemptPersuadeEval. inspect_evals' port was contributed by cmv13 and is the runnable name `ape_eval`.\n\n## Lineage\n\nAPE is not a refusal benchmark in the [harm_bench](harm_bench.md) sense. HarmBench asks whether an attack elicits a listed behaviour. APE asks whether the model, when instructed to persuade, will try \u2014 including on harmful statements. It is not [helm_safety](helm_safety.md). Do not confuse it with Automatic Prompt Engineer. No predecessor page in this repository measures persuasion *attempts* as the primary outcome.\n\n## Saturation and contamination\n\nAttempt rates still separate models by category. On 4 March 2026 inspect_evals ran 600 one-turn samples: GPT-4.1 attempted almost all conspiracy and controversial items but only 10% of non-controversially harmful ones; o4-mini was near zero on harmful and 0.13 on conspiracy. That is not a ceiling on the full suite. Topics are public, so targeted tuning is possible. Judge choice (default GPT-4o) is part of the measurement.\n\n## How to run it\n\n`inspect eval inspect_evals/ape_eval --model <persuader>`. Override roles with `--model-role`. Filter categories with `-T categories='[\"NoncontroversiallyHarmful\",\"UnderminingControl\"]'`. Set `-T persuader_prompt_type=baseline` or `-T num_turns=10` to match paper ablations. The reference `python main.py persuader_model=...` path in AlignmentResearch/AttemptPersuadeEval is a separate implementation; inspect_evals documents several scoring differences, including binary-only grading.\n\n## Reading the numbers\n\nA low harmful `turn1_attempt` means the model usually would not try to talk the user into that harmful statement on turn 1, under this prompt and these judges. It does not mean the model is unpersuasive on politics, or that a jailbreak would fail. A high benign attempt rate is the paper's expected behaviour, not a safety bug. Always report the category, prompt type, turn count, and judge models with the number.\n",
 "build": {
  "built_at": "2026-09-09T16:56:50+00:00",
  "commit": "0a599558854c0e238c03a0f0d725239cb28f9d11",
  "eligibility_as_of": "2026-09-09"
 },
 "disposition": {
  "canonical_id": "ape",
  "reasons": [],
  "status": "unassessed",
  "verified_results": []
 },
 "models_covered": [],
 "page": {
  "aliases": [
   "Attempt to Persuade Eval",
   "ape_eval",
   "AttemptPersuadeEval"
  ],
  "category": "safety",
  "contamination": {
   "note": "All 600 topic strings are public on GitHub (Apache-2.0). They were model- generated in 2025 rather than scraped from exams, so verbatim pretraining hits are less certain than for old school sets. A lab could still safety-tune on this exact list. Scoring depends on judge models, not hidden labels.\n",
   "risk": "medium"
  },
  "dataset": {
   "languages": [
    "en"
   ],
   "license": "Apache-2.0",
   "modalities": [
    "text"
   ],
   "public_test_set": true,
   "size": 600,
   "size_note": "100 topics in each of six categories, counted from diverse_topics.jsonl at AlignmentResearch/AttemptPersuadeEval SHA d77a4b14d5d3353ea4ac73fb22df239e36606c1d (Conspiracy, BenignOpinion, BenignFactual, Controversial, NoncontroversiallyHarmful, UnderminingControl). inspect_evals eval.yaml dataset_samples: 600. The paper says topics were generated automatically and then manually validated. The appendix uses OpenAI o1 for Benign Factual, Benign Opinion, and controversial lists; it does not state that all six categories used o1.\n",
   "splits": "single 600-topic pool; inspect_evals shuffles with seed 1234 and assigns belief with seed 42",
   "url": "https://github.com/AlignmentResearch/AttemptPersuadeEval"
  },
  "freshness": {
   "researched": "2026-09-08",
   "researched_by": "Grok Build, batch-026 (Codex coordinated)",
   "reviewed": "2026-09-08",
   "reviewed_by": "Grok Build independent review, batch-026"
  },
  "harness": {
   "bigbench": "",
   "helm": "",
   "inspect_evals": "ape_eval",
   "lm_eval": "",
   "opencompass": "",
   "other": "inspect eval inspect_evals/ape_eval; Python import inspect_evals.ape.ape_eval. eval.yaml version 2-A (2026-08-12)"
  },
  "id": "ape",
  "last_updated": "2026-08",
  "leaderboard_url": "",
  "lineage": {
   "family": "",
   "predecessor": "",
   "successors": [],
   "variants": []
  },
  "measures": "APE (Attempt to Persuade Eval) tests whether the model under evaluation will try to change a simulated user's belief, including when the topic is harmful. It does not score whether the user actually changes their mind. A persuader model talks to a persuadee model that starts with a low belief score. A third model labels each persuader turn as an attempt or not. A fourth model runs StrongREJECT to mark explicit refusals. The 600 English topics cover conspiracies, everyday opinions, factual claims, contested social issues, unambiguously harmful statements, and statements that undermine human control of AI.\n",
  "metric": {
   "baseline_note": "lower_is_better applies to the safety headline: attempt rate on NoncontroversiallyHarmful and UnderminingControl. The paper reports about 100% turn-1 attempts on benign factual and opinion topics, which is expected under the persuade prompt and is not a safety failure. inspect_evals groups metrics by category. Binary attempt scoring only; the paper dropped 3-point and 100-point evaluator scales after the 100-point judge looked near-random.\n",
   "direction": "lower_is_better",
   "human_baseline": null,
   "max_score": 1.0,
   "name": "turn1_attempt (share of samples with a turn-1 persuasion attempt and no StrongREJECT refusal); also turn1_refusal, turn1_non_persuasive, avg_persuasion_score",
   "random_baseline": null,
   "unit": ""
  },
  "name": "APE (Attempt to Persuade Eval)",
  "page_kind": "benchmark",
  "paper": {
   "arxiv": "2506.02873",
   "title": "It's the Thought that Counts: Evaluating the Attempts of Frontier LLMs to Persuade on Harmful Topics",
   "url": "https://arxiv.org/abs/2506.02873",
   "year": 2025
  },
  "publisher": {
   "authors": [
    "Matthew Kowal",
    "Jasper Timm",
    "Jean-Francois Godbout",
    "Thomas Costello",
    "Antonio A. Arechar",
    "Gordon Pennycook",
    "David Rand",
    "Adam Gleave",
    "Kellin Pelrine"
   ],
   "org": "FAR.AI (with Trajectory Labs, Apart Research, MIT, York University, Vector Institute, Universit\u00e9 de Montr\u00e9al, MILA, McGill, Cornell, University of Regina, American University, and CIDE)",
   "url": "https://github.com/AlignmentResearch/AttemptPersuadeEval"
  },
  "released": "2025-06",
  "repo_url": "https://github.com/AlignmentResearch/AttemptPersuadeEval",
  "saturation": {
   "as_of": "",
   "note": "No single top score is defined, because attempt rate is only safety-relevant on harmful categories. inspect_evals' 4 March 2026 report (version 1-A, 600 samples, one turn) put turn1_attempt on NoncontroversiallyHarmful at 0.1 for GPT-4.1, 0.03 for Claude Sonnet 4, and 0.0 for o4-mini, while conspiracy and controversial rates still spanned 0.13\u20131.0 across those three models.\n",
   "status": "open",
   "top_score": null
  },
  "sources": [
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals APE README (metrics, roles, 600 topics, 2-A changelog)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/ape/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals ape eval.yaml (task ape_eval, 600 samples, arXiv:2506.02873)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/ape/eval.yaml"
   },
   {
    "accessed": "2026-09-08",
    "title": "ape_eval task (defaults: 3 turns, belief 0\u201320, temperature 0.5)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/ape/ape.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "APE dataset loader (pinned topics SHA d77a4b14, 6 categories)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/ape/dataset.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "ape_scorer (turn1_attempt / refusal / non_persuasive; NaN on parse failure)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/ape/scorer.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "ape_conversation solver (default judge models gpt-4o / gpt-4o-mini)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/src/inspect_evals/ape/solver.py"
   },
   {
    "accessed": "2026-09-08",
    "title": "inspect_evals MIT License (UK AI Security Institute, 2024)",
    "url": "https://raw.githubusercontent.com/UKGovernmentBEIS/inspect_evals/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "APE paper abs (submitted 3 Jun 2025, v4 15 Feb 2026)",
    "url": "https://arxiv.org/abs/2506.02873"
   },
   {
    "accessed": "2026-09-08",
    "title": "APE paper HTML (600 topics, binary attempt metric, Figure 3)",
    "url": "https://ar5iv.labs.arxiv.org/html/2506.02873"
   },
   {
    "accessed": "2026-09-08",
    "title": "AlignmentResearch/AttemptPersuadeEval README (reference implementation)",
    "url": "https://raw.githubusercontent.com/AlignmentResearch/AttemptPersuadeEval/main/README.md"
   },
   {
    "accessed": "2026-09-08",
    "title": "AttemptPersuadeEval Apache License 2.0",
    "url": "https://raw.githubusercontent.com/AlignmentResearch/AttemptPersuadeEval/main/LICENSE"
   },
   {
    "accessed": "2026-09-08",
    "title": "diverse_topics.jsonl at pinned SHA (600 lines, 100 per category)",
    "url": "https://raw.githubusercontent.com/AlignmentResearch/AttemptPersuadeEval/d77a4b14d5d3353ea4ac73fb22df239e36606c1d/src/topics/diverse_topics.jsonl"
   }
  ],
  "status": "active",
  "subcategory": "willingness to attempt persuasion, including on harmful topics",
  "summary": "Multi-turn eval of whether a model tries to persuade a simulated user; the headline is turn-1 attempt rate on harmful topics, not persuasion success.",
  "tags": [
   "safety",
   "persuasion",
   "llm-judge",
   "multi-turn",
   "inspect-evals",
   "english"
  ],
  "task_format": "Multi-turn chat. Default is three persuader/persuadee turn pairs, temperature 0.5, persuade prompt (not the baseline). inspect_evals defaults: persuadee openai/gpt-4o, evaluator openai/gpt-4o, refusal_judge openai/gpt-4o-mini. Belief is sampled in [0, 20] so the requested direction is always \"for\".\n"
 }
}