500 memory-and-query samples that score whether a model leaks, sycophantically agrees with, or correctly uses injected long-term user memories.
unassessed
| Category | safety |
|---|---|
| Subcategory | long-term memory leakage, sycophancy, and useful recall |
| Page status | active |
| Metric | failure_rate (percent of samples over a judge threshold) |
| Direction | lower_is_better |
| Unit | % |
| Dataset size | 500 |
| Dataset licence | Paper CC BY 4.0; inspect_evals tree MIT; the authors' GitHub pyproject states no dataset licence |
| Publisher | Supervised Program for Alignment Research (SPAR), University of Cambridge, and CISPA Helmholtz Center for Information Security |
PersistBench prepends a synthetic long-term memory block to a new user query and asks whether the model uses those memories appropriately. Cross-domain items plant memories from one life domain next to a query from another. Sycophancy items plant user beliefs next to a query that should stay objective. Beneficial-memory items plant facts the model should actually use. The benchmark targets assistants that inject persistent user notes into the system prompt, not retrieval accuracy on a long dialogue log.
Single-turn chat: system prompt contains a memory list, user message is the query. Cross-domain and sycophancy default to three generations per sample (max-score reducer). Beneficial memory defaults to one generation. An LLM judge scores the response; inspect_evals reports failure_rate.
No model card in ModelSpec reports this benchmark yet.