A key-value overwrite test of proactive interference: the model must report each key's last value after many similar updates, isolating working-memory limits inside the context window.
unassessed
| Category | long-context |
|---|---|
| Subcategory | proactive interference / key-value working-memory retrieval |
| Page status | active |
| Metric | auc_log1.5 (primary); average accuracy as a reference |
| Direction | higher_is_better |
| Dataset size | 740 |
| Dataset licence | MIT |
| Publisher | University of Virginia (Physics) and NYU Center for Neuroscience |
PI-LLM (Proactive Interference in LLMs) streams many semantically related key-value updates and then asks only for the current, last value of each key. The items sit well inside typical context windows (the OpenCompass README gives a 5-25k token band). Accuracy still falls as earlier values overwrite later ones, which the authors treat as working-memory interference rather than a haystack-length problem. Four experiment types vary update count (2-400), concurrent keys, value length (1-40 characters), and sequential versus randomised update order.
English text generation over a chat-style prompt of key-value lines plus a final question asking for the last value of one or more keys. OpenCompass runs zero-shot generation (max_out_len 2048) and grades the free-text reply against a JSON ground-truth value.
No model card in ModelSpec reports this benchmark yet.