Six procedural-generation tasks at 0.5K, 2K, and 8K output lengths that test whether long-context models can follow a procedure and emit a structured trace.
unassessed
| Category | long-context |
|---|---|
| Subcategory | six procedural-generation tasks at 0.5K / 2K / 8K output lengths |
| Page status | active |
| Metric | mean score (task-specific exact/structured match, averaged; lm-eval group longproc uses metric score, unweighted) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1709 |
| Dataset licence | Apache-2.0 (princeton-pli/LongProc LICENSE and Hugging Face card) |
| Publisher | Princeton Language and Intelligence (Princeton University) and The University of Texas at Austin |
LongProc (Long Procedural Generation) asks a model to execute a stated procedure and write a long structured output, not a short answer from a needle in a haystack. Six tasks: HTML-to-TSV extraction, line-by-line pseudocode-to-C++, path traversal on a one-out-edge city graph, theory-of-mind location/belief tracking, Countdown arithmetic search, and constrained travel planning. Difficulty is the required output length (about 500, 2K, or 8K Llama-3 tokens). English text. Rule-based scoring against a gold trace.
Prompt in, long structured generation out (TSV, C++, route listing, belief log, search trace, or itinerary). Paper evaluation uses greedy decoding and a 0.5K-1K token buffer; reasoning models may generate up to 16K tokens. lm-eval tasks are generate_until, num_fewshot 0, temperature 0, with max_gen_toks 1024 / 3072 / 9216 on the 0.5k / 2k / 8k YAML includes. `unsafe_code: true` because the pseudocode task compiles C++.
No model card in ModelSpec reports this benchmark yet.