LIBRA (Long Input Benchmark for Russian Analysis)

Russian long-context suite of 18 tasks (21 in the 2024 paper), scored mainly with exact match from 4k up to 512k tokens.

Also known as: Long Input Benchmark for Russian Analysis, LIBRA Mini

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	long-context
Subcategory	Russian long-context retrieval, QA, multi-hop, and counting (4k–512k tokens)
Page status	active
Metric	exact match (libra_score), with F1 on ruQasper and a count score on ruSciPassageCount
Direction	higher_is_better
Unit	%
Dataset size	15224
Dataset licence	MIT
Publisher	SaluteDevices, AIRI, MIPT, and Ecom.tech (Hugging Face org ai-forever)

What it measures

LIBRA tests whether a model can use a long Russian document, not a short prompt. Items pair a context with a question. Skills run from finding a planted passkey, through QA and multi-hop combination, to counting unique paragraphs. Contexts are binned by length from 4k tokens in the 2024 paper up to 128k, and in the May 2026 Hugging Face release up to 512k. The language is Russian. Several tasks are translations or adaptations of English long-context sets (QuALITY, BABILong, LongBench, L-Eval), not those English pages themselves.

Task format

Zero-shot Russian generation over a long context plus a question. lm-eval tasks set do_sample false and temperature 0. Output length caps vary by task (8 to 256 tokens in the YAMLs read here). A Hugging Face test split per config; optional filter on the length field (for example 8p, 32p).

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub