Sentence-level extraction of KPI names and values from SEC 10-K filings; an information-extraction task, not a reasoning benchmark, and HELM's version tests only a simplified 4-tag NER slice of the original 12-tag task.
unassessed
| Category | domain |
|---|---|
| Subcategory | named entity recognition and relation extraction of key performance indicators from SEC 10-K filings |
| Page status | active |
| Metric | adjusted (word-weighted) F1 |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 1355 |
| Dataset licence | The GitHub repository carries a repository-wide MIT licence (confirmed via the GitHub API); the paper itself does not separately state a licence for the released annotation data. |
| Publisher | University of Bonn and Fraunhofer IAIS, Germany (with Hochschule Bonn-Rhein-Sieg) |
KPI-EDGAR gives a model a single sentence from a real US company's 10-K annual report and asks it to pull out the key performance indicators (KPIs) mentioned -- things like revenue or net sales -- along with the numeric or monetary values attached to them, and to label each value by what period it refers to (current year, prior year, or two years prior). This is an information-extraction task: the model is not asked to reason about, summarise or judge the filing, only to identify and tag specific spans of text. The original paper frames it as joint named entity recognition (NER) and relation extraction, since a KPI and its value are linked as a pair rather than tagged independently; the HELM implementation used by most current harness runs simplifies this to plain NER over four of the original twelve entity types, without the relation-extraction step.
Given a sentence, generate a comma-separated list of extracted phrases, each tagged in brackets with one of a fixed set of entity types (HELM's simplified version uses four: kpi, cy, py, py1). The original paper's full task additionally pairs each value entity to the KPI entity it belongs to and covers eight further entity types (rate-of-change and coreference tags among them).
No model card in ModelSpec reports this benchmark yet.