Eleven real-world English classification tasks with 50 labeled examples each; official score is macro-F1 on a hidden test set.
unassessed
| Category | domain |
|---|---|
| Subcategory | 11 real-world English few-shot text classification tasks with 50 public labels each |
| Page status | unknown |
| Metric | macro-F1 averaged over 11 tasks (official); HELM quasi_exact_match / exact match |
| Direction | higher_is_better |
| Dataset size | 29262 |
| Dataset licence | other (per-task mix: CC-BY-4.0, CC-BY-SA-4.0, CC-BY-NC-4.0, MIT/CC-BY-4.0, unlicensed) |
| Publisher | Ought |
RAFT (Real-world Annotated Few-shot Tasks) is a meta-benchmark of 11 naturally occurring English classification problems. Each task ships 50 labeled training examples, chosen at random rather than class-balanced, plus an unlabeled test set. Tasks include adverse-drug effect detection, 77-way banking intent, NeurIPS impact-statement risk, reading-level labelling, legal overruling, semiconductor organisation type, systematic-review inclusion, TAI-safety bibliography tagging, terms-of-service unfairness, tweet hate, and tweet complaint detection. The intended skill is few-shot classification that mirrors deployment, including long inputs and many labels, not a synthetic GLUE-style suite. This is not retrieval-augmented fine-tuning, which shares the acronym.
Label-string classification. Official evaluation uploads test labels for weekly scoring. HELM run name raft:subset=<id> generates a Label field (max_tokens 30) with default five in-context examples from a resplit of the 50 public labels, and scores exact match rather than macro-F1. HELM subsets: ade_corpus_v2, banking_77, neurips_impact_statement_risks, one_stop_english, overruling, semiconductor_org_types, systematic_review_inclusion, tai_safety_research, terms_of_service, tweet_eval_hate, twitter_complaints.
No model card in ModelSpec reports this benchmark yet.