Fine-tuning benchmark of 15 sub-7B-class models on 23 NLP datasets with 11 correctness, runtime, cost, energy, and CO2 metrics on four hardware setups.
unassessed
| Category | composite |
|---|---|
| Subcategory | small-model fine-tuning: correctness, compute, and energy across 23 NLP datasets |
| Page status | active |
| Metric | medal ranking over 11 metrics (accuracy, F1, BLEU, ROUGE, METEOR, perplexity, runtime, FLOP, cost, CO2, energy) |
| Direction | higher_is_better |
| Unit | medals / per-metric scores |
| Dataset size | 799594 |
| Dataset licence | CC-BY-SA-4.0 (arXiv preprint); upstream datasets keep their own licences |
| Publisher | FPT University; Aalborg University; Technische Universität Berlin / HiveIntel GmbH; RMIT University; DFKI / HiveIntel GmbH |
SLM-Bench compares small language models as fine-tuned task models, not as frozen zero-shot chat systems. Each run fine-tunes one of 15 open models on one of 23 English NLP datasets, then scores quality plus runtime, FLOPs, dollar cost, energy, and CO2. Tasks cover question answering, classification, NER, reasoning, math word problems, reading comprehension, and data-to-text. The point is the trade-off among accuracy, speed, and environmental cost under matched hardware, not a single exam score.
Load a dataset through the paper's unified loader, fine-tune with LoRA-style hyperparameters searched on a validation split, then score with the metric family for that task. Classification and QA use accuracy and F1. Generation and topic extraction use BLEU, ROUGE, METEOR, and perplexity. Resource metrics are taken from the host (Lightning AI in the paper) or from ML CO2 Impact and Zeus. The main paper reports NVIDIA L4; A10 and two Jetson Orin AGX sizes are claimed for the leaderboard.
No model card in ModelSpec reports this benchmark yet.