BAAI TACO: 1,000 competition-style Python problems (plus a 25k-problem train split) scored by executing generated code as pass@k.
unassessed
| Category | coding |
|---|---|
| Subcategory | competition-style Python generation graded by test-case execution |
| Page status | active |
| Metric | pass@k (OpenCompass TACOEvaluator; FlagOpen compute_metric.py) |
| Direction | higher_is_better |
| Unit | % |
| Dataset size | 26443 |
| Dataset licence | Apache-2.0 |
| Publisher | Beijing Academy of Artificial Intelligence (BAAI) / FlagOpen |
TACO gives a model a natural-language programming-contest statement, optional starter code, and a call-based or stdin convention, and requires a Python program that passes the problem's hidden tests. Items come from contest sites rather than interview-style function stubs. Each row also carries topic, algorithm, skill, and difficulty labels so reporters can slice the 1,000-item test set. English problem text, Python solutions. This is not [mc_taco](mc_taco.md) (temporal commonsense) and not [tac](tac.md) (travel agents).
Free-form Python generation. OpenCompass TACODataset loads Hugging Face BAAI/TACO, evaluates the test split, and prepends starter plus "Use Standard Input format" or "Use Call-Based format". The default taco_gen_c7893a config is zero-shot, max_out_len 512, num_repeats 1. taco_levels_gen_411572 runs one config per difficulty with max_out_len 1024.
No model card in ModelSpec reports this benchmark yet.