Intent Recognition

A 693-item BIG-bench seven-way English intent task on SNIPS utterances, scored with multiple_choice_grade.

Also known as: BIG-bench intent_recognition, SNIPS intent recognition (BIG-bench)

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.
Categorydomain
Subcategoryseven-way English SNIPS intent classification (BIG-bench JSON)
Page statusunknown
Metricmultiple_choice_grade
Directionhigher_is_better
Unit%
Dataset size693
Dataset licenceApache-2.0
PublisherGoogle (BIG-bench collaboration); Hong Kong University of Science and Technology authors

What it measures

intent_recognition asks which of seven SNIPS intents an English utterance expresses: add_to_playlist, book_restaurant, get_weather, play_music, search_screening_event, search_creative_work, or rate_book. The BIG-bench README presents it as a few-shot dialogue-system probe in the GPT-3 style. It is not [banking77](banking77.md), which uses 77 banking labels.

Task format

Seven-option multiple choice with preferred metric multiple_choice_grade. task_prefix lists the seven intents. example_output_prefix is "Intent: ". append_choices_to_input is false. Canary GUID embedded. Dummy-model header reports 693 multiple-choice queries.

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub