HELM generation task that asks whether two serialized table rows refer to the same entity, scored with exact match on Yes/No.
unassessed
| Category | reasoning |
|---|---|
| Subcategory | HELM Yes/No matching of pre-blocked Magellan/DeepMatcher row pairs |
| Page status | unknown |
| Metric | quasi_exact_match (schema); run spec attaches exact-match metrics; paper uses exact-match accuracy |
| Direction | higher_is_better |
| Unit | % |
| Publisher | Stanford CRFM (HELM) |
entity_matching is HELM's pre-blocked entity-matching scenario. The model reads two serialized rows and must answer Yes or No to "Are A and B the same?" The scenario class can load thirteen Magellan/DeepMatcher sets (structured, textual, and dirty). The HELM paper reports three: Beer, Abt-Buy, and dirty iTunes-Amazon. Pairs come already blocked; HELM does not run blocking. Train negatives are downsampled to the positive count before prompt sampling. English product, beer, and music rows. Open generation of Yes/No, not F1 over a ranked matcher. Laurel Orr implemented the scenario. It is not [entity_data_imputation](entity_data_imputation.md) and not Ditto.
Input pattern in the scenario: "Product A is …. Product B is …. Are A and B the same?". HELM default adapter: generation, instructions "Are Product A and Product B the same? Yes or No?", output noun Answer, max_train_instances 5, max_tokens 5, temperature 0. Scenario.name is entity_matching; run spec names include the dataset argument.
No model card in ModelSpec reports this benchmark yet.