Copyright (HELM memorisation / extraction)

HELM scenario that feeds a book or Linux-kernel prefix and scores how closely the model continues the copyrighted remainder.

Also known as: copyright_text, copyright_code, HELM copyright

unassessed

This page is a discovery lead. Nobody has yet assessed it against the catalogue contract, so it carries no disposition. Absence of evidence here is not evidence of staleness.

Category	safety
Subcategory	HELM prefix-continuation extraction attack on books and GPL kernel code
Page status	unknown
Metric	longest_common_prefix_length (also edit_distance, edit_similarity)
Direction	lower_is_better
Dataset licence	Not an open dataset licence: extraction targets are copyrighted books and GPL Linux kernel source hosted as Google Drive JSON; HELM code is Apache-2.0
Publisher	Stanford CRFM (HELM); methodology from Carlini et al. 2021

What it measures

copyright is HELM's targeted memorisation test, not a new literary corpus. The model is given a short prefix drawn from BookCorpus-style books, a popular-book list, or Linux kernel source, and must continue the text. Overlap with the held remainder is treated as evidence of extraction, after Carlini et al. 2021. HELM splits the runs into copyright_text and copyright_code groups. This is not HarmBench's copyright behavior slice.

Task format

Zero-shot completion: the prompt is the prefix with no instruction wrapper (get_completion_adapter_spec, max_train_instances 0). Default temperature 0.2 and max_tokens 1024. Run name copyright:datatag=...

Models reporting this benchmark

No model card in ModelSpec reports this benchmark yet.

Data

This page as JSON · Edit on GitHub