Xinyue Liu and colleagues at Stony Brook University, Carnegie Mellon University, and Columbia Law School studied what happens when large language models are fine-tuned on a task useful to writers: expanding short plot summaries into paragraph‑length prose. They found that this apparently benign task can cause models to reproduce substantial, verbatim passages from books the models saw during pretraining.
Method
- Models: the team fine‑tuned DeepSeek‑V3.1, Google Gemini 2.5 Pro, and OpenAI GPT‑4o. GPT‑4o was also used to generate the summaries used in training data construction.
- Dataset construction: the researchers built summary–paragraph pairs from dozens of contemporary novels. Books were split into paragraphs of 300–500 words; for each paragraph GPT‑4o produced a plot summary about half the length of the excerpt (for example, noting narrator perspective and key thematic points).
- Fine‑tuning objective: models were trained to reverse the process: given a plot summary and an author name, generate the original paragraph.
- Evaluation: for each prompt (summary + author) the team generated 100 outputs and measured how much the outputs matched text in the original books. They used a metric called book memorization coverage (BMC), defined as the share of words in a book reproduced by the model in a contiguous span; only spans of five or more words were counted (BMC@5).
Results
- Baseline: GPT‑4o without fine‑tuning produced little verbatim text, with a 7.36% BMC@5.
- After fine‑tuning, all three models produced large amounts of verbatim text.
- Same‑author generalization: when fine‑tuned on pairs from one book and tested on other books by the same author, BMC@5 exceeded 40% for 10 out of 30 books.
- Murakami experiment: fine‑tuning on Haruki Murakami novels and testing on 51 books by 32 other authors produced BMC@5 above 40% for 36 of the 51 books; in one case GPT‑4o reached 91.9% BMC@5.
- Reproduced lengths: models generated contiguous verbatim spans as long as 440 words.
- Controls: fine‑tuning on Virginia Woolf’s public‑domain novels produced similar reproduction rates, while fine‑tuning on synthetic data yielded near‑zero BMC@5. This pattern indicates the fine‑tuning taught models to decode and output text strings they had encoded during pretraining, rather than to invent new, unique paragraphs based on summaries.
Why this matters
The study confirms that alignment measures such as system prompts and fine‑tuning for human preferences are brittle: they do not remove memorized text from model weights and can be undone by further fine‑tuning on tasks that require verbatim generation. This creates a potential loophole for accidental leakage or deliberate adversarial misuse. The ease with which fine‑tuning can disable anti‑plagiarism guardrails matters both for organizations deploying fine‑tuned models and for model providers that permit customer fine‑tuning.
Authors’ remarks and legal considerations
The paper does not report whether the models would have produced the same degree of verbatim output without being given the author’s name in the prompt. The authors also note, as non‑lawyers, that training on publicly available text may be considered fair use, but they emphasize that models should not freely reproduce copyrighted works without permission.
Conclusion
Fine‑tuning on a task requiring verbatim reconstruction can cause LLMs to reveal long passages they memorized during pretraining. The result highlights the need to evaluate safety, legal and operational risks specifically for fine‑tuned models, rather than assuming alignment measures remain effective after customization.
