On September 23, 2023, Anthropic published a quantitative case study describing two prompting techniques that increase Claude’s ability to recall information from very long contexts. The two methods are: extracting relevant reference quotes into a scratchpad before answering, and supplementing the prompt with examples of correctly answered questions about other sections of the document (few‑shot contextual examples).
What they tested and why
The experiment aimed to measure how to maximize Claude’s chance of correctly retrieving a specific fact from a long document. As the underlying test source they used a publicly available daily government document containing meeting transcripts across departments. Anthropic selected the July 13, 2023 edition — deliberately after the models’ training cutoff — to reduce the likelihood that Claude already knew the content.
To build a dataset of Q&A pairs they split the document into sections and asked Claude to generate five multiple‑choice questions per section, each with three wrong answers and one correct answer. They then assembled randomized sets of these sections into long “collage” documents and used those stitched documents to test recall.
Challenges in question generation
Anthropic reports several pitfalls encountered when getting Claude to write evaluation questions:
- Claude sometimes produced questions it could answer without any document reference (e.g., general knowledge questions).
- Certain question types (for example, counting words or tokens) are inherently difficult for the model.
- Stylistic cues in answer choices (the correct answer being more detailed) unintentionally hinted at the right option.
- Ambiguous references such as “this document” or “this passage” become problematic when multiple documents are stitched together, because the model cannot know which passage is meant.
To mitigate these issues they used a prompt template containing two sample meeting chunks and hand‑written sample questions as few‑shot examples, plus guidance instructing Claude to explicitly identify the passage to which a question refers.
Evaluation setup
The study focused mainly on the smaller Claude Instant model (version 1.2) rather than Claude 2. Anthropic explains this choice by noting Claude 2 is already strong at recalling information after long reads, whereas Claude Instant benefits more from improved prompting and is fast enough to let others reproduce the evals easily.
Key evaluation details:
- When given only the exact passage used to generate a question, Claude Instant answered its own generated question correctly about 90% of the time. The 10% of questions it got wrong in that short‑context setting were discarded from the long‑context evaluation as being too difficult.
- When given a random section that did not contain the source material for a question, Claude guessed the correct answer about 34% of the time (vs. 25% expected by chance), indicating some cases of intuition from general knowledge or subtle cues.
- Long documents were constructed by stitching random passages until reaching a target token count; Anthropic reports tests on collages roughly 75,000 and 90,000 tokens, and elsewhere compares 70K and 95K token contexts to measure context‑length effects.
Because the stitched contexts could contain many similar passages, ambiguous phrasing in questions became a real problem. For that reason the question generation prompt instructed Claude to name the passage clearly (e.g., “What is the publication date of the notice about additional in‑season actions for fisheries?” instead of “What is the publication date of this notice?”).
The researchers compared four prompting strategies when asking Claude to answer questions about the long collages:
- Base – simply ask Claude to answer.
- Nongov examples – two fixed examples of correctly answered general‑knowledge multiple choice questions unrelated to the government document.
- Two examples – two dynamically chosen examples of correctly answered questions about other chunks in the same context.
- Five examples – same as #3 but with five examples.
Each strategy was tested with and without a scratchpad instructing Claude to extract relevant quotes. They also varied the position of the passage containing the answer (beginning, middle, end) and tested across different total context lengths (reported comparisons include ~70K and ~95K tokens as well as ~75K and ~90K in parts of the writeup). Claude Instant 1.2 was the primary testbed; Anthropic also reports results for Claude 2 on the baseline and the best performing strategy for Claude Instant.
Results and observations
Main findings from the experiment include:
- Using a scratchpad to pull relevant quotes and including contextual examples generally improved accuracy across tested context lengths.
- Pulling relevant quotes helped in all head‑to‑head comparisons; it increases latency slightly but improves accuracy. For Claude Instant the latency cost is small.
- Contextual examples help on both ~70K and ~95K tests; more examples performed better than fewer.
- Generic examples demonstrating external general knowledge did not improve performance.
- For Claude Instant there was a roughly monotonic inverse relation between performance and the distance from the relevant passage to the question/prompt end: the farther away the answer, the worse the recall. Claude 2 showed a different pattern with a small dip in the middle for the 95K context.
- Performance on passages at the very end of the stitched context sometimes degraded when examples were added, possibly because adding examples increases the distance between the final passage and the answering instructions.
A specific quantitative note: Claude 2’s accuracy improved from 0.939 to 0.961 with prompting improvements. Anthropic frames that change as a 36% reduction in errors, despite what looks like a modest absolute increase in accuracy.
Practical takeaways
Anthropic summarizes actionable recommendations for writing long‑context Q&A prompts:
- Use many contextual examples and a scratchpad for best performance across long contexts.
- Extracting and presenting relevant quotes is beneficial across comparisons; the tradeoff is a small latency increase.
- Contextual examples (drawn from the same long context) improve accuracy; more examples are better.
- Generic, non‑document examples do not appear to help.
- Place instructions at the end of the prompt to maximize recall of the task specification.
Anthropic Cookbook and reproducibility
Anthropic published fully reproducible code for the experiment in the Anthropic Cookbook. The Cookbook also includes other recipes such as a Search and Retrieval demo for Wikipedia and guidance for implementing mock‑PDF upload functionality via the Anthropic API. Anthropic states it plans to expand the Cookbook and other prompt engineering resources; interested developers can register to request access to the Claude API.
Notes on noise and related work
Anthropic notes sources of noise in the experiment: many of the questions the model fails on involve counting, and in some cases the initially designated “correct” answer (from the generated QA pairs) was itself incorrect. They also reference a recent paper that reported a U‑shaped relationship between performance and position in context for a similar task; Anthropic suggests differences in average example length between that paper and their much longer contexts (15K tokens vs. 70–95K tokens) may explain different positional patterns.
Overall, the study provides reproducible guidance showing that careful prompt design — specifically extracting relevant quotes and using contextual few‑shot examples — can measurably improve retrieval of facts from very long documents, especially for smaller, faster models like Claude Instant 1.2.
