Mirac Suzgun and colleagues at Stanford University, with computing support from Together AI, evaluated how large language models (LLMs) equipped with web‑search tools perform on daily news questions.
The researchers framed the problem as three essential steps for answering factual questions not covered by a model’s training data: (i) receiving a well‑formed question, (ii) retrieving a relevant document, and (iii) extracting facts from that document. Any failure in these steps degrades final accuracy — for example, a question that embeds an incorrect name, retrieving a document that lacks the needed fact, or inability to process the document’s language.
Method
The authors ran daily tests from February 9 through February 22, 2026. Questions were based on BBC News reports in six languages: Arabic, English, French, Hindi, Russian, and Turkish.
Tested models (all with web‑search tools): Google Gemini 3 Flash and Gemini 3 Pro, xAI Grok 4, Anthropic Claude 4.5 Sonnet, and OpenAI GPT‑5 and GPT‑4o mini.
Each day Gemini 3 Flash produced 25 multiple‑choice questions per language (150 questions total per day). Questions included temporal context (for example, “Today is February 10, 2026…”) and offered five candidate answers; the correct choice was a verifiable detail such as a number, location, or quotation.
Models were evaluated on three formats: (a) the original five‑option multiple‑choice questions, (b) altered questions that introduced a single false premise (wrong actor, timeline, or detail) and added an “insufficient information to answer” option, and (c) the original prompts without supplied choices (free‑response).
To classify causes of incorrect outputs the authors used majority voting among Claude Opus 4.7, GPT‑5.4, and Gemini 3 Pro; errors were sorted into eight categories such as retrieval failures, retrieving a relevant source that contained different details than the one used to compose the question, inadequate comprehension, and mistaken timeline.
Results
- Overall performance: models did well on well‑formed English questions. The top four models exceeded 90% accuracy on the original multiple‑choice task: Gemini 3 Flash 95.6%, Grok 4 95.0%, Gemini 3 Pro 93.7%, and Claude 4.5 Sonnet 90.4%.
- GPT‑5 scored 85% and GPT‑4o mini 69% on the same task. With free‑response prompts, accuracy of all models dropped by 11 to 22 percentage points while their relative rankings remained similar.
- Language gaps: every model performed worst on Hindi questions (average 79.3%). Models often cited English‑language sources (for example, English Wikipedia) even when questions were asked in other languages, notably Hindi.
- Error causes: the most common cause of incorrect answers was retrieval failure (38.8%). The second most frequent cause was retrieving a topically relevant source that nevertheless contained “smart but wrong” details that did not correctly answer the specific question (32.7%).
- False premises: for altered questions containing a false premise, Grok 4 performed best at 70% accuracy. GPT‑5 reached only 19% accuracy, barely above random chance for six possible answers.
Implications
The study shows that LLMs augmented with web search can find factual information effectively, but their success critically depends on three components: receiving factual input, finding relevant documents, and correctly extracting facts. Because most failures were due to retrieval, the authors argue that improving retrieval infrastructure — better indexing coverage, improved ranking of sources, and stronger handling of non‑English queries — may yield larger gains than merely scaling model parameters for tasks involving timely information.
The authors also note an expanding opportunity: web search engines tailored for agentic (rather than human) use are a rapidly growing area, and there is considerable room to build better search systems designed for these agentic LLMs.
Details
- Test window: February 9–22, 2026
- Source base: BBC News articles in Arabic, English, French, Hindi, Russian, Turkish
- Models tested: Gemini 3 Flash, Gemini 3 Pro, Grok 4, Claude 4.5 Sonnet, GPT‑5, GPT‑4o mini
- Leading error modes: retrieval failures (38.8%), retrieving relevant but factually mismatched sources (32.7%)
(All percentages and results are drawn from the authors’ daily evaluations.)
