Using a behavioral framework called knowledge profiling and a new benchmark, WikiProfile, researchers find that state-of-the-art large language models (LLMs) more often fail to retrieve stored facts than to encode them. Evaluations over 2,150 Wikipedia-derived facts and roughly 4.5 million model responses across 13 LLMs show that encoding is near saturation in leading models, while recall remains a substantial bottleneck.
What is knowledge profiling?
Knowledge profiling changes the analysis unit from single questions to facts. Instead of judging whether a single question was answered correctly, the approach classifies each fact into one of five profiles: (1) encoding failure (fact not stored), (2) recall failure (fact stored but not accessible), (3) direct recall, (4) recall with thinking, and (5) inference without encoding. These profiles distinguish whether a fact is encoded and how easily it can be accessed, including whether access requires intermediate computation or chain-of-thought.
Behavioral definitions
- Encoding: A fact is encoded if the model can correctly reproduce it in contexts similar to pre-training (measured via proposition completion and contextual prompts that prime the model without giving the answer).
- Knowledge: A model "knows" a fact if it answers semantically equivalent direct and reverse questions correctly across phrasings.
- Recall: A model recalls a fact when it can produce a known, encoded fact; direct recall is immediate generation, while recall with thinking requires intermediate steps. Inference without encoding occurs when a model produces a correct answer via reasoning despite lacking an encoded fact.
WikiProfile benchmark
WikiProfile contains 2,150 naturally occurring facts extracted from Wikipedia. Each fact is paired with ten tasks: two for encoding, four for knowledge evaluation, and four multiple-choice recognition tasks. Questions (direct and reverse) were created through generation, refinement, and filtering, subject to search-engine grounded checks and manual validation to ensure unambiguous, minimal questions with unique answers. The extraction pipeline was automated and driven by Gemini-2.5-Pro with thinking; prompts were optimized on a held-out subset.
Evaluation setup
The study evaluated 13 LLMs, each tested with and without thinking. For each model/fact/task combination eight responses were sampled; automated autoraters graded about 4.5 million responses.
Main finding: recall is the bottleneck
Frontier models such as Gemini-3-Pro and GPT-5 show very high encoding rates (around 95–98%), yet they still fail to directly recall 26–34% of facts. Even when thinking is allowed, 11–12% of facts remain unrecalled. Scaling model size reduces encoding failures substantially, but recall failures persist and compose a larger share of residual errors. In short, factual errors in top LLMs are increasingly about using stored knowledge reliably rather than acquiring it.
Why recall fails
The study links recall difficulty to the conditions under which facts were learned: divergence between query phrasing, context, or ordering at inference time and the training-time context makes retrieval harder. Two systematic patterns stand out:
-
Rare facts: Low-popularity facts are encoded at rates close to popular facts, so encoding gaps are modest. However, recall gaps are larger for rare facts — many long-tail facts are present in parameters but harder to access.
-
Reverse questions: The so-called reversal curse — where a model knows "A is B" but fails on "What is A given B?" — appears mainly as a recall problem. In open-ended generation (recall), reverse questions are harder than direct ones; in multiple-choice recognition (verification), reverse questions are no harder and often easier. That dissociation shows the bidirectional knowledge is frequently present and recognizable, but generation-style recall suffers when the query direction differs from the learning context.
Thinking as a recovery mechanism
Enabling thinking (chain-of-thought or thinking-optimized inference) improves recall most where direct recall is weakest, notably for rare facts and reverse questions. In thinking-optimized models, thinking recovers roughly 40–65% of encoded-but-not-directly-known facts. It helps much less for facts that are not encoded. This pattern indicates that thinking primarily facilitates access to already encoded facts rather than mainly producing correct answers via long multi-hop inference. Thinking carries computational cost, and determining when to invoke it remains an open question.
Takeaway
Knowledge profiling provides a finer-grained diagnosis of factual behavior in LLMs. Applying it to Wikipedia facts suggests a shift in priorities: with encoding near saturation in frontier models, further gains in factuality may come less from scaling model size or adding data and more from improving retrieval and utilization of already encoded knowledge, including selective use of thinking-enabled inference.
