Identifying the Root Cause of Factual Errors
Recent research from Google Research has identified a critical bottleneck in the factuality of large language models (LLMs): recall. The study, published in August 2026, focuses on frontier LLMs such as Gemini-3 and GPT-5, revealing that these models often encode a vast amount of factual information but struggle to retrieve it when needed. This contrasts with the assumption that simply increasing model size or expanding data coverage would automatically resolve factual inaccuracies.
The Knowledge Profiling Framework
The research introduces a ‘knowledge profiling’ framework designed to measure both encoding and recall. This framework utilizes a new benchmark, WikiProfile, containing 2,150 Wikipedia-derived facts, each paired with ten questions designed to assess encoding, recall, and recognition. The framework classifies facts into five profiles: encoding failure, recall failure, direct recall, recall with thinking, and inference without encoding. This shift in analysis focuses on whether a fact is accessible rather than simply whether a specific question was answered correctly.
Operationalizing Recall
The study operationalizes recall through three key behavioral notions: Encoding, Knowledge, and Recall. Encoding refers to the ability to reproduce a fact in a pre-training-like context. Knowledge represents the ability to answer semantically equivalent questions across different phrasings. Recall is defined as knowing an encoded fact, categorized as either direct recall (without thinking) or recall with thinking (requiring intermediate computations).
Evaluation and Results
Experiments evaluated 13 LLMs, including Gemini-2.5-Pro and GPT-5, both with and without enabling ‘thinking’. The evaluation process generated approximately 4.5 million responses, graded automatically by prompted LLM autoraters. The key finding was that factual encoding is close to saturation across these models, while recall remains a significant bottleneck. The research highlights the importance of improving the mechanisms that allow LLMs to effectively utilize the factual knowledge they have already encoded.
