The research investigated how Large Language Models (LLMs) handle deictic ambiguity within a draft-verify-revise inference pipeline. This pattern involves an initial draft generation, a critique and feedback stage, and a final revision. The study examined whether LLMs at each stage could correctly resolve context-dependent expressions, such as "previous," which can have multiple interpretations. A synthetic dataset of 10 base examples was used, rendered under three distinct conditions. These conditions manipulated which LLM – the draft stage assistant or the critique stage grader – resolved the expression correctly, and the degree of independent reasoning required by the meta-evaluator during the revision phase.
Six LLMs from three providers were tested across 21 reasoning effort configurations. Evaluation metrics included balanced accuracy, measured as the unweighted mean of sensitivity and specificity. Initial results showed balanced accuracy ranging from 0.156 to near-perfect levels. GPT-5.2 demonstrated significant improvement, rising to 0.942 with increased reasoning effort, while Gemini 3 Pro maintained accuracy above 0.94 across all levels. Notably, at low reasoning effort, Gemini 3 Pro outperformed GPT-5.2 at higher effort levels by approximately 5% in terms of cost per trial.
Analysis of the meta-evaluator’s reasoning revealed a tendency to rely on surface cues rather than operational reasoning when errors occurred. This suggests potential vulnerabilities in the pipeline if the meta-evaluator’s judgment is flawed. Context engineers implementing draft-verify-revise pipelines should consider this and make the intended referent explicit at each stage to mitigate the risk of deictic shifts.
The study included an ablation experiment that removed error classification labels from the grader’s feedback. The dataset comprised 10 base examples, rendered in three conditions. The research utilized e-values for sequential testing. The total paper included 86 pages, including supplementary materials, 4 figures, 10 supplementary figures, 17 supplementary tables, and 11 supplementary exemplars. Code and data were available via a provided URL.
Source: https://arxiv.org/abs/2609.12162
