A study investigates whether internal circuit overlap in large language models (LLMs) can predict their generalization across different input formats in arithmetic reasoning.
Using attribution patching, the researchers localized the circuits responsible for solving numeric and verbal arithmetic problems in three languages: English, Spanish, and Italian. They then tested whether the degree of overlap with the numeric circuit correlated with the model's performance on verbal formats.
Findings indicate that circuit overlap accounts for the relative difficulty of verbal formats and the models' ability to solve them correctly. This approach rivals supervised probes while requiring no labeled data, providing insights into model internals relevant for reasoning tasks.
Understanding these internal mechanisms can inform model evaluation and robustness, especially in multilingual and format-varied contexts.
Source: https://arxiv.org/abs/2609.04463