When language models reason in chain-of-thought or exchange free-text intermediates, they serialize structured information into natural language. How much tree-structured compositional content survives this bottleneck? We propose a round-trip protocol that answers this question empirically for tree-structured expressions. A generator converts a procedurally generated arithmetic expression into a word problem, a separate extractor recovers the expression from the word problem alone, and symbolic equivalence provides an exact oracle. Evaluating all pairwise combinations of sixteen models yields…
Read the original at Apple machine learning research: The Communication Bottleneck: A Round-Trip Study of Tree-Structured Expression Serialization in Language Models
Source: https://machinelearning.apple.com/research/communication-bottleneck-serialization