Pause-token methods insert special tokens into sequences to improve reasoning in large language models. This study investigates how pause tokens influence the training process, beyond their inference-time use.
Two controlled experiments reveal that masked pauses at reasoning boundaries help retain learned information and encode more downstream-step data. This approach, called Masked Boundary Pause (MBP), consistently improves reasoning performance across models.
The findings suggest that pause tokens act as a training-dynamics intervention, balancing retention and adaptation. This strategy benefits models like Qwen and Llama, with gains in math and code reasoning tasks while maintaining general language understanding.
Extending these results, the mode-preserving approach also enhances other training strategies, such as GRPO. These insights inform how pause tokens can be used to optimize training dynamics in large language models.
Source: https://arxiv.org/abs/2609.04489