The research introduces Osprey, a system designed to improve the performance of drafters used in speculative decoding. Drafters are typically trained against a narrow distribution for a single target model, leading to a collapse in acceptance rates when the workload changes. The authors argue that pretraining, a standard practice for target models, has been difficult to apply to drafters due to target-specific training recipes. Osprey addresses this by bootstrapping drafters from off-the-shelf pretrained small language models, treating broad pretraining as a reusable asset. The system employs pruning, next-token pretraining, and vocabulary alignment to adapt to each target. Empirical results show a 16.1% improvement in mean acceptance length for Qwen3-8B, 21.2% for Llama-3.3-70B-Instruct, and 22.7% for the 229B MiniMax-M2.5, alongside a 17.5% increase in tokens per second. These gains were most pronounced on out-of-domain and multilingual data. The code is available at https://github.com/LeanModels/Osprey.
Source: https://arxiv.org/abs/2609.09338