This paper introduces X-CoSD, a collaborative speculative decoding (CoSD) framework designed for large language model inference. The framework utilizes a small language model (SLM) on a device to draft candidate tokens, while a server LLM verifies them. Existing CoSD methods rely on a shared vocabulary and substantial communication due to residual resampling. X-CoSD addresses this by employing hybrid resampling, splitting the resampling process across the device and server based on vocabulary overlap. The system requires transmission only for the common-vocabulary region. An enhanced variant, X-CoSD-E, further improves efficiency through server resampling with device verification (SR-DV), where the server sends replacement candidates and probabilities for local device verification. Both X-CoSD and X-CoSD-E maintain the server LLM distribution and demonstrate significant speed improvements while preserving generation quality. This approach is relevant for deploying LLMs in resource-constrained environments.
Source: https://arxiv.org/abs/2609.09166