Hugging Face released an experimental draft model called LFM2.5-VL-3B with a new technique called speculative decoding. This method speeds up inference without changing output quality. It achieves up to 3.13 times faster decoding on devices and 2.66 times on high-end GPUs. The model adds 280 million parameters, which is about 8.9% more than the original 3 billion parameters.
The model uses a shared architecture for both vision and language. It captures hidden states from fixed layers and conditions on them to draft tokens. It operates on hidden-state vectors of the same size, regardless of input type. The training involved multiple epochs on a mixture of vision-language data.
The draft model supports popular frameworks like llama.cpp, MLX-VLM, and SGLang from day one. It can run on various hardware, including edge devices and GPUs. Tests on six vision tasks showed significant speedups in decoding and overall latency.
However, speedups are limited by the time spent on vision encoding and prefill stages. These stages take up much of the total time, especially on less powerful hardware. Speculative decoding only speeds up the decoding stage, so overall gains depend on the workload.
The model is available in formats like Safetensors and GGUF. It is open-weight, meaning anyone can download, fine-tune, and deploy it. It aims to run efficiently anywhere, from edge devices to data centers.
Developers can try the model by following instructions for their preferred framework. They can also use it to improve the speed of vision-language applications while keeping costs low.
Source: https://huggingface.co/blog/LiquidAI/lfm2-5-vl-dspark
