Whisper, a foundation model for automatic speech recognition, can produce hallucinated transcripts, especially with inputs containing little or no speech. A new method applies low-rank projection of decoder activations during inference to mitigate this issue.
The approach estimates a compact hallucination-associated subspace from non-speech calibration data. During inference, decoder hidden states are projected away from this subspace. Two variants are evaluated: always-on projection, which applies to all inputs, and gated projection, which applies only when non-speech is detected.
Results show that always-on projection reduces hallucination rate from 31.31% to 2.44%, while gated projection reduces it to 3.74%. Gated projection also maintains lower false rejection rates of genuine speech and causes a small increase in word error rate.
This method offers a way to suppress hallucinations without retraining, providing a controllable balance between hallucination reduction and speech recognition performance.
Source: https://arxiv.org/abs/2609.04561