Audio Large Language Models (Audio LLMs) have improved in audio understanding but can still rely on textual cues or linguistic priors for answers. Training on data where answers cannot be inferred from text alone aims to address this issue.
The research investigates what internal processes enable the model to use audio effectively. It finds that replacing audio with silence or unrelated sounds causes larger performance drops in trained models compared to pretrained ones.
Acoustic information primarily influences the model's representations of answer choices in early-to-middle layers. Training increases the influence of audio on final predictions in middle-to-late layers. The learned weights significantly impact specific layer bands.
These findings clarify how training enhances the model's ability to utilize acoustic evidence, which is relevant for engineers optimizing Audio LLMs for audio-dependent tasks.
Source: https://arxiv.org/abs/2609.04637