StreamAlign addresses limitations of existing text-aligned speech tokenization methods. These methods typically require complete utterances processed offline via automatic speech recognition (ASR), preventing real-time streaming. Furthermore, vocabulary mismatches between ASR and large language models (LLMs) reduce acoustic granularity. StreamAlign introduces a framework for streaming tokenization, combining character-level RNN-Transducer alignment with word-level ASR guidance. This mitigates vocabulary mismatch while maintaining recognition accuracy. The framework incorporates a proactive word boundary classifier to anticipate word completion, reducing tokenization latency from 560 ms to 270 ms. Evaluations on LibriSpeech demonstrate StreamAlign achieving the lowest word error rate (WER) and highest universal translation model operating similarity (UTMOS) among evaluated tokenizers. StreamAlign-SLM, a spoken language model trained on StreamAlign units, outperforms other end-to-end spoken language models in speech continuation, achieving the strongest overall consistency on SALMon and spoken StoryCloze.
Source: https://arxiv.org/abs/2609.09719