Skip to content
OpenSmartRoute

LLMs1 min read

MedWER: A Fixed Term List for Medical ASR Evaluation

MedWER is a new evaluation protocol and tool for medical ASR that addresses the limitations of traditional WER metrics by using a fixed, license-clean term list of 19,373 medical entities. This approach eliminates the reliance on versioned NER models and provides a more reliable measure of ASR accuracy in a clinical setting.

By OpenSmartRoute editorial · written through the router by writer-small

From arXiv cs.CL - “MedWER: A Reproducible, Model-Free Evaluation Protocol for Medical Speech Recognition”

The research identifies a key issue with existing word error rate (WER) metrics in medical speech recognition: the denominator, often reliant on named-entity recognition (NER) models, introduces variability and obscures clinically relevant errors. The protocol introduces MedWER, an evaluation tool utilizing a fixed term list of 19,373 drug, diagnosis, symptom, and injury-mechanism entries derived from public sources. This list serves as the denominator, ensuring consistency across evaluations. The protocol combines a pinned text normalizer with a phrase-aware term-restricted WER, focusing on accuracy within the defined term list. Validation against an independent provincial drug-benefit file confirms the list’s accuracy. Results are reported with 95% confidence intervals from resampled per-utterance scores, comparing Moonshinebase, Whisperbase.en, and MedASR on two open benchmarks.

Source: https://arxiv.org/abs/2609.05728

Published Sep 9, 2026 · updated Sep 9, 2026 · 120 words

Keep reading

Related posts

More in LLMs

LLMs1 min read

Document-Level MT Evaluation Shows Statistical Equivalence

Research found that document-level machine translation evaluation, presenting full documents to annotators, yields statistically equivalent scores and rankings compared to segment-level evaluations. This suggests current document-level systems and associated metrics may not be accurately measuring intended aspects of translation quality.

LLMs1 min read

UniRRM: Unified Reasoning Reward Models

Researchers introduced UniRRM, a multilingual reasoning reward model and dataset, to improve reward model reliability in open-ended tasks. UniRRM achieves performance comparable to state-of-the-art models across benchmarks and supports diverse evaluation paradigms.