This research investigated the process of identifying and classifying errors in Quran recitation transcripts. A human annotation effort was conducted on 100 production recording cases, resulting in 348 scored units and 162 localized events across ten defined labels. The annotation process included distinguishing between unresolved mistakes, repetitions, repairs, and accepted spelling differences. An executable evaluator was developed to score these labels and word positions, providing a baseline for comparison.
Initial pilot testing involved eight single 20-minute runs across three coding agents and eight models. The pilot produced label-aware F1 scores ranging from 0.143 to 0.892. Seven models performed above all baselines, while one model’s performance fell below the naive diff due to a missing normalization step. The pilot measured only half of the task, as annotation was built before running the experiment.
Analysis of the 972 gold-event instances revealed that the majority (970) had overlapping predictions. The remaining instances were determined to be related to span extent and the boundaries of labels, which are subject to adjudication rather than explicit text representation. Seven of the 162 events consistently defeated all six runs, with five related to specific orthographic rules. Despite a strong run, the same events remained undetected.
Correcting the adapter’s word coordinates recovered all five annotated repetition events. This highlights the need for careful verification of annotation interfaces before interpreting baseline failures. The pilot study demonstrated that the algorithm only addressed half of the task, as annotation was built before running the experiment. Source: https://arxiv.org/abs/2609.12085