Abstract:
Despite the rapid development in the field of automatic speech recognition (ASR) systems, the recognition quality remains poor for audio recordings with acoustic degradation (music, crowd shouts, sounds of machinery, etc.). This becomes especially important when implementing models in critical areas that are characterized by increased attention to control and reliability of the results (aviation, medicine, autopilots, etc.). Error in such areas is very expensive and the use of models becomes impossible, even if the error occurs rarely. To reduce risks, it is advisable to evaluate the expected recognition quality in advance.
This article proposes an approach to predicting the Word Error Rate (WER) based on the acoustic characteristics of the signal and calculating the perplexity of language models. The proposed method involves the creation of diverse sets of audio data by applying various types of acoustic observations to pure speech samples at various levels of quality and intelligibility. Unlike previous studies, a complete set of speech features is extracted and analyzed: prediction of the value of the signal-to-noise ratio (SNR), neural network sound quality metrics (NISQA, etc.) as well as the perplexity of the text of the ASR hypothesis using the language model as an additional feature for training a unified model.
Experiments are being conducted using modern speech recognition architectures to demonstrate the effectiveness of the proposed method in predicting WER in various acoustic conditions. It is shown that the inclusion of perplexity significantly improves the quality of the WER prediction, in particular for data where acoustic features are weakly correlated with recognition errors. The results are applicable for automatic evaluation of the expected quality of speech recognition and filtering of audio inputs.