DUK researchers discover error in OpenAI’s Whisper

Team says OpenAI’s accuracy-checking process fails to account for vowel signs used in Indian languages, including Malayalam

Researchers at Digital University Kerala (DUK) have uncovered a major error in OpenAI’s Whisper, an artificial intelligence system designed to transcribe speech into text.

Typically, AI companies test the accuracy of AI models, usually automatically, before releasing them for public use. This is to assess how precisely these systems handle spoken language. However, owing to an error in the quality-check process, OpenAI has reported the accuracy of its AI systems for Indian languages, including Malayalam, as better than it actually is.

This discovery was made by researchers at DUK’s Virtual Resource Centre for Language Computing (VRCLC), led by Elizabeth Sherly, with team members Kavya Manohar and Leena G. Pillai. Their research paper documenting the problem has been selected for presentation at the ongoing international conference on Empirical Methods in Natural Language Processing 2024 in Florida, U.S. The Association of Computational Linguistics (ACL) has also awarded the team a grant to support their presentation.

During their investigation, the researchers found that OpenAI’s accuracy-checking process fails to account for vowel signs used in Indian languages. For instance, when the Malayalam word ‘Digital University’ is stripped of its vowel signs and ‘chandrakkala’ (virama), it becomes ‘ഡ ജ റ റ ൽ യ ണ വ ഴ സ റ റ,’ losing its readability. By excluding these signs, errors go undetected, leading to an overestimated transcription accuracy. The team also observed that similar issues are present in Meta’s evaluations of its own AI models.

Regional language AI computing differs vastly from systems designed for languages such as English. The DUK’s VRCLC centre is committed to developing systems that account for these linguistic nuances, enhancing accuracy and usability of Indian languages in AI applications, a statement from the university said.

Published - November 15, 2024 08:42 pm IST