Global Speech Recognition Models Fail on Indian Languages
A national benchmark called "Voice of India" found that leading global speech recognition systems from OpenAI, Meta and Microsoft perform poorly on Indian languages, dialects and accents, while Indian models like Sarvam perform significantly better. The performance gaps, with error rates of 20-30% or higher, reportedly impact critical services including welfare applications, medical consultations and customer service.
What happened
A national benchmark called 'Voice of India' evaluated 15 Indian languages across approximately 35,000 speakers, testing Automatic Speech Recognition (ASR) systems from global providers including OpenAI, Microsoft, Meta and Google, as well as Indian models like Sarvam.
The benchmark, developed by Josh Talks in collaboration with AI4Bharat at IIT Madras, assessed real-world speech including code-switching and regional variations across different age groups, genders, regions, socio-economic backgrounds, device types and acoustic environments.
Reported impact
OpenAI's transcription models showed over 55 per cent Word Error Rate (WER) in Indian speech, while global models generally achieved 20-30 per cent error rates. In dialects like Bhojpuri and Maithili, some models failed to transcribe 2 out of every 3 words correctly.
Meta's error rates in Tamil and Malayalam were 2–3x higher than Indian systems, and Microsoft does not support 6 out of 15 tested Indian languages. These performance gaps impact critical services including welfare applications, medical symptom consultations, and customer complaint handling.
Index analysis
Sarvam's models consistently ranked first or second across nearly every language and dialect tested, including major languages like Hindi and Bengali as well as regional languages like Odia and Assamese.
Google Gemini remained competitive with Sarvam across the benchmark, while OpenAI's GPT-4o models trailed by over 50 percentage points in overall accuracy compared to Sarvam.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.