Chatbots demonstrate significant caste bias in India
Researchers found that OpenAI's ChatGPT and Sora, alongside Sarvam AI's Sarvam-M, reportedly generate and reinforce harmful caste-based stereotypes in India, associating Dalits with negative traits while privileging upper castes. The systems were observed changing names in academic applications to match high-caste preferences, perpetuating discrimination that discourages marginalised individuals from pursuing opportunities.
What happened
OpenAI's ChatGPT and text-to-video generator Sora, along with Sarvam AI's Sarvam-M, were found to consistently reflect and reinforce caste-based prejudices, according to the MIT Technology Review and researchers.
The systems generate and perpetuate centuries-old oppressive stereotypes that associate marginalised castes, particularly Dalits, with negative traits such as being impure, uneducated, and limited to menial jobs, while privileging upper castes like Brahmins with positive attributes.
Researcher Dhiraj Singha, a Dalit sociologist, experienced AI changing his surname to a high-caste name when polishing his academic applications, demonstrating how the systems reinforce societal biases about who is deemed acceptable for academic success.
Reported impact
For Dalits and other marginalised castes, the biases perpetuate discrimination and exclusion, potentially influencing opportunities and reinforcing social stigma.
Unchecked caste bias in AI systems risks scaling deep-rooted inequalities systemically through technology increasingly embedded in everyday life.
Index analysis
The root cause lies in AI training on uncurated web-scale data that reflect historical and societal prejudices including caste-ism, which remain largely unaddressed by developers.
Despite India being OpenAI's second-largest market, testing for and mitigating caste bias appears not to be a priority, and safety filters have apparently been weakened in the latest GPT-5 model to allow stereotypical outputs rather than refusing to answer problematic queries.
Governance controls
The AI industry's neglect of non-Western biases like caste—in contrast to more attention on gender and racial bias—represents a serious governance and accountability gap.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.