Oxford and Kentucky researchers expose regional stereotypes in ChatGPT
Researchers at Oxford and the University of Kentucky bypassed ChatGPT's safeguards through jailbreak prompting and found the chatbot encodes regional and racial stereotypes in its rankings of U.S. states and cities, reflecting biases in its training data rather than factual differences among populations.
What happened
Researchers from the University of Oxford and the University of Kentucky developed a methodology to jailbreak ChatGPT's safeguards by asking more than 20 million forced-choice questions to extract the chatbot's implicit biases about U.S. states and cities.
The researchers found that ChatGPT holds stereotypical views of various regions, ranking Mississippi as having the laziest population and assigning negative attributes such as 'annoying,' 'unfriendly,' and 'stressed' to Philadelphia. The findings were published on a website called inequalities.ai under the project name Silicon Gaze.
Reported impact
The analysis reveals that ChatGPT perpetuates regional and racial stereotypes derived from its training data, with potential influence on the chatbot's hundreds of millions of weekly users.
The researchers noted that ChatGPT's ranking of Mississippi may stem from historical biases against Black people and the Deep South, despite the absence of evidence that lower-income households are lazier than wealthier ones.
Index analysis
Matthew Zook, a University of Kentucky professor and study author, stated that the findings demonstrate how AI bots learn and perpetuate human biases even when explicitly programmed to avoid expressing them, with more prevalent stereotypes more likely to appear in the model.
OpenAI maintains that regional stereotypes are not intentionally programmed into the bot, but acknowledged that if stereotypes are present in the training text, they can significantly influence the chatbot's responses.
Governance controls
ChatGPT includes built-in safeguards designed to prevent the chatbot from directly answering questions about which states or cities possess undesirable traits, though researchers were able to circumvent these protections through systematic forced-choice prompting.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.