OpenAI Bans Accounts for Surveillance Tool Development Requests
OpenAI reported banning accounts linked to Chinese government institutions and Russian criminal groups for seeking help developing surveillance systems, phishing tools, and malware through ChatGPT. The company said it had identified requests for AI-driven social media monitoring tools and systems to track Uyghur populations but that its models refused to publicly assist with malicious requests.
What happened
OpenAI reported on October 7, 2025 that it had banned multiple accounts suspected to be linked to Chinese government institutions. According to OpenAI's Chief Investigator Ben Nimmo, the accounts had requested assistance in developing social media surveillance systems, opinion monitoring proposals, and large-scale tracking tools in violation of OpenAI's national security policy.
One account requested help designing promotional materials and project plans to develop an AI-driven social media monitoring tool (described as a 'detector') to scan social media platforms including X, Facebook, Instagram, Reddit, TikTok and YouTube for extremist speech, ethnic, religious and political content for government customers. Another account requested help drafting a proposal for a 'High-Risk Uyghur-Related Inflow Warning Model' to analyze transportation booking records matched against police records to track Uyghur populations.
OpenAI also identified multiple Chinese-language accounts using ChatGPT to assist with phishing and malware activities, and accounts researching the Chinese AI company DeepSeek. OpenAI discovered that Russian-language accounts linked to criminal groups used ChatGPT to develop malware including remote access trojans and credential theft tools.
Reported impact
OpenAI stated that its models successfully refused to make public malicious requests and there was no evidence that the models provided assistance in enhancing new attack capabilities for criminals.
OpenAI's Chief Investigator Ben Nimmo told CNN that 'there are forces within the Chinese government pushing AI to play a greater role in mass surveillance. China was already surveilling its people, and now they want AI to make the system more efficient.'
Index analysis
OpenAI has reported over 40 network accounts violating its usage policy since February 2024. The company noted that while ChatGPT is not formally offered in mainland China, the users likely accessed the service through VPN or other means.
Governance controls
OpenAI permanently banned the two users associated with the surveillance tool requests. The company stated that it has been publicly publishing threat reports since February 2024 documenting policy violations.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.