Japan Delays AI Child Abuse Risk Assessment Tool Over Accuracy Concerns
Japan's children and family affairs agency postponed deployment of an AI system designed to assess child abuse risk after trial testing revealed the system incorrectly assessed about 60% of cases and missed critical risk indicators, raising concerns about child safety.
What happened
Japan's children and family affairs agency (こども家庭庁) announced in December 2024 that it was delaying the deployment of an AI system designed to assess whether children suspected of abuse should be taken into temporary protective custody. The system was developed starting in 2022 with a budget of approximately 1 billion yen.
Trial verification in 10 municipalities using 100 past child welfare cases showed that about 60% of the AI's risk assessments were questioned by child welfare center senior staff. The AI failed to account for important risk factors, such as the depth of injuries, the specific location of injuries on the head or face, and qualitative assessments like a child expressing fear of their caregiver.
Reported impact
The system produced significant misassessments that endangered children. In concrete examples cited, the AI underestimated risk in cases where children reported severe violence such as 'being beaten to near death' or 'being struck on the head against the floor,' because the input form lacked appropriate fields to capture the severity and nature of injuries.
Index analysis
The system required staff to input many items (91 fields covering wounds, custody refusal, abuse history, and caregiver attitude) but could not adequately represent critical risk information, resulting in both false negatives and instances where the AI's scores contradicted professional judgment in nearly two-thirds of test cases.
Governance controls
こども家庭庁 decided to postpone deployment, determining that the system was not ready for release. The trial results were documented in reports released publicly by the agency and the Nomura Research Institute in late December 2024.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.