Researchers warn of "X-hacking" risk in explainable AI systems
Researchers at the German Research Center for Artificial Intelligence warned of "X-hacking," a risk in explainable AI systems where models make correct predictions but provide misleading explanations. The practice—involving cherry-picking or automated searching for models with specific explanation patterns—poses particular concern in medical and social science applications where explanations guide critical decisions.
What happened
Researchers from the German Research Center for Artificial Intelligence (DFKI) reported at an international machine learning conference that 'X-hacking' is an overlooked risk in explainable AI, in which AI systems make correct predictions but explain them in completely different ways.
X-hacking describes two mechanisms: cherry-picking models with the strongest explanatory power that support expected results, and automated search where AutoML systems can optimize both predictive performance and find models with specific explanation patterns. The risk of these practices is often underestimated.
Reported impact
Even when models produce nearly identical results, feature importance can vary significantly, a concern particularly sensitive in medical research and social science applications where explainable models often form the basis for critical decisions.
Index analysis
The term X-hacking derives from statistical P-hacking, where researchers use data manipulation techniques to achieve statistically significant results that may lack actual meaning. This can lead to false positive findings and is a form of data manipulation.
Governance controls
DFKI researchers recommend that disciplines using AutoML should be aware of methodological risks rather than simply trusting software, and call for critical and reflective use of automated machine learning tools.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.