Pieces Technologies settles claim of misleading AI accuracy statements
Pieces Technologies agreed to a settlement with Texas Attorney General after allegedly making false claims about the accuracy of its generative AI products used in hospitals. The company reportedly used misleading metrics claiming error rates below 1 per 100,000, which an investigation found to be inaccurate and potentially deceptive.
What happened
Pieces Technologies deployed its generative AI products at several Texas hospitals to summarize patients' conditions and treatment. The company developed metrics claiming its healthcare AI products were 'highly accurate' with a 'severe hallucination rate' of '<1 per 100,000' and a 'critical hallucination rate' of '<.001%'.
An investigation by the Texas Attorney General found that these metrics were likely inaccurate and may have deceived hospitals about the accuracy and safety of Pieces' products.
Reported impact
At least four major Texas hospitals provided patients' healthcare data in real time to Pieces for its generative AI product to process. The deceptive claims about accuracy put the public interest at risk by potentially misleading hospital staff about the reliability of AI-generated summaries used in patient treatment decisions.
Index analysis
The products exhibited hallucination behavior, with actual error rates appearing to exceed the claimed metrics. Pieces developed accuracy metrics that did not reflect the true performance of its generative AI systems.
Governance controls
As part of the settlement, Pieces agreed to clearly and conspicuously disclose the definition of any metrics used to describe its generative AI products' output and the methods used to calculate such metrics. Pieces also agreed to prohibit false, misleading, or unsubstantiated representations regarding any feature or appropriate use of its products, and must document known or reasonably known harmful or potentially harmful uses or misuses for current and future customers.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.