Malicious AI models bypass Hugging Face security scanning via novel Pickle compression technique
Researchers at Reversing Labs discovered two malicious machine learning models on Hugging Face that evaded the platform's Picklescan security tool by using 7z compression instead of the standard ZIP format for PyTorch files. The models exploited a novel "nullifAI" technique that prevents detection of dangerous Pickle serialization code during deserialization.
What happened
Researchers at Reversing Labs discovered two malicious machine learning models available on Hugging Face that were not flagged as unsafe by the platform's security scanning mechanisms.
The malicious models exploited a novel attack technique by abusing Pickle file serialization in PyTorch format compressed using 7z instead of the standard ZIP format, preventing detection by Picklescan.
The two models used broken Pickle files, suggesting they were proof-of-concept models for testing a novel attack method rather than actual functional malicious ones.
Reported impact
Pickle files pose inherent security risks because they allow Python code to execute during deserialization, and on open ML platforms like Hugging Face where collaboration is prioritized, developers often prioritize ease of use over security.
Index analysis
Reversing Labs researchers identified shortcomings in Hugging Face's Picklescan detection tool, which relies on a blacklist of dangerous functions—a basic security feature that is not scalable or adaptable as known threats morph and new threats emerge.
Picklescan demonstrated an inability to properly scan broken Pickle files and could not detect models compressed using non-standard formats outside its expected ZIP compression scheme.
Governance controls
Hugging Face operates Picklescan to detect malicious models, but the tool's blacklist-based approach has proven insufficient to identify sophisticated evasion techniques.
Classification
Sources and evidence
This record was researched and written by the Index, based on the following original reporting.