ReDoS vulnerability in Hugging Face Transformers Nougat tokenizer
A regular expression denial of service vulnerability was discovered in Hugging Face's Transformers library Nougat tokenizer component. The vulnerability has been patched through a code fix that simplifies the problematic regular expression pattern to prevent excessive backtracking.
What happened
A regular expression denial of service (ReDoS) vulnerability was identified in the Nougat tokenizer component of Hugging Face's Transformers library, tracked as CVE-2024-12720.
The vulnerability existed in the post_process_single function of tokenization_nougat_fast.py, where a regular expression pattern could cause excessive backtracking when processing certain inputs.
Governance controls
The vulnerability was addressed in commit deac971c469bcbb182c2e52da0b82fb3bf54cccf, which modified the regular expression to limit backtracking by simplifying the pattern from r"^#+ (?: \.?(?:\d |[ixv ])+)* \s*(?:$|\n\s*)" to r"^#+ (?: [\d+\.]+ |[ixv \.]+)? \s*(?:$|\n\s*)".
Classification
Sources and evidence
Sources not yet published
This record was researched and written by the Index from original reporting. Its sources are still being verified by an editor, and they are listed here once that check is complete. Until then the record carries no reviewer.