Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations
United States · Framework · National Institute of Standards and Technology
High impact. A reviewer confirms each change before it is listed here.
Our summary
This NIST technical report establishes a common taxonomy and terminology for adversarial machine learning, covering attack types, stages, attacker capabilities, and mitigation strategies for both predictive and generative AI systems. It is designed to inform future standards and practice guides for assessing and managing AI system security.
Written by the Index in our own words. It is not legal advice; the official text is the authority.
Inside this policy
Every section of this policy in order. Our notes summarise each one in plain language and do not quote the official text, which is the authority.
Drafted by the Index An automated summary of the official text, not yet reviewed by a person.Reviewed Checked by a person against the official text.
8 sections
Jump to a chapter
1 Introduction
Adversarial machine learning studies attacks exploiting the probabilistic and data-driven characteristics of machine learning systems, which lack information-theoretic security proofs and are vulnerable to severe failures in high-stakes applications through training data manipulation, test-time adversarial modifications, and trojan insertion. NIST provides standardized terminology and taxonomy covering evasion, poisoning, and privacy attacks for predictive systems, and poisoning, direct prompting, and indirect prompt injection for generative systems. The report categorizes common attacks and mitigations for both predictive and generative AI systems across sections covering predictive AI attacks, generative AI attacks and mitigations, and field challenges and limitations of mitigation techniques.
2 Predictive AI Taxonomy
2 sections
2.1 Attack Classification
Adversarial attacks on AI systems are classified by learning method and stage (training or deployment), attacker objectives (availability, integrity, privacy), attacker capabilities (training data control, model control, testing data control, label limit, source code control, query access), and attacker knowledge level (white-box with full knowledge, gray-box with partial knowledge, black-box with minimal or no knowledge). Training-time poisoning attacks involve controlling training data or model parameters; deployment-time evasion and privacy attacks target trained models to manipulate predictions or extract information. Attacks cause availability breakdown disrupting service access, integrity violation forcing misperformance through evasion or poisoning, or privacy compromise leaking information about training data, weights, or architecture.
2.2 Evasion Attacks and Mitigations
Evasion attacks generate adversarial examples through white-box optimization methods (L-BFGS, Fast Gradient Sign Method, DeepFool, Carlini-Wagner, Projected Gradient Descent) or black-box techniques (zeroth-order, decision-based, transferability from substitute models), with applications across image classification, audio, video, text, and cybersecurity, including documented real-world attacks such as over 80,000 attempts against facial recognition and phishing detector bypasses. Physical attacks operate in real-world conditions using eyeglass frames and stickers, while universal attacks generalize across architectures. Resilient mitigations include adversarial training with iteratively generated examples, randomized smoothing creating certifiable robustness against l2-norm bounded perturbations, and adaptive testing against stronger attacks to avoid gradient masking bypasses.
3 Formal verification: Another method for certifying the adversarial robustness of
1 section
Formal methods like SMT solvers and abstract interpretation can verify neural network robustness, with systems such as Reluplex, AI2, DeepPoly, ReluVal, and Fast Geometric Projections scaling these techniques to larger networks. These approaches face limitations in scalability, computational cost, and support for algebraic operations; all mitigation techniques create trade-offs between robustness and accuracy, and increase computational costs during training.
3.1 Attack Classification
The taxonomy organizes adversarial attacks on GenAI systems by targeted properties (availability, integrity, privacy, misuse enablement), required capabilities (query access, model control, resource control), and attack classes including data poisoning, model poisoning, jailbreaking, direct and indirect prompt injection, prompt extraction, and model extraction. Attacks occur across the system lifecycle: training-stage attacks include data poisoning from public sources and malicious third-party models, while deployment-stage attacks exploit the shared channel for data and instructions, enabling prompt injection, jailbreaking, prompt extraction, and indirect prompt injection in RAG applications. Model outputs used without human oversight can enable violations if attackers induce unaccounted-for behavior.
4 Agents: An LLM-based AGENT relies on iteratively processing the output of an LLM
LLM-based agents face four categories of attacks targeting availability, integrity, privacy, and misuse enablement, with attackers varying in capabilities (training data control, query access, resource control, model control) and knowledge (white-box to black-box). Attacks include data and model poisoning during training, direct prompt injection and optimization-based attacks through query access to override instructions or bypass defenses using gradient methods, prefix injection, refusal suppression, style injection, role-play, encoding and character transformation; automated attacks using attacker models generate transferable jailbreaks including Crescendo attacks applying iterative benign prompts. Mitigations address poisoning through cryptographic hash verification, data filtering, mechanistic interpretability, and treating models as untrusted, while information extraction attacks steal training data with extraction rates exceeding 8% for certain models, extract proprietary prompts and system designs, extract context from retrieval-augmented generation databases, and derive architecture and parameter information from black-box systems.
6 Monitoring & Reporting
Defense strategies span the model lifecycle through safety training, adversarial training, vulnerability assessments including red teaming and bug bounties, and continuous evaluation following deployment. Deployment-time mitigations for prompt injection include isolating user input, detecting harmful outputs, modifying inputs, aggregating outputs, and monitoring user activity; model providers can restrict inference parameters, limit query richness and quantity, sanitize training data, unlearn harmful capabilities, and watermark content. Indirect prompt injection attacks compromise availability through time-consuming tasks or disable capabilities, and compromise integrity by inducing false information or malicious redirects; defenses include fine-tuning task-specific models, training hierarchical trust relationships, and input filtering.
Controls
What this policy requires, forbids, grants or penalises, one control at a time and in our own words. Each control links to the official text, which is the authority. This is not legal advice.
Drafted by the Index Drafted automatically from the official text, not yet reviewed by a person.Reviewed Checked by a person against the official text.
Themes covered 0 of 35
Choose a theme to see only its controls. About the themes
Risk and scope 0 of 4
- Risk classification, not covered
- Prohibited practices, not covered
- Risk management system, not covered
- Impact assessment, not covered
Data and privacy 0 of 4
- Data governance and quality, not covered
- Data subject rights and consent, not covered
- Cross-border data transfer, not covered
- Copyright and training data, not covered
Transparency and records 0 of 4
- Transparency and disclosure to users, not covered
- Labelling of synthetic content, not covered
- Technical documentation, not covered
- Record keeping and logging, not covered
Technical safeguards 0 of 4
- Human oversight, not covered
- Accuracy, robustness and cybersecurity, not covered
- Bias and non-discrimination, not covered
- Testing and evaluation, not covered
Assurance and lifecycle 0 of 4
- Conformity assessment and certification, not covered
- Registration and notification, not covered
- Incident and breach reporting, not covered
- Post-market monitoring and corrective action, not covered
Rights and protected groups 0 of 4
- Automated decision-making rights, not covered
- Children and vulnerable groups, not covered
- Biometric identification and surveillance, not covered
- Whistleblowing, complaints and redress, not covered
Roles and organisation 0 of 6
- Supply chain and provider-deployer duties, not covered
- General purpose and foundation models, not covered
- Accountability and governance roles, not covered
- Training and AI literacy, not covered
- Public sector use and procurement, not covered
- Innovation support and sandboxes, not covered
Enforcement and sectors 0 of 5
- Enforcement, penalties and regulators, not covered
- Sector rules: health, not covered
- Sector rules: finance and insurance, not covered
- Sector rules: employment, not covered
- Sector rules: education, not covered
The full list of controls could not be loaded, so only the first 0 are shown.
Change history
No changes recorded since we began monitoring this policy.