Four Principles of Explainable Artificial Intelligence
United States · Guideline · National Institute of Standards and Technology
Our summary
This draft publication presents four foundational principles for AI systems that provide explanations alongside their outputs: systems must deliver supporting evidence, ensure explanations are understandable to users, maintain accuracy between explanations and actual decision processes, and operate only within designed parameters. The work synthesizes research across computer science, engineering, and psychology to establish guidelines for trustworthy and explainable AI.
Written by the Index in our own words. It is not legal advice; the official text is the authority.
Inside this policy
Every section of this policy in order. Our notes summarise each one in plain language and do not quote the official text, which is the authority.
Drafted by the Index An automated summary of the official text, not yet reviewed by a person.Reviewed Checked by a person against the official text.
6 sections
Jump to a chapter
Cover page and publication details
Patent holders must disclose essential patent claims related to this guidance and either disclaim holding such claims or commit to licensing them on reasonable, non-discriminatory terms, with or without compensation. The document introduces four foundational principles for explainable AI systems spanning computer science, engineering, and psychology, covering multiple types of explanations and algorithms to organize existing explainable AI work and guide future research rather than prescribe algorithmic methods. AI systems used in high-stakes decisions must be accompanied by explanations of their logic and rationale to meet legal and regulatory requirements and foster trust.
Explanation Accuracy and Knowledge Limits
Four principles of explainable artificial intelligence are: Explanation, requiring systems to provide evidence or reasoning for all outputs; Meaningful, requiring explanations to be understandable to individual users and tailored to their prior knowledge and experiences; Explanation Accuracy, requiring explanations to correctly reflect how the system generated its output; and Knowledge Limits, requiring systems to operate only under their design conditions or when confident in their output. Explanations serve five categories of purposes: user benefit, societal acceptance, regulatory and compliance, system development, and owner benefit, and vary in complexity and required expertise depending on their intended consumers. Best practices include designing explanations for understanding, ensuring high interpretability for high-stakes decisions, and using explanations to both improve and evaluate systems.
Overview of Explainable AI Algorithms
Explainable AI approaches distinguish between existence, meaningfulness, and accuracy of explanations, with trade-offs between explanation meaningfulness and accuracy depending on application context and user requirements. Technical approaches include transparent explanations revealing inner workings, post-hoc interpretability providing useful information, and counterfactual explanations describing minimal input changes; researchers have developed self-explainable models, global algorithms approximating non-interpretable models, and per-decision algorithms explaining individual decisions. Developers measure explanation meaningfulness through human simulatability, with explanation accuracy being particularly important for self-explainable models though research measuring it remains limited.
Self-Explainable Models
Self-explainable models such as decision trees, linear regression, and logistic regression provide inherent interpretability, with variants including decision sets, optimal classification trees, additive trees, Bayesian rule lists, and generalized additive models developed to improve accuracy while maintaining interpretability, and techniques like lasso regularization reducing feature complexity. Global explainable AI algorithms treat systems as black boxes and produce explanatory models, including SHAP using game theory to compute feature importance, TCAV representing neural networks as human-friendly concepts, and visualization methods like partial dependence plots and individual conditional expectation curves. Per-decision algorithms explain individual system decisions without generalization, with LIME building local interpretable models around specific decisions and counterfactual explanations identifying minimal input changes needed to alter decisions.
Adversarial attacks and saliency explanations
Local explanation techniques including saliency pixel algorithms can be exploited by adversaries to hide biases or manipulate outputs when explanation accuracy is imperfect. Evaluating explainable AI systems requires benchmarking against human explanation capabilities, though requiring explicit explanations can interfere with automatic expert reasoning. Explanations must be meaningful and intelligible to their specific audience, but humans frequently provide inaccurate introspections about their own decision processes and confabulate reasons for choices they cannot recall making, and forensic science explanations often mislead lay audiences.
Knowledge limits principle
Explainable artificial intelligence systems must produce explanations that are meaningful to humans, reflect the system's true processes accurately, and express the system's knowledge limits, though practical needs may require trade-offs such as prioritizing brief, meaningful explanations over detailed, accurate ones in time-sensitive scenarios. Humans have restricted competence in meeting these criteria, serving as a performance benchmark for AI systems; recent regulations require explainable AI to establish accountability, and combining human and AI strengths in collaborative settings may yield superior explanations. Meaningful explanations enable users to modify behavior or contest decisions and help developers maintain and deploy systems appropriately, so advancing explainable AI requires studying human factors affecting decisions, including preferences for simpler explanations and susceptibility to context and bias.
Controls
What this policy requires, forbids, grants or penalises, one control at a time and in our own words. Each control links to the official text, which is the authority. This is not legal advice.
Drafted by the Index Drafted automatically from the official text, not yet reviewed by a person.Reviewed Checked by a person against the official text.
Themes covered 5 of 35
Choose a theme to see only its controls. About the themes
Risk and scope 1 of 4
- Risk classification, not covered
- Prohibited practices, not covered
- Impact assessment, not covered
Data and privacy 0 of 4
- Data governance and quality, not covered
- Data subject rights and consent, not covered
- Cross-border data transfer, not covered
- Copyright and training data, not covered
Transparency and records 1 of 4
- Labelling of synthetic content, not covered
- Technical documentation, not covered
- Record keeping and logging, not covered
Technical safeguards 2 of 4
- Human oversight, not covered
- Testing and evaluation, not covered
Assurance and lifecycle 0 of 4
- Conformity assessment and certification, not covered
- Registration and notification, not covered
- Incident and breach reporting, not covered
- Post-market monitoring and corrective action, not covered
Rights and protected groups 0 of 4
- Automated decision-making rights, not covered
- Children and vulnerable groups, not covered
- Biometric identification and surveillance, not covered
- Whistleblowing, complaints and redress, not covered
Roles and organisation 1 of 6
- Supply chain and provider-deployer duties, not covered
- General purpose and foundation models, not covered
- Accountability and governance roles, not covered
- Public sector use and procurement, not covered
- Innovation support and sandboxes, not covered
Enforcement and sectors 0 of 5
- Enforcement, penalties and regulators, not covered
- Sector rules: health, not covered
- Sector rules: finance and insurance, not covered
- Sector rules: employment, not covered
- Sector rules: education, not covered
The full list of controls could not be loaded, so only the first 9 are shown.
Four principles of explainable AI 4 controls
- ObligationApplies to: ProviderDrafted by the Index
Providers should design explainable systems to provide evidence or reasoning that supports their outputs.
Four principles of explainable AIOfficial text of Four principles of explainable AI (opens in a new tab) - ObligationApplies to: ProviderDrafted by the Index
Providers should ensure that explanations are understandable to the intended users of the system.
Four principles of explainable AIOfficial text of Four principles of explainable AI (opens in a new tab) - ObligationApplies to: ProviderDrafted by the Index
Providers should ensure that explanations accurately reflect how the system generated its outputs.
Four principles of explainable AIOfficial text of Four principles of explainable AI (opens in a new tab) - ObligationApplies to: ProviderDrafted by the Index
Providers should design systems to operate only within their intended scope or when they are confident in their results.
Four principles of explainable AIOfficial text of Four principles of explainable AI (opens in a new tab)
Explanation requirements and context 2 controls
- ObligationApplies to: ProviderDrafted by the Index
Providers should tailor explanations to match the response time urgency and detail level required by different use cases.
Explanation requirements and contextOfficial text of Explanation requirements and context (opens in a new tab) - ObligationApplies to: ProviderDrafted by the Index
Providers should make explanations flexible enough to address diverse tasks, users, and requirements.
Explanation requirements and contextOfficial text of Explanation requirements and context (opens in a new tab)
Self-explainable models and explanation accuracy 2 controls
- Applies to: ProviderDrafted by the Index
Providers may use self-explainable models such as decision trees or linear regression that generate their own explanations rather than relying on post-hoc methods.
Self-explainable models and explanation accuracyOfficial text of Self-explainable models and explanation accuracy (opens in a new tab) - ObligationApplies to: ProviderDrafted by the Index
Providers should design systems that output confidence values reflecting the certainty of their predictions.
Self-explainable models and explanation accuracyOfficial text of Self-explainable models and explanation accuracy (opens in a new tab)
Explanation Accuracy and Adversarial Risks 1 control
- ObligationApplies to: ProviderDrafted by the Index
Providers should protect against adversaries exploiting incomplete explanation accuracy to hide system biases or obscure the true decision-making process.
Explanation Accuracy and Adversarial RisksOfficial text of Explanation Accuracy and Adversarial Risks (opens in a new tab)
Change history
No changes recorded since we began monitoring this policy.