Guidance on privacy and the use of commercially available AI products
Australia · Guideline · Office of the Australian Information Commissioner
Our summary
This guidance helps organisations understand how Australia's Privacy Act applies when they adopt commercially available AI systems. It addresses the risks that arise when personal information is fed into or produced by AI tools, including synthetic or inferred data, and recommends organisations conduct due diligence before deploying such systems. The guidance emphasises transparency, privacy by design principles, accuracy safeguards, and advises against inputting sensitive personal data into public AI chatbots.
Written by the Index in our own words. It is not legal advice; the official text is the authority.
Inside this policy
Every section of this policy in order. Our notes summarise each one in plain language and do not quote the official text, which is the authority.
Drafted by the Index An automated summary of the official text, not yet reviewed by a person.Reviewed Checked by a person against the official text.
31 sections
Jump to a chapter
- We use cookies on this site
- Top five takeaways
- Quick reference guide
- Overview
- Artificial intelligence and privacy
- Interaction with Voluntary AI Safety Standard
- What should organisations consider when selecting an AI product?
- What are the key privacy risks when using AI?
- What transparency and governance measures are needed?
- What ongoing assurance processes are needed?
- Checklists
We use cookies on this site
Top five takeaways
Privacy rules apply to both data entered into AI systems and information they produce if it relates to a person. Organisations must assess whether commercial AI products suit their needs, considering testing, human oversight, risks and access controls. Businesses should clearly disclose AI use in privacy policies and identify public-facing AI tools like chatbots.
Quick reference guide
1 section
This guidance helps organisations follow privacy laws when selecting and using commercially available AI products, including free tools like chatbots. Although it covers various AI systems processing personal information, it focuses especially on generative and general-purpose AI with high-risk potential. Many AI uses involve low privacy risk, but deploying personal information in AI is a significant community concern and high-risk activity, so organisations should adopt careful approaches aligned with potential risks.
Downloads
Overview
3 sections
Who is this guidance for?
The guidance targets organisations deploying AI systems that process personal information, whether built with such data or collecting, storing, using or disclosing it. Deployers include anyone supplying or using AI systems for products or services, either internally or affecting external parties. Common applications include chatbots, content-generation tools like text-to-image generators, and productivity assistants for writing and transcription.
How to use this guidance
The guidance highlights key privacy considerations and rules organisations must apply when selecting and deploying AI products, though it does not comprehensively cover all privacy risks or other regulatory frameworks. It emphasises generative and general-purpose AI tools as particularly useful topics and notes their specific risks from probabilistic operation and large training datasets. Organisations should adopt proportionate, risk-based approaches and can navigate the document by topic headings to find relevant sections, checklists and examples.
Introductory terms
AI broadly refers to machines performing tasks normally requiring human intelligence, encompassing both established and newly advanced systems. The document defines technical concepts: AI models are mathematical foundations like GPT-4, while AI systems are complete applications like ChatGPT. General-purpose AI can serve multiple purposes including integration into other systems, while narrow AI focuses on specific tasks; generative AI creates content resembling training data, including large language models for text and multimodal foundation models for multiple data types.
Artificial intelligence and privacy
2 sections
AI technologies provide economic and societal benefits but create privacy risks due to their reliance on large datasets containing personal information. Organisations using commercially available AI must consider whether it involves personal data and, if so, embed privacy protections into system selection and use.
How does the Privacy Act apply?
The Privacy Act and Australian Privacy Principles apply to all AI use involving personal information, including training and testing. Personal information covers any data that could identify or be reasonably linked to an individual, including false information generated by AI systems, and is protected regardless of accuracy.
Privacy risks and harms
AI systems carry significant privacy risks including bias and discrimination in outputs based on training data, lack of transparency in personal information use and decision-making, increased data breach risks from large amounts of stored data, individuals losing control over personal information collected without knowledge or consent, and generative AI-specific risks such as misuse for disinformation, identity theft, and inaccurate but credible outputs; they may also inadvertently output actual personal information from training datasets even when instructed to create fictional content, particularly with limited relevant data. These risks are especially severe for children, vulnerable populations, First Nations people, and those underrepresented in training datasets, with discrimination resulting from algorithmic bias reflecting historical inequalities.
Interaction with Voluntary AI Safety Standard
The National AI Centre developed ten voluntary guardrails for safe and reliable AI deployment without creating new legal obligations. This guidance on Privacy Act compliance assists organisations in following existing Australian laws while addressing these guardrails.
What should organisations consider when selecting an AI product?
3 sections
Organisations must conduct due diligence to understand how an AI product works and its privacy risks before adoption. Privacy by design should be embedded from the start through risk management processes, and privacy impact assessments help identify and mitigate potential harms to individuals.
Is the product appropriate for its intended uses?
Organisations should identify their intended uses of AI and evaluate whether the product is appropriate for those uses, particularly in high privacy risk activities like decision-making with legal effects on individuals. Organisations must understand the training data sources to assess diversity, relevance and reliability, test products to identify risks of inaccuracy or bias, and understand system limitations. While developers may implement safeguards such as preventing personal information in outputs, organisations must implement their own processes to ensure compliance with accuracy obligations under APP 10.
What are the potential security risks?
Organisations must evaluate whether AI system use complies with obligations to protect personal information from unauthorised access, and assess product safety including known security incidents and owner safeguards. Consideration of deployment environment, such as local versus cloud hosting and server location, is necessary to manage disclosure risks outside Australia.
Who will have access to data in the AI system?
Organisations should review terms and settings to understand whether the service provider can access their input or generated data, as this may conflict with Australian Privacy Principle 6 restrictions on disclosing personal information for secondary purposes. Where product features allow data collection for further training, ongoing disclosure to developers, or interface links to third parties that would expose personal information, organisations must consider compliance measures such as disabling such features, limiting staff access to personal information entry, providing required disclosure notices, or implementing robust training and auditing.
What are the key privacy risks when using AI?
3 sections
Organisations should understand how they handle personal information when using AI and identify whether specific risks associated with their intended uses can be managed. The guidance advises considering whether using personal information with AI is necessary and appropriate, and addresses key privacy matters related to data input, output generation, collection and accuracy.
Can we input personal information into an AI system?
Sharing personal information with AI systems poses privacy risks because it becomes difficult to track, control or remove such information once entered. Organisations should proceed cautiously and only input personal information if they can comply with privacy law requirements around disclosing information for secondary purposes.
What are the requirements of APP 6?
When will an individual reasonably expect a secondary use of their information for an AI-related purpose?
What privacy obligations apply if we use AI to collect or generate personal information?
Can we collect personal information through an AI chatbot?
Can we use an AI product to generate or infer personal information?
What obligations do we have to ensure the accuracy of AI systems?
Privacy Act-regulated organisations must comply with accuracy obligations requiring them to take reasonable steps to ensure collected personal information is accurate and up-to-date, and that information used or disclosed is accurate, up-to-date, complete and relevant for its purpose. AI systems built on biased or incomplete information risk perpetuating and amplifying those biases.
What are the specific accuracy risks of generative AI?
What are the requirements of APP 10?
What transparency and governance measures are needed?
1 section
Entities covered by the Privacy Act must adopt reasonable practices and systems to comply with privacy law, maintain current privacy policies, notify individuals about personal information collection, and transparently communicate how AI systems process personal data. Opacity in AI technologies, particularly advanced models, can obstruct organizations' capacity to clarify their data processing methods to individuals, yet organizations should nonetheless implement measures ensuring open disclosure of AI system usage to meet legal requirements, strengthen public accountability and foster confidence.
What practical measures can we take to promote transparency?
Organisations should establish policies to inform individuals about personal information use in connection with AI systems, include AI-related collection purposes and disclosures in Australian Privacy Principle 5 notices, establish procedures for explaining AI outputs, and train staff on how AI products handle personal information. When using AI to assist in decisions with legal or significant effects on individuals' rights, organisations must understand how the system produces outputs, provide meaningful explanations to affected individuals, and ensure human verification of accuracy and ability to overturn decisions. Organisations should also establish procedures and resources to understand and explain how their AI systems handle, share and produce personal information, which supports compliance with APP 1 and APP 5, strengthens organisational accountability and increases customer confidence.
What ongoing assurance processes are needed?
Throughout an AI product's lifecycle, organisations should implement internal policies defining permitted and prohibited uses, establish human oversight and verification of outputs, conduct regular audits of product performance and usage, and train staff on product functions and limitations. This ensures the product remains reliable and appropriate for its intended purposes.
Checklists
1 section
A checklist for selecting an AI product addresses whether the system is appropriate and reliable for intended uses, potential security risks, accuracy and training data considerations, and compliance with privacy law obligations regarding safeguards for high-risk activities.
Downloads
Controls
What this policy requires, forbids, grants or penalises, one control at a time and in our own words. Each control links to the official text, which is the authority. This is not legal advice.
Drafted by the Index Drafted automatically from the official text, not yet reviewed by a person.Reviewed Checked by a person against the official text.
Themes covered 16 of 35
Choose a theme to see only its controls. About the themes
Risk and scope 3 of 4
- Risk classification, not covered
Data and privacy 3 of 4
- Copyright and training data, not covered
Transparency and records 2 of 4
- Labelling of synthetic content, not covered
- Technical documentation, not covered
Technical safeguards 4 of 4
Assurance and lifecycle 1 of 4
- Conformity assessment and certification, not covered
- Registration and notification, not covered
- Incident and breach reporting, not covered
Rights and protected groups 1 of 4
- Children and vulnerable groups, not covered
- Biometric identification and surveillance, not covered
- Whistleblowing, complaints and redress, not covered
Roles and organisation 2 of 6
- Supply chain and provider-deployer duties, not covered
- General purpose and foundation models, not covered
- Public sector use and procurement, not covered
- Innovation support and sandboxes, not covered
Enforcement and sectors 0 of 5
- Enforcement, penalties and regulators, not covered
- Sector rules: health, not covered
- Sector rules: finance and insurance, not covered
- Sector rules: employment, not covered
- Sector rules: education, not covered
The full list of controls could not be loaded, so only the first 39 are shown.
Top five takeaways 4 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations must assess whether commercial AI products match their needs, including testing, human oversight, privacy and security risks, and access controls.
- ObligationApplies to: DeployerDrafted by the Index
Deployers should inform users through privacy policies and notices that they use AI, and clearly mark public-facing AI tools like chatbots.
- ObligationApplies to: DeployerDrafted by the Index
Privacy law requires deployers to comply with data protection principles when using AI systems to create or infer personal information about identified or reasonably identifiable people.
- ObligationApplies to: DeployerDrafted by the Index
Deployers must only use personal information for the primary purpose it was collected for, or for a secondary purpose if the person consented or would reasonably expect it and it is related to the original purpose.
How does the Privacy Act apply? 1 control
- ObligationApplies to: DeployerDrafted by the Index
Organisations covered by the Privacy Act must understand their obligations under Australian Privacy Principles when using AI systems, including how AI may collect, use or disclose personal information.
Is the product appropriate for its intended uses? 5 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations should identify how they intend to use AI and evaluate whether the selected product is appropriate for those uses, including whether it has been tested for them.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should examine whether intended AI uses constitute high privacy risk activities, particularly where the system will make decisions with legal or significant effects on individuals' rights.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should understand the training data sources of the AI system to assess whether data is sufficiently diverse, relevant and reliable for the intended use and to identify bias and discrimination risks.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should carefully test AI products before deployment to understand risks of inaccurate or biased answers, particularly customer-facing products like chatbots.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should understand the limitations of the AI system and the protections implemented by the developer, while recognizing that mitigations are not foolproof.
What are the potential security risks? 4 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations should evaluate whether AI system use complies with obligations to protect personal information from unauthorised access, misuse, interference, loss and disclosure under APP 11.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should actively ensure that using an AI system will not expose personal information holdings to additional security risks.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should assess the general safety of the AI product, including searching for past security incidents and evaluating security measures implemented by the product owner.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should consider the deployment environment for the AI product, including whether it operates on premises or in the cloud, and assess risks of personal information disclosure outside Australia.
Who will have access to data in the AI system? 3 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations should carefully review service terms and settings to understand whether AI developers or third parties can access data they input into or generate from commercial AI products.
- ObligationApplies to: DeployerDrafted by the Index
If AI system operation allows disclosure of personal information to developers or third parties, deployers must assess compliance with data protection laws restricting secondary use and implement protective measures such as disabling risky features, providing notice, or prohibiting staff from entering personal information.
- ObligationApplies to: DeployerDrafted by the Index
Organisations must conduct due diligence when selecting commercial AI products to identify potential privacy risks and understand how the product will be used and its privacy impact.
What are the requirements of APP 6? 2 controls
- ObligationApplies to: DeployerDrafted by the Index
When personal information stays within an organisation's control in an AI system, it is a use of information; when made accessible outside the organisation, it is a disclosure, and both must comply with data protection law.
- ObligationApplies to: DeployerDrafted by the Index
Deployers must identify the purposes for which they will use personal information with AI systems and determine whether secondary purposes are authorised by law, such as when the person consented or would reasonably expect the use.
When will an individual reasonably expect a secondary use of their information for an AI-related purpose? 2 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations should seek consent for secondary uses of personal information for AI-related purposes when they cannot clearly establish that such use was within the individual's reasonable expectations at the time of collection.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should use or disclose only the minimum amount of personal information necessary for secondary AI-related purposes and should consider privacy-preserving techniques to minimise personal information.
Can we collect personal information through an AI chatbot? 2 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations that collect personal information through AI chatbots must comply with the requirements that collection be reasonably necessary for their functions and carried out by lawful and fair means.
- ObligationApplies to: DeployerDrafted by the Index
Organisations collecting personal information through AI chatbots should take reasonable steps to notify individuals about matters related to the collection of their personal information.
Can we use an AI product to generate or infer personal information? 2 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations using AI to infer or create personal information must ensure that the generated information is reasonably necessary for their functions and is created only by lawful and fair means.
- ObligationApplies to: DeployerDrafted by the Index
Organisations must destroy or de-identify personal information created through AI if they are not permitted to collect that information under privacy requirements.
What are the requirements of APP 10? 4 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations should take reasonable steps to ensure that personal information they collect is accurate, current and complete.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should take reasonable steps to ensure that personal information they use and disclose is accurate, current, complete and relevant for its purpose.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should design their processes to treat AI outputs as probabilistic assessments rather than facts, and should clearly record that outputs are probabilistic and identify the source data and system used.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should carefully consider whether proposed uses of AI can be conducted in a way that complies with privacy obligations regarding accuracy.
What practical measures can we take to promote transparency? 6 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations should establish policies and procedures to inform individuals about how their data is used in connection with AI through privacy statements.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should ensure collection notices specify any AI-related purposes for processing personal information, creation of information through AI systems, and disclosures to AI system developers.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should establish procedures to explain AI-related decisions and outputs to individuals affected by them.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should train staff to understand how AI products collect, generate, use or disclose personal information and to give meaningful explanations of AI outputs.
- ObligationApplies to: DeployerDrafted by the Index
For high-risk AI uses such as decisions with legal or similarly significant effects, organisations must understand how the product produces its outputs so they can verify its accuracy.
- ObligationApplies to: DeployerDrafted by the Index
For high-risk AI uses such as decisions with legal or similarly significant effects, organisations should be able to give meaningful explanations to affected individuals.
What ongoing assurance processes are needed? 4 controls
- ObligationApplies to: DeployerDrafted by the Index
Organisations should establish internal policies that clearly specify which uses of AI products are permitted and which are prohibited.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should set up procedures through which people monitor and verify AI outputs, especially where the outputs contain personal information or shape decisions about individuals.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should conduct regular audits or monitoring of how the AI product performs and how staff members use it.
- ObligationApplies to: DeployerDrafted by the Index
Organisations should train staff on the specific AI product, including its functions, limitations, risks and permitted and prohibited uses.
Change history
No changes recorded since we began monitoring this policy.