AI incident
What Is an AI Incident?
An AI incident is a real-world event where an AI system causes, contributes to, or nearly causes harm, failure, or another unwanted outcome. It can affect safety, money, privacy, rights, services, or trust.
AI incidents often involve unsuitable use, incomplete data, weak approval, unclear instructions, excessive permissions, or unchecked outputs. A credible near miss matters when the same conditions could have caused harm.
The question is not only whether a model was wrong, but whether its output or action had, or could have had, meaningful consequences.
What Counts as Harm in an AI Incident?
Harm can be direct, such as an unsafe action, or indirect, such as denial of a legitimate service. Its seriousness depends on context.
| Harm category | Plain-language example | Likely affected parties |
|---|---|---|
| Physical safety | An AI-assisted system gives unsafe guidance that is followed without review. | Patients, workers, customers, bystanders |
| Financial loss | A fraud model blocks legitimate payments or approves suspicious ones. | Customers, banks, merchants |
| Discrimination and bias | A hiring screen ranks qualified applicants lower because training data reflected unequal treatment. | Applicants, employees, employers |
| Privacy | A chatbot exposes personal information from a document it should not access. | Individuals, clients, staff |
| Security | An AI tool follows a malicious instruction and reveals confidential system data. | Organizations, users, partners |
| Misinformation | A generated summary invents a policy requirement and is shared as fact. | Customers, voters, employees, the public |
| Legal or reputational harm | An automated moderation decision wrongly labels a lawful post as harmful. | Users, publishers, platform operators |
| Service disruption | An automated routing tool overloads one service and causes an outage. | Customers, operations teams, suppliers |
AI Incident, Near Miss, Model Error, and Data Breach
These terms overlap but are not interchangeable. Clear labels help teams choose the right response.
| Term | Meaning | Example |
|---|---|---|
| AI incident | An AI-related event that causes harm or a material unwanted outcome. | An automated eligibility decision wrongly denies an essential benefit. |
| Near miss | A harmful outcome was narrowly avoided. | A reviewer catches an unsafe recommendation before it is sent. |
| Model error | An incorrect output with no meaningful consequence. | A chatbot gives wrong opening hours, and the user notices immediately. |
| Conventional software outage | A service failure not caused by AI behavior. | A database fault takes an application offline. |
| AI-related data breach | Unauthorized exposure or access involving an AI system, its data, or connections. | A retrieval tool returns another customer’s private record. |
| Malicious misuse | A person deliberately uses AI to deceive, disrupt, or attack. | A scammer creates a convincing impersonation message. |
One event can fit several categories. A prompt injection attack may be malicious misuse, a security incident, and a privacy breach.
How an AI Incident Happens
AI incidents usually involve technical and human factors. Mapping the chain supports prevention and investigation.
- A team defines a use case and assumptions about users, data, and acceptable risk.
- Data is selected or connected, with possible gaps, bias, stale information, or access-control problems.
- The model receives instructions, guardrails, and sometimes tools that can search, send messages, alter records, or trigger workflows.
- People may make unexpected requests or use the system beyond its intended purpose.
- An incorrect output, hostile input, integration failure, or weak review creates an unsafe decision or action.
- Monitoring, feedback, or a downstream failure reveals the problem.
- The organization contains the issue, investigates, and changes the system or process.
Common Types of AI Incidents
Failure mechanisms differ, so controls that work for one type may not address another.
| Type | Key failure mechanism | Realistic example |
|---|---|---|
| Predictive-system failure | Poor data or changing conditions make a prediction unreliable. | A demand forecast causes a store to understock essential goods. |
| Generative hallucination | The model produces plausible but false content. | A support assistant invents a refund policy. |
| Biased decision | Patterns in data or design create unfair treatment. | A screening tool disadvantages candidates from a particular group. |
| Privacy exposure | Data is retained, retrieved, or disclosed improperly. | An internal assistant reveals restricted project details. |
| Prompt injection or tool misuse | Untrusted content manipulates an AI system with connected tools. | An agent is tricked into sending sensitive files externally. |
| Adversarial attack | Inputs are crafted to evade or distort a system. | A manipulated image bypasses an inspection model. |
| Autonomous agent failure | Multiple actions amplify an incorrect plan or goal. | An agent repeatedly changes records after misreading a request. |
| Deepfake misuse | Synthetic media falsely represents a real person. | A fake executive voice message requests a payment. |
Examples of AI Incidents in Everyday Settings
Context changes severity. An unreviewed error in a high-impact decision is more serious than a harmless draft typo.
- A hiring tool unfairly ranks applicants using past employment patterns.
- A healthcare chatbot gives information a patient mistakes for clinical advice.
- A service assistant discloses order details without proper verification.
- A banking model freezes a legitimate account and blocks urgent funds.
- An education tool wrongly flags original work as AI-generated.
- A moderation system removes lawful speech while missing credible threats.
- A logistics optimizer uses outdated maps and routes drivers through unsuitable roads.
- A cybersecurity assistant recommends a step that interrupts a critical service.
How to Assess AI Incident Severity
Assess impact, scale, reversibility, and intent or exploitability. Consider the seriousness of harm, the number of people, records, transactions, or systems affected, whether damage can be corrected, and whether the event can be repeated easily.
Raw incident counts are not a good risk measure. One privacy exposure involving sensitive records may matter more than many minor errors. Record near misses because they reveal weak controls.
How to Respond to an AI Incident
A calm, repeatable response protects people and preserves facts for investigation.
- Protect people and stop unsafe automated actions, including integrations where needed.
- Preserve prompts, outputs, timestamps, access records, model versions, and relevant decisions.
- Contain the issue by limiting access, revoking credentials, or rolling back changes.
- Notify accountable technical, security, legal, operational, and leadership owners, plus affected parties when appropriate.
- Investigate the technical cause and workflow conditions that allowed harm.
- Remediate through configuration, data, permissions, review steps, training, or product changes.
- Test the fix before restoring normal operation.
- Document facts, decisions, impacts, and lessons in a structured incident report and improve the wider incident response process.
Benefits of AI Incident Reporting
Reporting works best when people can raise good-faith concerns without automatic blame. It turns failures into evidence for safer design.
- Earlier detection of recurring errors, unsafe prompts, and weak integrations.
- Better product improvements based on real conditions.
- Clearer accountability for vendors, owners, and approvals.
- Stronger governance evidence for audits, customers, and regulators.
- Better risk decisions across incidents.
- Learning that reduces repeated failures.
Practical Limits and Common Pitfalls
Programs fail when they collect reports without learning from them. Investigations and disclosures must also protect sensitive information.
- Underreporting caused by fear, unclear channels, or uncertainty.
- Unclear ownership across vendors, product teams, and business units.
- Poor logs that prevent reconstruction of what the system saw or did.
- Treating every inaccurate output as equally severe.
- Blaming the model while ignoring unsafe workflows or weak approvals.
- Relying on benchmarks instead of realistic edge-case testing.
- Over-sharing details that expose personal data, security weaknesses, or confidential information.
Preventing AI Incidents Before Deployment
Prevention cannot guarantee zero incidents. The goal is to make harm less likely, easier to detect, and easier to reverse.
- Set clear use boundaries and prohibited tasks.
- Test realistic inputs, edge cases, and misuse scenarios according to risk.
- Check whether data represent affected users and conditions.
- Require human approval for high-impact or irreversible actions.
- Apply least-privilege access.
- Monitor outputs, actions, failures, and complaints after deployment.
- Maintain rollback plans, manual alternatives, and shutdown authority.
- Let users challenge harmful or incorrect outcomes.
- Review suppliers’ security, data handling, updates, and notification terms.
AI Incident Management and Governance
AI incident management is the ongoing practice of preparing for, detecting, responding to, and learning from AI-related failures. Product, security, operations, legal, compliance, and leadership teams each have roles.
Useful governance records intended use, data sources, permissions, evaluations, approvals, incidents, and corrective actions. Reviews should identify changes, missed signals, and needed controls. Higher-impact systems need stronger oversight, but compliance paperwork alone does not make AI safe.
on Emergent today

