⚡ Quick Summary
This issue brief from the Center for Security and Emerging Technology offers one of the most empirically grounded analyses of AI harm to date. Rather than speculating about hypothetical risks, it works backwards from real incidents recorded in the AI Incident Database to explain how harm actually materializes in practice. The core contribution is a clear typology of six “AI harm mechanisms,” split between intentional and unintentional harm, which together map the pathways from AI deployment to real-world damage. The report challenges common assumptions that more capable or general-purpose models are inherently more dangerous, showing instead that harm frequently arises from single-purpose systems, poor integration choices, weak oversight, and misaligned incentives. It positions incident analysis as a critical governance tool and argues that effective AI risk management must be sociotechnical, context-aware, and adaptive rather than model-centric.
🧩 What’s Covered
The report is structured around six mechanisms of AI harm, derived from detailed analysis of over 200 documented incidents. Three mechanisms involve intentional harm: harm by design, where systems are explicitly built to cause harm; AI misuse, where users repurpose otherwise benign systems for malicious ends; and attacks on AI systems, including integrity, confidentiality, and availability attacks such as prompt injection or model exploitation. These sections illustrate how developers, users, and attackers contribute differently to risk and why mitigation must be actor-specific.
The remaining three mechanisms address unintentional harm. AI failures cover errors, bias, degradation, or malfunction, with case studies spanning criminal justice, healthcare, facial recognition, and autonomous vehicles. Failures of human oversight examine how human–AI teams break down due to automation bias, anchoring, institutional pressure, or poor interface design, showing that “human-in-the-loop” is not automatically protective. Integration harm focuses on systems that function as intended but cause damage because they are deployed in the wrong context or without considering downstream effects, such as misinformation amplification, workforce destabilization, or diversion of public resources.
Methodologically, the report explains how these mechanisms were identified and validated, discusses limitations of incident data, and highlights the role of media bias and underreporting. It also connects the framework directly to current governance tools, including the EU AI Act’s risk modeling and impact assessment requirements, arguing that incident-derived mechanisms offer a practical foundation for compliance and policy design.
💡 Why it matters?
This report fundamentally reframes how AI risk should be understood and governed. It demonstrates that focusing governance primarily on model capability, scale, or compute is insufficient and often misdirected. Many of the most severe harms arise from ordinary systems deployed carelessly, without adequate oversight, redress mechanisms, or contextual analysis. By grounding risk assessment in real incidents, the framework helps organizations anticipate failure modes they would otherwise miss, particularly integration harm and human oversight failures. For policymakers, it supports more targeted, proportionate regulation. For organizations, it offers a concrete lens for risk modeling, incident response, and internal accountability. Most importantly, it reinforces that AI safety is not a purely technical property but an outcome of design, deployment, governance, and human behavior combined.
❓ What’s Missing
While the framework is strong analytically, the report stops short of offering a consolidated, operational checklist that organizations could directly implement across the AI lifecycle. Mitigation strategies are discussed per mechanism, but not synthesized into a unified governance playbook. The analysis also largely excludes diffuse, long-term, or systemic harms that do not surface as discrete incidents, such as environmental impact, labor deskilling, or societal dependency on AI systems. Finally, although the report acknowledges overlapping mechanisms, it does not deeply explore compound harms where multiple mechanisms interact simultaneously in complex deployments.
👥 Best For
This resource is especially valuable for AI governance leads, risk managers, compliance teams, and policymakers who need empirically grounded tools for AI risk assessment. It is also highly relevant for organizations preparing for EU AI Act compliance, incident reporting regimes, or internal AI audits. Researchers and practitioners working on sociotechnical risk, human–AI interaction, and impact assessments will find it a strong conceptual anchor.
📄 Source Details
Issue Brief published October 2025 by the Center for Security and Emerging Technology (CSET). Author: Mia Hoffmann, Research Fellow focusing on AI governance.
📝 Thanks to
Mia Hoffmann and the CSET research team for building one of the clearest, incident-driven frameworks currently available for understanding how AI systems actually cause harm in the real world.
