The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms

⚡ Quick Summary

This paper introduces the AI Model Risk Catalog, the largest dataset of developer-identified AI risks to date, derived from over 64,000 Hugging Face model cards. The authors compare these developer-reported risks to two other key sources: the MIT Risk Repository (capturing researcher-envisioned risks) and the AI Incident Database (cataloging real-world harms). The study reveals that while developers focus on technical and bias-related risks, both developers and researchers underreport harms stemming from human interaction—particularly fraud, misinformation, and malicious use—despite their prevalence in real incidents. This gap underscores a need for structured, contextualized risk reporting that accounts for real-world usage dynamics.

🧩 What’s Covered

The study builds a new AI Model Risk Catalog by:

1. Extracting Risks: Analyzing 64,116 model cards from Hugging Face (2024 snapshot) with risk-related sections, and isolating 2,863 unique risk mentions after de-duplication and validation.
2. Taxonomizing Risks: Using both the DeepMind taxonomy (Weidinger et al.) and the MIT AI Risk Repository taxonomy to categorize risks. Most fall under “Discrimination & Toxicity” (44%) or “AI System Safety, Failures & Limitations” (37%) .
3. Comparative Analysis: Contrasting the catalog with:This reveals divergences in focus:
   • The MIT Risk Repository (967 research-defined risks)
   • The AI Incident Database (869 real-world cases)
• Developers focus on technical faults and bias.
• Researchers prioritize systemic and ethical concerns.
• Real-world harms stem mostly from misinformation and malicious use—areas underrepresented in both expert groups .
4. Risk Reporting Trends (2022 → 2024):
   • Risk reporting quality has declined—fewer unique entries and shorter descriptions.
   • Developers increasingly reuse generic risk templates, especially for fine-tuned LLMs.
   • The share of risk categories has shifted: bias risks increased, while misinformation risks dropped by 3x .
5. Modality-Specific Risks (Page 15):
   • Non-text modalities (e.g., image, audio, video) are increasingly associated with risks like privacy, security, and malicious use.
   • Multimodal models saw a 3x increase and are more likely to raise concerns about fraud and identity manipulation.
6. Recommendations:
   • Urge developers to adopt structured and contextualized risk descriptions.
   • Encourage researchers to focus more on social engineering and human interaction risks.
   • Propose integration with auditing tools like RiskRAG, ExploreGen, and BenchmarkCards.

💡 Why it matters?

The AI safety ecosystem remains siloed: developers, researchers, and incident reporters speak different “languages” of risk. This paper’s comparative framing exposes a critical gap—real-world AI harms are dominated by misuse and manipulation, yet developers and researchers focus on system-centric or ethical concerns. Without bridging this gap, risk governance frameworks (like the EU AI Act or NIST RMF) may remain blind to the most urgent and prevalent dangers. The AI Model Risk Catalog provides an urgently needed resource to ground risk analysis in actual model behavior and real-world impact, offering a template for more actionable, interdisciplinary AI assurance practices.

❓ What’s Missing

• User and community perspectives are absent—no input from civil society, NGOs, or end-users.
• Risk severity and likelihood are not quantified—hindering prioritization.
• Linkage to regulatory frameworks like ISO 42001 or the EU AI Act is not directly explored.
• The catalog excludes proprietary or enterprise models, which may present higher-stakes risks.
• The study doesn’t test how well cataloged risks help in predicting or mitigating future incidents.

👥 Best For

• AI developers seeking to improve model documentation and risk foresight
• Researchers exploring the divergence between theoretical risk and real-world impact
• Auditors and regulators aiming to align assurance practices with actual harms
• Journalists and analysts investigating emerging gaps in AI governance
• Toolmakers designing risk assessment and LLM red-teaming platforms

📄 Source Details

• Title: The AI Model Risk Catalog: What Developers and Researchers Miss About Real-World AI Harms
• Authors: Pooja S. B. Rao, Sanja Šćepanović, Dinesh Babu Jayagopi, Mauro Cherubini, Daniele Quercia
• Institutions: University of Lausanne, Nokia Bell Labs, IIIT Bangalore, University of Oxford, Politecnico di Torino
• Date: August 21, 2025
• Availability: https://social-dynamics.net/ai-risks/catalog
• Preprint: arXiv:2508.16672v1

📝 Thanks to

Pooja S. B. Rao and the team at Social Dynamics Lab for an indispensable tool in evidence-based AI risk reporting.