⚡ Quick Summary
This pilot maps what today’s AI governance actually covers—across risks, mitigations, sectors, lifecycle stages—by auto-coding 950+ entries from CSET’s AGORA archive with Claude 4.5 and validating against human reviewers. The team tested five LLMs (including GPT-5) using quadratic-weighted Cohen’s κ and found model-to-human agreement comparable to or above human-to-human on the small gold set, then scaled to the full corpus. Early signals: strong attention to governance failure, security vulnerabilities, and robustness; weak attention to AI welfare/rights and multi-agent risks; sector focus on public administration, national security, and R&D. Clear limitations (US/English skew, over-scoring tendencies, taxonomy ambiguities) are openly discussed, and an open CC-BY dataset/visualization is planned. (See executive summary and contents, pp. 2–3; methods, pp. 4–6; preliminary results, pp. 7–9; limitations, pp. 9–11; expected outputs, p. 12.)
🧩 What’s Covered
- Scope & Motivation. The team seeks a reproducible pipeline that maps governance texts to the MIT AI Risk and Mitigation Taxonomies, clarifying what is addressed vs. neglected across a fast-growing rule landscape (Research Motivation, p. 4).
- Methodology. Six pilot documents from varied stakeholders were hand-coded by six researchers across three layers: (1) risk subdomains, (2) mitigations, (3) additional governance dimensions (actors, lifecycle, status, sectors). Coverage scores use a 1–5 scale; models generate labels, quotes, and confidence, then are compared to human consensus with Cohen’s κ (pp. 4–6).
- Model Evaluation. Five LLMs—Claude Sonnet 4, Claude Opus 4.1, Claude Sonnet 4.5, GPT-5, Gemini 2.5 Pro—were benchmarked on 24 risk subdomains × 6 docs (144 points/model). All achieved “substantial” mean κ; Opus 4.1 and GPT-5 exceeded human-to-human mean κ. Prompt refinements reduced over-scoring (pp. 5–6). Table 1 summarizes κ ranges (mean/min/max).
- Scaled Coding of AGORA. Claude 4.5 was chosen for full-corpus classification (performance/cost). Figure 1 (p. 7) shows risk coverage distribution; Figure 2 (p. 8) shows sectoral coverage. Strongest coverage: governance failure, AI system security vulnerabilities & attacks, lack of capability/robustness. Weakest: AI welfare & rights, multi-agent risks (pp. 7–9).
- Limitations & Biases. Dataset skew (US/English), document timeliness, LLM misreadings of nuanced subdomains (e.g., “governance failure”), granularity of 5-point scale, confidence-biased over-scoring, and κ’s blind spots to systematic mislabels are candidly detailed (pp. 9–11).
- Planned Outputs. Open CC-BY visualizations, preprint, and a queryable database covering risk/mitigation scores, legislative status, lifecycle, actors/roles, sectors, and AGORA metadata (p. 12).
💡 Why it matters?
Governance teams often ask “what’s covered, where, and by whom?” This project moves beyond lists of principles into measurable coverage across risks and mitigations, with transparent evidence and inter-rater metrics. That enables: (1) gap-driven prioritization (e.g., welfare/rights, multi-agent risks); (2) sector targeting (beyond public sector & defense); (3) standards alignment (MIT risk taxonomy, NAICS sectors, OECD/NIST lifecycle); and (4) repeatable monitoring as new laws/standards land. In short, it’s a baseline to coordinate policymakers, auditors, and builders on the actual state of AI governance, not just aspirations. (See Exec Summary, Methods, Preliminary Results, and Expected Outputs.)
❓ What’s Missing
- Global balance. Heavy US/English weighting risks skewing coverage; non-English originals and evolving translations remain a hurdle (p. 9).
- Granularity & semantics. A 5-point scale may over-promise precision; ambiguous subdomain boundaries (e.g., governance failure vs. governance processes) invite systematic mislabels (pp. 9–10).
- Temporal validity. Some AGORA entries may be outdated or superseded; alignment to “in-force” status and versioning is key (p. 9).
- Frontier-risk lenses. Reviewers suggest clearer slices for catastrophic risks, misuse pathways (bio/info hazards), and frontier-AI thresholds (p. 11).
👥 Best For
- Policy units & regulators planning updates and wanting evidence-backed gaps by risk/sector.
- Standards bodies & auditors mapping controls to observed coverage.
- Corporate AI governance/compliance benchmarking internal policies vs. external norms.
- Researchers & civil society analyzing trends, overlaps, and blind spots across jurisdictions. (See “Potential audiences”, p. 12.)
📄 Source Details
- Title: Mapping the AI Governance Landscape: Pilot test and update
- Authors: Simon Mylius, Peter Slattery, Yan Zhu, Mina Narayanan, Adrian ThinnYun, Alexander Saeri, Jess Graham, Michael Noetel, Neil Thompson
- Orgs: MIT AI Risk Initiative, FutureTech, MIT, CSET
- Date: October 2025
- License: CC BY 4.0
- Core exhibits: Figure 1 (risk subdomain coverage, p. 7); Figure 2 (sector coverage, p. 8); κ comparison table (p. 6).
📝 Thanks to
Kudos to the author team and contributors acknowledged in the report (e.g., Graham Ryan, Himanshu Joshi, Emre Yavuz, Sophia Lloyd George, Echo Huang, Clelia Lacarriere, Lenz Dagohoy, Henry Papadatos, Aidan Homewood). (Acknowledgements, p. 13.)
Note: The chart on p. 7 visualizes strongest attention to governance failure, security vulnerabilities/attacks, and robustness; p. 8 highlights concentration in public administration, national security, and scientific R&D; and pp. 9–11 detail dataset/model limitations and κ caveats.
