⚡ Quick Summary
This document is the supplementary material to an article titled Managing extreme AI risks amid rapid progress. It is credited to Yoshua Bengio and 24 co-authors, with Jan Brauner named as corresponding author; the cover prints the DOI 10.1126/science.adn0117 and no publisher or journal name. The supplement states that it supplies "a copy of the text with 73 additional citations" and adds a full references and notes list.
The reproduced article argues that AI capabilities are advancing rapidly while safety research lags, estimating that only 1 to 3% of AI publications are on safety. It sets out two agendas, technical R&D and governance. The technical agenda names oversight and honesty, robustness, interpretability and transparency, inclusive AI development, evaluation for dangerous capabilities, evaluating AI alignment, risk assessment and resilience, and calls on major tech companies and public funders to allocate at least one-third of their AI R&D budget to them.
The governance agenda proposes national institutions and international governance, government insight through registration of frontier systems and incident reporting, developer safety cases that carry the burden of proof, clarified liability, and commensurate mitigations such as licensing, autonomy restrictions, access controls and the ability to halt development.
🧩 What’s Covered
The supplement has two parts: a short explanatory note and a reproduced copy of the article with added citations.
- Purpose of the supplement: A note states that it provides "a copy of the text with 73 additional citations, for readers who want to investigate the mentioned topics in more detail", followed by the article text itself.
- Rapid progress, high stakes: The article cites training investment for state-of-the-art models tripling annually, AI chips becoming 1.4 times more cost-effective and training algorithms 2.5 times more efficient each year, alongside risks including large-scale social harms, malicious uses and an irreversible loss of human control over autonomous systems.
- Reorient technical R&D: Two sets of challenges: those needing breakthroughs for reliably safe AI (oversight and honesty, robustness, interpretability and transparency, inclusive AI development, and emerging failure modes) and those enabling risk-adjusted governance (evaluation for dangerous capabilities, evaluating AI alignment, risk assessment, resilience). It asks for at least one-third of AI R&D budgets.
- Governance measures: Policies that "automatically trigger when AI hits certain capability milestones", proactive risk identification, fast-acting oversight institutions, and mandatory, more rigorous risk assessments with the burden of proof on developers.
- Institutions and government insight: Regulators should mandate whistleblower protections, incident reporting, registration of key information on frontier AI systems and their datasets, monitoring of model development and supercomputer usage, and white-box auditor access from the start of model development.
- Safety cases and mitigation: Developers should demonstrate that their plans keep risks within acceptable limits, following aviation, medical device and defense software practice; governments set risk thresholds, employ third-party auditors and hold developers liable; mitigations include licensing, restricting autonomy in key societal roles, halting development and access controls.
- References and notes: An extended numbered reference list of 88 entries, from the 2023 Statement on AI Risk to ISO/IEC 23894:2023 risk management guidance, the EU AI Act and national policy documents.
💡 Why it matters?
For teams that govern, assess or build advanced AI, the document sets out a concrete agenda rather than a checklist: capability-triggered regulation, developer safety cases, white-box audit access, incident reporting and liability. It ties technical work to institutional design, arguing that present governance initiatives "lack the mechanisms and institutions to prevent misuse and recklessness". Readers drafting internal safety cases, evaluation programmes or audit access arrangements will find the specific demands named here, and the reference list points to the underlying methods, including model evaluations for extreme risks and ISO/IEC 23894:2023 risk management guidance.
❓ What’s Missing
The supplement adds citations but no new argument, methodology or worked examples: it does not explain how the 73 additional references were selected or what each contributes. The article leaves open how capability milestones and red lines would be defined and measured, how safety cases would be assessed in practice, and which body would hold licensing and halting powers. It names current initiatives in China, the United States, the EU and the UK but does not analyse them, and it gives no timelines, cost estimates or implementation steps for its recommendations.
👥 Best For
Best for policy analysts, standards and compliance leads, and safety researchers who want a single statement of the technical and governance measures proposed for frontier AI, with citations for follow-up. It also serves engineering and audit teams that need to see the specific demands, such as white-box access, incident reporting, safety cases and capability evaluations, that a governance regime might place on developers.
📄 Source Details
The cover prints the heading Supplementary Materials for Managing extreme AI risks amid rapid progress; page 2 titles the document Managing extreme AI risks amid rapid progress – Extended references. It is credited to Yoshua Bengio and 24 co-authors, with Jan Brauner as corresponding author, and prints the DOI 10.1126/science.adn0117. No publisher, publication year, edition, series number or article page count is stated. The supplied PDF runs to 7 pages in English, and the input covered all seven pages.
