⚡ Quick Summary
Published by AI Security Institute, this report sets out a snapshot of its current research priorities for understanding the capabilities, impacts and mitigations of advanced AI. The Institute presents its role as equipping governments with a scientific understanding of serious emerging AI risks, while building assessment infrastructure, tools and practices. It identifies three routes to impact: informing state awareness, developing international protocols, and acting as an independent technical partner to frontier AI labs.
The agenda divides work into Risks Research and Solutions Research. Risks Research covers cyber misuse, dual-use science, criminal misuse, autonomous systems, societal resilience and human influence, alongside generalised research on the science of evaluations and capabilities post-training. Solutions Research develops safety cases, empirically tests technical mitigations and promotes external research. Its technical programme includes automated and expert-led evaluations, threat modelling, adversarial red-teaming, capability elicitation, control environments and alignment research centred initially on AI-system honesty. The document also records examples including a suite of 80+ automated cyber evaluations, a 50-task autonomous replication benchmark, AgentHarm, and ControlArena.
🧩 What’s Covered
The agenda follows the Institute’s approach from research priorities through risk assessment to technical mitigations.
- Research approach and impact: Sets prioritisation criteria around severe, widespread and national-security harms, advanced AI capabilities, and interventions suited to a government-backed research organisation. It explains the intended pathways from research to government awareness, international protocols and collaboration with model developers.
- Risk research framework: Separates domain-specific research from generalised measurement research. The domain list comprises cyber misuse, dual-use science, criminal misuse, autonomous systems, societal resilience and human influence; the Institute notes that it does not cover details of its dual-use science work here.
- Cyber and criminal misuse: Describes cyber threat scenarios, rapid automated assessments, expert probing, cyber range evaluations and model-layer mitigations. Criminal-misuse work maps threat models, tests compliance with harmful requests, uses expert red-teaming, and aims to improve refusal training and user monitoring.
- Autonomous systems: Applies Fault Tree Analysis, System-Theoretic Process Analysis and safety case sketches to loss-of-control risks. It uses directed acyclic graphs to break dangerous capabilities into sub-capabilities, and identifies replication, weight exfiltration, sandbagging, collusion and deceptive reasoning as evaluation concerns.
- Societal resilience and human influence: Proposes monitoring AI adoption and societal-scale risks using quantitative and qualitative evidence. Human-influence work uses lab-based user studies and surveys to examine persuasion, deception, para-social relationships and imperceptible influence.
- Measurement and elicitation: The Science of Evaluations programme stress-tests evaluation claims, methods and uncertainty analysis, including hierarchical Bayesian modelling. Capabilities Post Training develops prompting, tools, agent architectures, fine-tuning and reinforcement learning to assess models’ performance ceilings.
- Safeguard analysis: Covers adversarial attacks on refusal systems, fine-tuning APIs and unlearning; evaluation principles; defence-in-depth; data and model poisoning; prompt injection; and risks associated with open-weight models. It describes AgentHarm and planned human jailbreaking studies and bug bounties.
- Control and alignment: Presents safety cases, realistic mock environments and adversarial control evaluations for systems used within frontier developers’ infrastructure. Alignment research combines theoretical and empirical work on honesty, asymptotic guarantees, scalable oversight, exploration, distribution shift and automated alignment research.
💡 Why it matters?
The agenda is relevant to teams deciding how to assess advanced models before and after deployment because it connects model capabilities to concrete risk scenarios, human input requirements and mitigation testing. It distinguishes rapid evaluations from expert probing and calls for calibration against human baselines, helping frame what an evaluation result can and cannot establish.
For developers and public bodies, it also links technical work to deployment decisions: safeguards are tested against jailbreaks and third-party attacks, while safety cases structure the claims, arguments and evidence needed to justify particular control measures. The document explicitly positions its findings for policy and governance interventions, standards and protocols, and targeted safety improvements with frontier developers.
❓ What’s Missing
This is an agenda rather than a full methods or results report. The Institute states that, because of the sensitivity of its work, it cannot publish the full scope of its methods and research objectives. Although dual-use science is listed as a priority risk domain, the document explicitly says it will not cover details of that work. Many sections describe planned research rather than completed studies, and their future-objective lists are stated to be non-exhaustive and subject to consultation and government priorities. The criminal-misuse evaluation example also acknowledges that evaluations run to date are text-to-text only, compliance-only and single-shot. It does not provide a complete inventory of evaluated models, detailed protocols, underlying datasets, full empirical results or implementation requirements for the proposed mitigations.
👥 Best For
Useful for UK government and national-security stakeholders, frontier-model developers, AI safety researchers and evaluation teams seeking a consolidated view of AISI’s research priorities. It is particularly suited to teams designing capability evaluations, threat models, safeguards, control protocols or alignment research for high-impact and agentic AI systems.
📄 Source Details
The UK AI Security Institute’s Research Agenda is an English-language, 32-page report published by the AI Security Institute in May 2025. No individual authors, edition, reference number or document URL are printed. The PDF has a cover followed by pages numbered 1–31 and is marked “OFFICIAL.”
