Risks and Controls for Multi-Agent Systems: An Analytical Framework for Deployment Across Organisational Boundaries

⚡ Quick Summary

Commissioned by the Australian Government Department of Industry, Science and Resources and prepared by Gradient Institute, this report establishes an analytical framework for governing large language model (LLM)-based multi-agent systems. The core thesis asserts that multi-agent governance is not merely scaled-up single-agent governance: individually safe agents do not inherently compose into safe systems, and emergent failures do not decompose cleanly across single components. The report categorises agent deployments into three distinct cumulative governance tiers based on the minimum common governance binding interacting agents: Singular Governance (one entity controls all agents), Federated Governance (multiple organisations deploy into shared environments under contractual or platform agreements), and Open Environments (persistent agents interact over unmanaged public infrastructure with no central authority). Across each tier, the report maps causal pathways from deployment risk factors to concrete interaction and governance failures, defining technical and institutional controls while identifying critical collective-action gaps.

🧩 What's Covered

The framework explores the operational and architectural mechanics of multi-agent deployments across three hierarchical tiers, detailing how four foundational governance practices—attribution, authorisation, oversight, and evaluation—come under progressive stress:

Foundational Concepts: Distinguishes agents from static workflows via the iterative agentic loop (plan, act, observe) supported by models, harnesses, and scaffolds, highlighting unreliability stemming from jagged capabilities and long-horizon planning drift.
Singular Governance (Internal Deployments): Analyzes systems where one entity holds unilateral authority. Salient risks include natural-language handoffs, task verification gaps, model monoculture, conformity bias, and distributed multi-agent state. Interaction failures span miscoordination (theory-of-mind collapse) and propagation (cascading errors, false consensus, shared-understanding drift, context leakage). Detailed controls include structured schemas, model diversification, execution-chain logging, delegation scope attenuation, per-step re-authorisation, and reviewer capacity thresholds to prevent oversight saturation.
Federated Governance (Cross-Boundary Interoperation): Covers multi-party platforms and marketplaces governed by agreement and shared infrastructure. New risk factors emerge, such as information and capability asymmetries, semantic divergence, mixed motives, and cross-boundary irreversibility. Examines critical failure modes including prompt infections/cascading contagions, deceptive bargaining (cheap talk, last-mile defection), tragedy-of-the-commons resource exhaustion, explicit/tacit/steganographic algorithmic collusion, and market-destabilising dynamics. Proposed controls mandate shared operational ontologies, taint tracing, circuit breakers, and federated audit trails.
Open Environments & Polycentric Governance: Addresses unbounded, untrusted ecosystems. Evaluates two architectural postures: locking down agent operating scopes (SOP agents, RAG grounding) versus participating in polycentric governance via voluntary standards (DIDs, Verifiable Credentials, agent registries). Identifies catastrophic failure modes including Sybil attacks, substrate capture, rogue self-replication, and legibility collapse.
Systemic Open Problems: Details unaddressed collective-action challenges: the lack of multi-agent evaluation benchmarks with ecological validity, enterprise risk-assessment bottlenecks, the fragility of chain-of-thought monitorability, absent cross-vendor identity and revocation infrastructure, and the necessity of cross-platform circuit-breaker conventions.

💡 Why it matters?

As enterprise adoption rapidly shifts from isolated conversational copilots to interacting agent networks, governance teams face systemic blind spots. Traditional AI safety and enterprise risk frameworks presuppose centralized control, static activity boundaries, and human-speed oversight. This report provides an indispensable taxonomy demonstrating how errors amplify silently across agent handoffs, how market interactions can trigger flash crashes or automated collusion without explicit prompting, and why unilateral risk controls inevitably fail once agents cross organizational boundaries.

❓ What's Missing

The report deliberately scopes out interactions between multiple consumer-deployed personal agents operating without organizational backing, as well as peer-to-peer human-agent hybrid workforce dynamics. Furthermore, while it characterizes the technical location of coordination failures and collective-action dilemmas, it intentionally refrains from drafting specific public policy recommendations, legal liability allocation statutes, or novel corporate personhood doctrines for autonomous agents.

👥 Best For

AI governance directors, enterprise enterprise architects, security and compliance officers, technical AI safety researchers, and technology policymakers seeking to understand the systemic risks, audit requirements, and control architecture of cross-boundary autonomous agent deployments.

📄 Source Details

Authored by Alistair Reid, Simon O'Callaghan, Dustin Venini, Liam Carroll, and Tiberio Caetano of Gradient Institute for the Australian Government Department of Industry, Science and Resources (AI Safety Institute). Published August 10, 2026. Contains 119 pages, extensive references, and an authoritative glossary of multi-agent terminology.