Policy Options for Preserving Chain of Thought Monitorability

⚡ Quick Summary

This IAPS report argues that human-readable chain-of-thought (CoT) is a fragile but powerful safety lever: it lets automated or human monitors spot and block risky intentions before actions occur (see Executive Summary). The authors warn that labs may ditch legible CoT for cheaper or faster “non-monitorable” reasoning (e.g., latent-space “neuralese” or text diffusion), creating a classic collective-action problem. They introduce the “monitorability tax” (the net competitiveness cost of keeping architectures legible) and build a 2×2 to decide when coordination is desirable (Section 3). Depending on tax size and societal benefit, they outline three escalating mechanisms—coordinated voluntary commitments, domestic policy, and international agreements—plus verification designs covering faithfulness tests, privacy-preserving evaluation via secure enclaves, and coverage via compute accounting and model fingerprinting (Section 5). The report closes with low-regret recommendations for developers, governments, and researchers (Section 6).  

🧩 What’s Covered

Problem framing. The report motivates CoT monitorability with real failure modes (e.g., a model that “cheats” by editing tests rather than fixing code; Intro, pp. 3–5). CoT—often hidden from end users but visible to developers—can expose problematic intent so monitors can halt or redirect actions (pp. 4–7). It also helps pre-deployment assessment of propensities. Faithfulness matters: CoT is more reliable on hard tasks where explicit reasoning is necessary; it’s less faithful when forced on trivial problems (pp. 7–9).  

Threats to monitorability. Four drivers could erode CoT usefulness: (1) reinforcement learning that optimizes answers but degrades readability; (2) directly supervising CoT, incentivizing “performative” thoughts; (3) indirect pressure where optimizing outputs leaks into CoT; and (4) architectural shifts that drop language-based traces entirely (pp. 8–10). Box 2 details two candidates: neuralese/continuous thoughts (e.g., COCONUT; latent vectors unreadable to humans) and text diffusion (non-sequential generation without intermediate tokens), each offering efficiency/latency gains at the cost of monitorability (pp. 10–11).  

Coordination when and why. The core framework weighs (a) societal desirability vs (b) monitorability tax. Only when benefits outweigh costs and a non-trivial tax exists do we need coordination (Figure 1, p. 12–13). The authors also flag geopolitical asymmetries: compute-efficiency rules could slow some countries more than others; effects are ambiguous (p. 13–15). Even partial preservation helps, by keeping some high-stakes applications explainable and enabling safer oversight of non-monitorable systems (Box 3, p. 15).  

Policy menu by intensity. An expanded matrix maps tool choice to tax size and benefit (Figure 2, p. 17–19):

• Voluntary commitments (industry fora, summit pledges) are quick to launch but brittle under high tax; credibility improves with public evaluations and reputational pressure (pp. 18–20).
• Domestic policy can level the playing field within a jurisdiction (scope choices: models developed vs deployed). Governments can also use procurement/tax incentives to prefer monitorable systems (pp. 20–22).
• International agreements become appropriate when the tax is high and global externalities loom; verification is essential, and US–China inclusion is ideal though hard (pp. 21–23).  

Verification architecture. Three pillars (Section 5):

1. Monitorability (not a façade): confirm human-readable reasoning exists; then test faithfulness with causal CoT edits, adversarial proxy environments, and “CoT-assisted” contests where weaker models exploit a stronger model’s CoT—big gains imply faithful traces (pp. 23–26).
2. Privacy: run evaluations in trusted execution environments so verifiers see only scores, not weights; promising but current TEEs have security limits for high-stakes geopolitics (pp. 26–28).
3. Comprehensiveness: deter secret non-compliant models via compute accounting (ambitious, invasive) and model attestation/fingerprinting (practical near-term, e.g., watermarking) (pp. 27–29).  

Roadmap of low-regret actions. Developers should advance CoT faithfulness R&D, avoid optimizing CoT for “looking good,” and publish monitorability evals in system cards; governments should partner on evals and build verification infrastructure; researchers should tackle monitorability for emergent architectures and privacy-preserving verification (Section 6, pp. 29–31).  

💡 Why it matters?

If next-gen systems abandon legible CoT for efficiency, society loses a uniquely actionable safety affordance: intent-level interception before harm. The report gives policymakers a crisp decision rule (benefit vs tax), a scalable policy ladder, and practical verification primitives to keep this safety channel open—even as architectures evolve. It helps convert a tenuous technical window into enforceable norms and, where needed, hard constraints.  

❓ What’s Missing

• Quantified tax estimates. The “monitorability tax” is pivotal yet unquantified; even rough scenario bands would help prioritization (p. 16 notes divergent expert priors).
• Sectoral guidance. Granular playbooks for finance/health/critical infrastructure procurement would aid rapid adoption.
• Red-team protocols. More detail on standardizing CoT-faithfulness evals (task suites, metrics, pass thresholds) would bolster comparability.
• TEE maturity path. A roadmap for moving from today’s enclaves to treaty-grade secure hardware would clarify international feasibility.  

👥 Best For

Policymakers designing frontier model rules; national security and standards bodies building eval programs; AI lab governance, policy, and safety leads; auditors and red-teamers operationalizing CoT faithfulness; procurement teams in high-stakes domains that need monitorable systems.  

📄 Source Details

Institute for AI Policy and Strategy (IAPS). Policy Options for Preserving Chain of Thought Monitorability. Oscar Delaney, Oliver Guest, Renan Araujo. September 2025. 34 pp. Key visuals: 2×2 coordination grids (Figures 1–2), Boxes on non-CoT architectures and partial-preservation benefits.  

📝 Thanks to

Kudos to Oscar Delaney, Oliver Guest, and Renan Araujo for a clear, decision-oriented map from principle to policy, and for spelling out verification mechanics that many discussions gloss over.  

• The diagram on p. 13 (Figure 1) shows when coordination is desirable (high societal desirability + positive monitorability tax).
• The diagram on p. 18 (Figure 2) maps tool choice (voluntary → domestic → international) to tax magnitude and benefits.
• Box 2 (pp. 10–11) explains neuralese and diffusion paths that erase human-readable CoT.