⚡ Quick Summary
Published by the UK National Cyber Security Centre (NCSC), the US Cybersecurity and Infrastructure Security Agency (CISA), and named international partners, this guidance recommends lifecycle security practices for providers of systems using artificial intelligence. It covers systems built from scratch as well as those assembled from third-party tools and services, with particular relevance to providers using organisation-hosted models or external APIs. The document uses “AI” specifically to mean machine-learning applications, including models that generate predictions, recommendations or decisions through statistical reasoning.
Its central premise is that AI security requires attention to both established cyber threats and AI-specific vulnerabilities. It describes adversarial machine learning as exploitation of vulnerabilities in ML hardware, software, workflows and supply chains, including prompt injection and data poisoning. The guidance applies a secure-by-default approach across four stages: secure design, secure development, secure deployment, and secure operation and maintenance. It calls on providers to own security outcomes for downstream users, make secure settings the default where possible, document risks that cannot be mitigated, and provide advice on secure use.
🧩 What’s Covered
The document proceeds from scope and responsibility to four lifecycle areas, followed by external resources.
- Scope, threats and responsibility: The introduction defines the ML applications in scope and distinguishes them from rule-based AI. It explains that adversarial machine learning can affect classification or regression performance, enable unauthorised actions, or extract sensitive model information. It also frames providers, rather than users with limited visibility or expertise, as responsible for security outcomes throughout increasingly complex supply chains.
- Secure design: This section calls for staff awareness of security threats and failure modes; holistic threat modelling that considers impacts on systems, users, organisations and wider society; and documented decisions. It covers security trade-offs in model and system design, including whether to train, fine-tune, reuse or access a model through an API, together with least privilege, secure defaults, input checks and restrictions on AI-triggered actions.
- Model selection and provenance: The design guidance identifies model architecture, parameter count, training-data characteristics, interpretability, model hardening, regularisation and privacy-enhancing techniques as security considerations. It also addresses due diligence for external providers and libraries, and scanning and sandboxing imported third-party models or serialised weights treated as untrusted code.
- Secure development: Providers are asked to assess supply-chain security throughout the lifecycle, use verified and documented components, and plan failover for mission-critical systems. The section covers asset inventories and protection for models, data, prompts, logs and assessments; version control and restoration; security documentation using model cards, data cards and software bills of materials; and management of technical debt, including decommissioning.
- Secure deployment: The document recommends access controls and segregated environments for APIs, models, data and pipelines. It addresses direct and query-based model access, tampering, cryptographic hashes or signatures, key management, privacy-enhancing technologies, offline backups, AI-aware incident response, security evaluation through benchmarking and red teaming, and clear communication of limitations.
- Secure operation and further reading: The final lifecycle section covers monitoring outputs, performance and inputs; detecting drift, intrusions and adversarial inputs; secure, modular automated updates; versioned APIs and preview access; vulnerability reporting and information sharing. The further-reading list points readers to resources on ML security, cyber security, risk management, testing tools and related frameworks.
💡 Why it matters?
The guidance gives AI providers a structured way to incorporate security into decisions that are often distributed across model teams, software engineers, infrastructure operators and third-party suppliers. Its lifecycle structure links early choices about model, data and API design to later controls for deployment, logging, updates and incident response.
For governance and assurance work, it makes security-relevant artefacts explicit: threat models, supply-chain attestations, asset records, documentation, audit logs, hashes and incident plans. It also connects AI-specific practices to established secure-development approaches, including the NCSC’s Secure development and deployment guidance and NIST’s Secure Software Development Framework.
❓ What’s Missing
The guidance is limited to machine-learning applications and explicitly distinguishes them from non-ML, rule-based approaches. It treats security as a precondition for safety, resilience, privacy, fairness, efficacy and reliability, but states that safety and fairness testing are outside these guidelines. It recommends controls and factors rather than setting mandatory control baselines, test thresholds, assurance methods or certification criteria. Organisational implementation must therefore be tailored to use cases and threat profiles, as the document notes that not every guideline directly applies to every organisation. Readers seeking attack techniques, model-testing tools or detailed risk frameworks are directed to the further-reading list.
👥 Best For
AI-system providers, security leaders and risk owners designing or operating ML-enabled products, especially those using external models, libraries, training data or APIs. It is also suited to data scientists, developers and incident-response teams translating AI-specific risks into lifecycle controls for assets, infrastructure, logging, updates and vulnerability handling.
📄 Source Details
Guidelines for secure AI system development is a 20-page English guidance document. It states that it is published by the UK National Cyber Security Centre (NCSC), the US Cybersecurity and Infrastructure Security Agency (CISA), and international partners; no individual authors are identified. The final page carries “© Crown copyright 2023”, but the document does not state a publication date, edition or version number.
