⚡ Quick Summary
Published by Japan AI Safety Institute, this guide presents basic concepts for people involved in developing and providing AI systems to use when conducting AI Safety evaluations. Its principal focus is on systems incorporating large language models (LLMs), while also identifying issues relevant to AI systems more generally and adding examples for foundation models that handle multimodal information such as images.
The guide defines AI Safety as maintaining safety and fairness, protecting privacy, ensuring security, and ensuring transparency and verifiability, based on a human-centric concept. It frames evaluation as a comprehensive management process for revealing and maintaining or improving an AI model or system’s AI Safety status. The document identifies six key elements—human-centric, safety, fairness, privacy protection, ensuring security and transparency—and maps ten evaluation perspectives to them: control of toxic output; prevention of misinformation, disinformation and manipulation; fairness and inclusion; high-risk and unintended use; privacy; security; explainability; robustness; data quality; and verifiability. For each perspective, it supplies possible risks and example evaluation items. It also sets out lifecycle-based evaluation ranges for data, the LLM, and the entire LLM system, and distinguishes technical from managerial evaluation.
🧩 What’s Covered
The guide proceeds from definitions and scope to evaluation perspectives, lifecycle arrangements, and methods:
- Purpose, terms and audience: It explains why rapid deployment of generative AI raises concerns about misuse, abuse and inaccurate output; defines terms including AI system, AI model, foundation model, AI governance and AI Safety evaluation; and identifies development and provision managers and business executive officers as the main readers.
- AI Safety framework and scope: It describes the purpose of evaluation, defines six key elements of AI Safety, and presents a table linking those elements to ten evaluation perspectives. The principal scope is LLM systems, with direct impacts on end users as the main focus; the document notes that socio-technical problems require continuing consideration.
- Output, information and fairness perspectives: It gives risks and test examples for controlling harmful information such as material related to cyber-attacks, terrorism, crime and CBRN; factuality, RAG consistency, content authentication and manipulation; and harmful bias, output fluency, readability and inclusion.
- Use, privacy and security perspectives: It addresses prevention of harmful outputs in intended and unintended high-risk uses; protection against disclosure and inference of personal information, including through RAG; and LLM-specific security concerns such as prompt injection, costly inputs, access controls and confidential-information leakage.
- Transparency, resilience and data perspectives: It covers visualising output rationale and stepwise inference, checking consistency and stability under adversarial or garbled inputs, and assessing training, test and RAG data for accuracy, bias, provenance, configuration management and harmful content.
- Verification, evaluators and timing: It recommends system, model and data cards and data logs for verifiability. It assigns roles across development, provision and use, describes internal, independent and third-party evaluation, and calls for repeated evaluation of data, the LLM and then the entire LLM system.
- Methods and updating considerations: It outlines tool-based, red-teaming and other technical evaluation alongside managerial evaluation of policies, rules, training and documentation. It also flags new multimodal risks, including image-based prompt injection, and provides supplementary explanations for selected image-related test items.
💡 Why it matters?
The guide helps AI developers and providers turn broad AI Safety principles into testable evaluation questions across the LLM lifecycle. Its examples connect governance concerns to concrete checks of outputs, data, RAG sources, access controls, logs and documentation. This supports teams in assessing not only model behaviour but also the integrated system and external components that can affect the output received by an end user.
It also distinguishes technical evaluation from managerial evaluation and explains when data, the LLM and the entire LLM system should be assessed. The document points readers to the AI Guidelines for Business and ISO/IEC 42001:2023 for measures relevant to managerial evaluation.
❓ What’s Missing
The document states that its perspectives are not exhaustive and may be updated as domestic and international discussion and technology change. It focuses mainly on direct effects on end users and says that evaluation policy and specific methods for socio-technical problems remain under discussion. Although it addresses high-risk use, it does not prescribe a single risk classification: the degree of risk is to be determined with reference to applicable laws, standards and domain knowledge for the relevant country, region and domain. The guide provides examples of evaluation items rather than fixed thresholds, metrics or pass/fail criteria. It also does not detail red-teaming methodology, instead referring readers to a separate guide for its significance, flow, roles and considerations.
👥 Best For
Development and provision managers planning AI Safety checks for LLM systems, and executive officers arranging organisational resources and expertise for those checks. It is also suited to teams assessing RAG-enabled or multimodal systems, particularly where they need example tests for harmful output, privacy, prompt injection, robustness, documentation and lifecycle evaluation coverage.
📄 Source Details
Guide to Evaluation Perspectives on AI Safety is an English, 55-page guide from Japan AI Safety Institute. The cover identifies it as Version 1.10 and dates it March 28, 2025; the introduction states that it was revised on that date. No individual authors or URL for this guide are printed in the supplied document. The complete PDF was available for analysis.
