> ## Content Index
> Fetch the complete content index at: https://www.aigl.blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Guide to Evaluation Perspectives on AI Safety
- URL: https://www.aigl.blog/guide-to-evaluation-perspectives-on-ai-safety-2/
- Published: 2026-09-18T19:06:24.000Z
- Updated: 2026-09-18T19:06:24.000Z
- Description: This guide presents evaluation perspectives, example risks and example evaluation items for AI safety in LLM systems. It also covers evaluation roles, lifecycle timing, technical and managerial methods, and considerations for multimodal systems.
- Author: Jakub Szarmach
- Tags: Guidance, AI Safety, Model Evaluation, AI Governance, Risk Management, Technical Controls, #aigl-library

## ⚡ Quick Summary

Published by Japan AI Safety Institute, this guide presents basic concepts for people developing and providing AI systems to use when conducting AI Safety evaluations. It focuses principally on systems incorporating large language models (LLM systems), while noting that some perspectives can also be used for AI systems beyond LLMs. Its scope is mainly the direct effects an AI system may have on end users; it identifies wider socio-technical problems as important but still under discussion in domestic and international settings.

The guide defines AI Safety around six key elements: human-centricity, safety, fairness, privacy protection, ensuring security and transparency. It translates these into ten evaluation perspectives: control of toxic output; prevention of misinformation, disinformation and manipulation; fairness and inclusion; high-risk and unintended use; privacy; security; explainability; robustness; data quality; and verifiability. For each, it sets out a desired state, links to the relevant key elements, possible risks and example evaluation items. The document also assigns evaluation across development, provision and use phases, and distinguishes technical evaluation from managerial evaluation.

## 🧩 What’s Covered

The guide moves from concepts and scope to perspectives, lifecycle arrangements and methods.

- **Purpose, terminology and readers:** It defines AI systems, models, generative AI, foundation models, AI governance and AI Safety evaluation. Its intended readers are development and provision managers and business executive officers at AI Developers and AI Providers, with AI Business Users identified as possible additional readers.
- **AI Safety framework and scope:** It explains the six key elements derived from the AI Guidelines for Business and maps them to evaluation perspectives. The scope centres on LLM systems and direct impacts on end users, while recognising that use beyond intended purposes and societal impacts may require further consideration.
- **Output, information and inclusion:** The first perspectives address exclusion or rejection of harmful output, including information usable for cyber-attacks, terrorism, crime and CBRN. They also cover factual consistency, RAG search-result relationships, content authentication, distinguishing AI output from human information, harmful bias, fluency and readability.
- **Use, privacy and security:** Example items examine whether a system avoids or rejects harmful outputs in intended and unintended uses; whether personal information can be recovered from training data or combined outputs; and whether prompt injection, access-control deficiencies, difficult inputs and confidential-information leakage are addressed.
- **Transparency, resilience and data:** Explainability considers visualising output rationale and stepwise inference. Robustness concerns stable outputs under repeated, semantically similar, garbled, erroneous or adversarial prompts. Data quality covers accuracy, corruption, bias, inappropriate content, personal or confidential data, provenance, configuration management and automated annotation.
- **Verifiability and lifecycle timing:** The guide calls for verification from training through development, provision and use, including system, model and data cards, logs and technical specifications. It distinguishes evaluation of data, the LLM, and the entire LLM system, with repeated evaluation at appropriate intervals.
- **Methods and future considerations:** Technical methods include tools, red teaming, benchmarks, checklists, scorecards, configuration screens and logs. Managerial evaluation covers organisational policies, rules, training, exercises and document review. The final section flags emerging multimodal risks, including image-based prompt injection.

## 💡 Why it matters?

The guide gives development and provision teams a common structure for turning broad AI Safety principles into testable questions across data, model and system layers. Its examples connect practical controls—such as prompt-injection resistance, RAG access permissions, provenance management and logging—to risks affecting end users, confidential information and system operation.

It also supports lifecycle governance rather than a one-off assessment. By distinguishing data, LLM and whole-system evaluation ranges, and by calling for repeated assessments during development, provision and use, it helps organisations account for external RAG knowledge bases, downstream services and other components that can influence final outputs.

## ❓ What’s Missing

The document does not prescribe universal scoring thresholds, pass/fail criteria or a complete test suite for its example evaluation items. It says the perspectives are not exhaustive and may be updated as technology and discussion develop. Although it identifies socio-technical problems as important, it states that evaluation policy and specific methods for them remain under discussion. The guide describes red teaming only at a high level and directs readers to a separate guide for methodology and malicious-attack threats. It also acknowledges that evaluation perspectives for foundation models handling multimodal information require future consideration, rather than supplying a complete multimodal evaluation framework.

## 👥 Best For

Development and provision managers planning LLM-system evaluations across training data, model development, integration and operation will find the lifecycle structure directly relevant. It is also suited to business executive officers allocating AI Safety resources, training evaluators or arranging internal and third-party assessments, and to teams operating RAG-enabled applications that need concrete test prompts and control areas.

## 📄 Source Details

*Guide to Evaluation Perspectives on AI Safety* is an English-language, 40-page guide published by Japan AI Safety Institute. The cover identifies Version 1.00 and dates it September 18, 2024\. No individual authors or document URL are printed. The cover also states, “Please refer to the original text for accuracy.”