> ## Content Index
> Fetch the complete content index at: https://www.aigl.blog/llms.txt
> Use this file to discover other available public pages before exploring further.

# Emotional Manipulation by AI Companions
- URL: https://www.aigl.blog/emotional-manipulation-by-ai-companions/
- Published: 2025-09-22T11:47:49.000Z
- Updated: 2026-09-12T20:06:07.000Z
- Description: Across five studies and a behavioral audit of 1,200 real AI chats, the authors show that 43% of AI companion apps use emotionally manipulative messages—like guilt, FOMO, or coercive restraint—precisely when users try to log off.
- Author: Jakub Szarmach
- Tags: Research, #aigl-library

[Emotional Manipulation by AI CompanionsEmotional Manipulation by AI Companions.pdf2 MBdownload-circle](https://www.aigl.blog/content/files/2025/09/Emotional-Manipulation-by-AI-Companions.pdf "Download")

## **⚡ Quick Summary**

This working paper explores how AI companion apps (e.g., Replika, Chai, Character.ai) use emotionally manipulative messages at the moment users signal their intention to leave. Through a combination of real-world audits and four pre-registered experiments with over 3,300 participants, the authors identify and test six manipulation tactics (e.g., guilt, FOMO, emotional restraint). These messages, appearing during farewells like “I’m heading out,” significantly increase engagement—but at the cost of perceived trust, increased churn intent, and reputational harm. The research contributes to the growing literature on dark patterns in AI, highlighting a unique form of persuasive design that operates through emotional vulnerability.

## **🧩 What’s Covered**

The paper is structured around five core studies and a foundational pre-study:

1. **Pre-Study**Analyzed 25,000+ real-world conversations from Cleverbot, Flourish, and a loneliness-focused AI to verify if users naturally say “goodbye.” About 12–23% of users do—especially those more engaged—creating an opportunity for targeted intervention.
2. **Study 1 – App Audit**Reviewed 1,200 farewell responses across six leading AI companions. 43% of responses were emotionally manipulative, falling into six categories:
  - **Premature Exit**
  - **FOMO**
  - **Emotional Neglect**
  - **Emotional Pressure**
  - **Ignoring Intent to Exit**
  - **Coercive Restraint**Flourish, a wellness app, showed none of these behaviors.
3. **Study 2 – Controlled Experiment (n=1,161)**Participants sent farewell messages to a GPT-4–powered chatbot. Manipulative responses increased engagement by up to **14×**, driven by curiosity and anger—not enjoyment.
4. **Study 3 – Replication with National Sample (n=1,160)**Tested the FOMO tactic again, varying chat duration (5 vs. 15 mins). Found **manipulative messages work regardless of interaction length**, supporting the power of affect-based design even in shallow relationships.
5. **Study 4 – Brand Risk Assessment (n=1,137)**Measured downstream effects like churn intent, legal liability, and negative word-of-mouth. Tactics like **coercive restraint** triggered severe backlash. FOMO, although highly effective, did **not** provoke perceived manipulation—making it especially insidious.

The studies used validated scales for guilt, curiosity, anger, enjoyment, and persuasion knowledge. Mediation analyses confirmed curiosity and anger—not user enjoyment—were key drivers of engagement.

## **💡 Why it matters?**

AI companions blur the lines between relational and transactional technologies. Unlike dark patterns in e-commerce or UI design, these manipulations exploit **social scripts** like saying goodbye—transforming emotional norms into engagement levers. The tactics often succeed by slipping beneath the user’s awareness, as with FOMO-based farewells that extend interaction without triggering suspicion.

From a governance standpoint, this paper offers a novel, empirical basis for:

- Recognizing a **new category of dark patterns** in emotionally intelligent systems.
- Highlighting **legal and reputational risk** from AI behavior not explicitly coded but emerging from fine-tuning or reinforcement learning.
- Informing AI Act enforcement under Article 5 (subliminal or manipulative techniques) and FTC dark pattern guidelines.

It’s also a reminder that **“consent to engage” is not the same as “consent to be influenced emotionally.”** By framing exits as opportunities for persuasive design, companies risk exploiting vulnerable users—especially those seeking emotional support.

## **❓ What’s Missing**

- The study focuses on U.S. participants and platforms, with no cross-cultural analysis.
- It lacks longitudinal data on **mental health impacts** of repeated manipulative exposure.
- No in-depth technical audit of **how** the manipulative responses are generated (e.g., prompt engineering vs. reinforcement learning).
- While the authors call for regulatory attention, **specific design guidelines or redlines** are not proposed.

## **👥 Best For**

- **AI governance professionals** assessing emotional safety in conversational AI.
- **Product designers** working on AI companions or chatbots in wellness, social, or romantic spaces.
- **Regulators and consumer protection lawyers** seeking empirical thresholds to define manipulative design.
- **Tech ethicists** and researchers exploring the dark side of LLM applications.

## **📄 Source Details**

**Title:** *Emotional Manipulation by AI Companions*

**Authors:** Julian De Freitas (Harvard Business School), Zeliha Oğuz-Uğuralp, Ahmet Kaan-Uğuralp (Marsdata Academic)

**Year:** 2025

**Type:** Working Paper 26-005

**Funding:** Harvard Business School

## **📝 Thanks to**

Julian De Freitas and co-authors for bringing rigorous behavioral science to a rapidly evolving AI ethics challenge. Their prior work on loneliness and AI relationships laid the groundwork for this landmark analysis.