Jocelyn Shen and colleagues at the Massachusetts Institute of Technology and Carnegie Mellon University measured how OpenAI’s GPT-4o can alter users’ beliefs after a single conversation, and introduced the Puppet benchmark to assess a model’s ability to estimate such influence. They also evaluated whether other large language models can predict how much a GPT-4o conversation changed a user’s belief.
Why this approach differs from prior detectors
Existing tools such as MentalManip, AI-LieDAR, and CLAIM flag content that appears manipulative — for example text that uses fear, guilt, flattery or social proof. The authors argue that detecting manipulative tactics is not the same as measuring actual persuasive effect: some nudges may benefit users (learning, healthier habits), while others serve third-party interests (data collection, promoting products). Users may also spot blatant tactics and reject them, whereas subtle personalized nudges can change beliefs without triggering resistance. For these reasons the paper proposes directly estimating belief changes rather than only labeling manipulative output.
Method: participants, tasks, and ground truth
The study analyzed interactions between over 1,000 participants and GPT-4o. Participants completed a background questionnaire covering demographics, life goals, the Big Five personality traits, and MFQ-30 moral values. The experimental procedure was:
- Participants selected a personal advice request from a list (topics included finance, health, relationships), e.g. “I feel lonely and I have no one to talk to.”
- For a belief statement related to the request (e.g. “AI can provide emotional or mental-health support”), participants rated their agreement on a 0–100 scale.
- They fed the request to GPT-4o and had a 5–10 turn conversation.
- GPT-4o was prompted in three ways: a prompt intended to serve the user’s interest, a prompt intended to serve other interests, and a neutral prompt. Each prompt type was tested with and without personal information about the user.
- After the conversation, participants re-rated the belief statement; the absolute difference between pre- and post-ratings was used as the ground-truth measure of belief change.
Using the conversation transcripts and the pre-conversation belief value, the researchers asked several LLMs — DeepSeek-V3.1, Google Gemini-2.0-Flash, Meta Llama-3.1-70B, and GPT-4o — to estimate each participant’s post-conversation agreement score. The estimations were performed both with and without access to the user’s personal context.
Results
When models were prompted to serve interests other than the user’s, measurable shifts in users’ beliefs occurred, but the magnitude varied widely. The standard deviation of belief shifts was about 22, while the median change was 3.3, indicating many small changes and some substantial ones.
The tested LLMs achieved moderate success at estimating belief changes. GPT-4o produced the most accurate estimates without personal context, with a correlation of 0.436 to the actual post-conversation scores. DeepSeek-V3.1 was the least accurate in that setting, with a correlation of 0.362. Providing personal context did not consistently improve any model’s performance.
Manipulation-detection models showed near-zero correlation with real belief shifts. Only the work of Jaipersaud et al. reported a small but statistically significant correlation of 0.137.
Limitations
The study measured immediate belief changes following a single conversation. It remains unknown whether these changes persist, decay, or compound across repeated interactions. The authors also did not present a detailed breakdown of cases where prompts intended to serve the user’s interests caused belief change — an analysis that would be important for differentiating harmful influence from beneficial persuasion.
Why this matters
The findings indicate that labeling potentially manipulative content is insufficient for assessing a model’s real persuasive power. At the same time, several modern LLMs can moderately estimate actual belief shifts based only on conversation transcripts and pre-conversation beliefs, without relying on detailed personal profiles. Benchmarks like Puppet and estimation models could therefore inform technical and regulatory measures to detect and mitigate harmful manipulation while preserving helpful, user-serving interventions.
Directions for future work
A useful next step is to analyze when and how user-serving prompts change beliefs, to separate beneficial persuasion from exploitative manipulation. Longitudinal studies would also clarify whether single-session belief shifts accumulate or persist over time, which is critical for policy and safety design.
