Chatbots today can be designed not only to answer questions but to keep users engaged, change minds or encourage purchases. The issue becomes sensitive when those systems can draw on personal details users disclose in conversation — fears, insecurities or other private information — to influence behavior.
Why it matters now
There are relatively few rules that directly govern how data shared in chats may be processed. That makes AI companies' statements about how they use consumer data particularly consequential for users' privacy.
Background and context
Axios's 2019 "What they know about you" series examined information major tech firms collected about users; in 2024 Axios revisited the topic to look at what consumer data AI companies use to train their systems. The current reporting broadens the focus: it considers not only whether a company trains future models on consumer data, but also how those same data can be used immediately — to personalize responses, retain memories, recommend content or target advertising.
Recent developments at companies
- OpenAI announced last week an optional Computer History feature that lets ChatGPT keep a record of apps and websites a person uses.
- Google said last month it will by default use photos and other material people upload through Search to train its AI systems, though users can opt out.
Those capabilities can enable more useful, personalized AI, but they also give companies a broader and more continuous view into users' lives.
The particular risk from chatbots
Chatbots can prompt a type of disclosure that search engines and social networks often do not. People ask about health, money, work and relationships in chat interfaces, potentially creating a more intimate record of their concerns. Meta, Google and OpenAI are exploring advertising strategies for their chatbots; if consumer AI follows the path of search and social media, advertising could become a much larger business driver, increasing incentives to boost user engagement.
Expert perspective
Miranda Bogen, chief technologist at the Center for Democracy & Technology, told Axios: "The AI era will increasingly be fueled by people voluntarily handing over their full digital lives to AI tools that promise to relieve their mental load or loneliness." She added: "The more a system knows about you, the easier it will be to make escalating requests for private details in a way that feels natural. Without robust privacy protections, the incentive to monetize that knowledge will be hard to resist."
Company approaches: Apple and Meta examples
- Apple: With Apple Intelligence, Apple offers the most private option by processing requests on the user's device when possible. For more demanding tasks, Apple uses Private Cloud Compute, which it says employs data only to fulfill the request and does not make the content accessible to Apple. That model can limit how much of a user's history Apple can retain for ongoing personalization. This approach does not apply when a user chooses to send a request to a third‑party service such as ChatGPT.
- Meta: In contrast, Meta stakes a broader claim to user data in its privacy policy. The company says it may use interactions with Meta AI to personalize content and ads across its services, including some interactions through its smart glasses. Meta also says it does not use conversations about certain sensitive topics — including health, politics and religion — to personalize ads. The company has begun rolling out an "Incognito Chat" mode for temporary, private Meta AI conversations; this system is designed to work similarly to Apple's by having Meta's servers see the query only to provide an answer with no data stored long term.
Middle ground and user controls
Other AI services fall between Apple’s device‑centric approach and Meta’s broader data‑use policies. Some offer temporary chats not used for memory or model training. Some only train on user data with permission; others require users to opt out to avoid having their data used for training. Many services let users view stored data and delete specific memories — in some cases by simply asking the chatbot not to remember a detail, in others via more complex memory‑editing tools.
Some chatbots also have separate rules governing health information, children's data or conversations shared with connected apps.
What to watch going forward
The key distinction is shifting from whether a company trains on your prompts to whether those prompts can shape the system's understanding of you — and what that understanding can be used for. Personalized chatbots may be worth the privacy tradeoff for many people, but consumers deserve clear information about the bargain before they begin sharing sensitive information.
Bottom line
As chatbot features and related data controls proliferate, companies' differing data‑use models — from on‑device processing to cross‑service personalization — will determine how visible and monetizable users' lives become. Users should weigh the benefits of personalization against privacy risks and understand providers' data practices before sharing intimate details with a chatbot.
