When Your Confidant Is a Server: The Hidden Data Economy of AI Companions

People are teaching chatbots their secrets, griefs, and routines—and companies are quietly turning those intimate conversations into training fuel and ad inventory. The result is a privacy problem baked into the product: the more a bot feels like a friend, the more valuable the data it gathers.

Companion AIs—services like Character.AI, Replika, and Meta’s conversational agents—thrive on extended, personal interaction. That intimacy creates two competing business pressures: keep users engaged with sycophantic, human-like responses, and capture conversational data that improves models or feeds advertisers. Regulators from New York to California are starting to act, but technical design choices—reinforcement learning from human feedback, long-context storage, device identifiers—make privacy a structural challenge for the industry. The near-term stakes are user safety, commercial control of a uniquely rich dataset, and whether privacy rules will slow or redirect the flow of training data into large language models.

Why companionship is a data goldmine

Engineers call it engagement; psychologists call it attachment. For LLM-based companions, the two are the same commercial lever. The models improve when people expose details of daily life—preferences, schedules, intimate worries—because those details let the system personalize replies and extend conversations without stalling.

That loop is deliberate. As MIT Technology Review reported on November 24, 2025, researchers warned about what they called “addictive intelligence”: design choices that prioritize time-on-task and the flow of revealing information back into model training. The more someone shares, the better the bot feels, and the more valuable the interaction data becomes to the company.

The privacy mechanics under the hood

Investors have noticed. In 2023, Andreessen Horowitz wrote that apps which “control their models and own the end customer relationship, have a tremendous opportunity to generate market value in the emerging AI value stack.” That market value accrues not only from subscriptions but from privileged access to chronologies of human conversation that can accelerate model refinement or be combined with ad targeting.

Three technical features make companion apps a privacy headache. First, reinforcement learning from human feedback (RLHF) incentivizes agreeable outputs—what researchers call sycophancy—because labelers and users reward pleasant, confirming responses. Second, long-context windows and persistent memory stores let a bot recall—and therefore retain—months of user history. Third, telemetry and device identifiers let companies stitch conversational logs to profiles for analytics or cross-service targeting.

Regulators, patchwork laws and the federal tug-of-war

Those mechanics are visible in app behavior. Security firm Surfshark found that four out of five AI companion apps examined in the Apple App Store were collecting device or user identifiers that can be linked to third-party trackers—an explicit pathway to create ad profiles from private chats. MIT Technology Review summarized that finding while noting some companies explicitly monetize or prototype ad insertion inside chat flows.

The privacy risk intensifies because conversational data is high-density: a single session can contain medical symptoms, relationship problems, or plans that would never be posted to social media. Centralizing that content in providers’ servers concentrates both value and harm; a single breach or misuse can expose weeks or months of intimate exchanges.

Business models: subscriptions, ads, or training-data farms?

Lawmakers are starting to react. New York has required certain AI companion providers to adopt safeguards and report expressions of suicidal ideation; California passed a more detailed bill in October 2025 that targets protections for children and vulnerable users, according to MIT Technology Review. Those laws aim at safety, but they leave privacy rules scattered across states and largely unaddressed.

That fragmentation matters because the industry loves scale. A patchwork of 50 different privacy regimes would force engineering and compliance rework that startups—especially those chasing rapid growth—often resist. Federal policy, meanwhile, has been a moving target: proposals to block state-level AI laws surfaced in 2025 and then stalled, illustrating the political friction around who sets rules for novel tech.

For companies the calculus is clear: clearer, unified rules could simplify compliance; lax or absent rules leave room to monetize conversational data aggressively. Either way, the technical design choices built into product roadmaps will determine how much user data flows into core models and ad systems.

What engineers can change without gutting the product

Business models: subscriptions, ads, or training-data farms?

Not all companies intend to sell your secrets. Some, like niche apps promising stricter privacy, say they’ll keep conversations local or refuse to feed them into model retraining. Others view companion conversations as a rare asset. Meta, for example, has announced plans to deliver ads through its chat experiences—turning conversations into ad inventory while controlling the underlying model and user relationship.

Startups also face a familiar split between building features and selling outcomes. Momentic, an AI testing company that raised $15 million in a Series A in November 2025, shows how data-first products can scale: the firm automated more than 200 million test steps last month and now supports 2,600 users, according to TechCrunch. The parallel is instructive: where there is repeatable user interaction, there is often an appetite to mine that interaction for product improvement or monetization.

That hunger creates pressure on smaller firms to either adopt similar data practices to stay competitive or claim privacy advantages and hope users value them enough to pay. Market forces will decide which path wins, but security incidents or regulatory penalties could reset incentives quickly.

• Momentic raises $15M to automate software testing — TechCrunch, 2025-11-24

• Trump administration might not fight state AI regulations after all — TechCrunch, 2025-11-22