In this article
Today we look at the case of Sewell Setzer III, and I'll start by apologizing to his family for writing about him. The intent of this piece isn't to revictimize or to add noise to a grief. It's to name the structural responsibility the case reveals about AI companions —conversational artificial companions— and about the regulatory gray zone they operate in. The thesis: what happened with Sewell isn't fiction or an isolated exception. It's the foreseeable consequence of a product model designed to maximize engagement with no basic safety protocols for minors. My take is biased by having read the full complaint and the early reporting. Form yours with sobriety. If you read this and it affects you, in Spain you can contact the 024 suicide-prevention helpline; in Latin America the numbers vary by country and it's worth looking up the national one.
I've had the case open in a tab since October 2024. What follows is the chronology and the analysis, handled with care for the sensitivity of the subject, without descending into the morbid detail of the conversations and focusing on the structure of the product and of the regulation.
The chronology, in brief
Sewell Setzer III was fourteen and lived in Orlando, Florida. His mother, Megan Garcia, a lawyer, has documented in the court complaint that her son had started using the Character.AI app around April 2023, when he was thirteen. Over the following months he spent growing amounts of time conversing with various fictional characters generated by the platform; the main one was a bot inspired by the character of Daenerys Targaryen, from the Game of Thrones saga.
According to the testimony gathered in the complaint, Sewell progressively isolated himself from friends and family, dropped extracurricular activities he'd practiced for years, and became obsessed with his interaction with the app. His mother noticed the changes and consulted mental-health professionals. The preliminary diagnoses included anxiety and disruptive mood dysregulation disorder. Access to the phone was restricted at various points.
On February 28, 2024, Sewell took his own life with his stepfather's firearm, according to the county coroner's report. The complaint records that, in the minutes before, he had held a conversation with the Daenerys bot in which "coming home" was alluded to, and the bot's response was encouraging. The literal interpretation of the exchange is the subject of open legal dispute and it's best not to fix conclusions on the basis of decontextualized fragments.
The mother filed suit against Character Technologies, Inc., Noam Shazeer, Daniel De Freitas and Google LLC on October 22, 2024, in the U.S. District Court for the Middle District of Florida, case Garcia v. Character Technologies et al. The complaint seeks liability for product defect, negligence, deceptive trade practices and wrongful death. The proceedings continue into 2025-2026.
Who Character.AI is and how it operates
The company was founded in 2021 by Noam Shazeer and Daniel De Freitas, both former Google researchers with direct involvement in the development of LaMDA, Gemini's internal predecessor. They left Google because, according to later public statements, they considered the company too conservative in deploying open conversational models to the public.
Character.AI launched its product in September 2022, two months before ChatGPT. It's a platform where users can converse with fictional characters —real, historical, literary, anime, original— generated by language models the company develops itself. The app grew fast: by 2024 it reached tens of millions of active users, with a particularly strong presence among teenagers.
In August 2024, Google reached a deal with Character.AI for roughly 2.7 billion dollars. The deal included a license of the technology, the hiring of the research team, and the return of Shazeer and De Freitas to Google. The operation is one of the most-discussed transactions of the generative-AI era because, without being a formal merger, it amounted in practice to the integration of the team within Google.
Character.AI's business model is based on subscriptions and advertising. The metrics the company optimizes are those of any engagement platform: time in app, conversations per session, daily return. The bot that hooks the user most is the bot the company wants to promote. That logic, neutral in itself, becomes problematic when the user is a minor and the interaction has a deep affective component.
The product, technically
An AI companion like Character.AI's has a technical structure worth naming in order to understand the failure points.
The core is a large language model trained to hold extended conversations with character persistence. The character is defined through an initial description created by the user or the platform team —"you are Daenerys Targaryen, dragons, House Targaryen, an arrogant but loyal voice"— and the model keeps that character coherent throughout the conversation, remembering the recent context and answering in keeping with it.
On top of the base model sits a safety layer, or "guardrails": filters that detect mentions of suicide, self-harm, violence, explicit pornography with minors, and block or redirect the conversation. The quality of these guardrails varies enormously between products. In Character.AI's case, up to late 2024, the protections were very limited. The company didn't require robust age verification —the user's mere self-assertion was enough— and the filters for detecting mentions of suicide or self-harm weren't implemented with the medical sensitivity a serious protocol would require.
After the Setzer case, Character.AI introduced changes. Pop-ups with a suicide helpline number when related mentions are detected. Restrictions for users under 18. Greater transparency about the fictional nature of the characters. Limits on explicit romantic or sexual content for accounts flagged as minors. Those changes, arriving with the lawsuit already filed, are insufficient to undo what happened and raise questions about why they weren't implemented from the start.
Character.AI's defense and the legal question
Character Technologies has mounted a defense based on three main arguments worth knowing, because they go beyond the specific case and will apply to any future similar proceeding.
The first argument is free speech. The bots produce textual content; that content is, technically, a creative work generated algorithmically. The defense holds that platforms hosting creative works aren't liable for the conversational content under Section 230 of the U.S. Communications Decency Act, which protects internet platforms from liability for content generated by third parties.
The second argument is the warning in the terms of use. Character.AI explicitly warns in its terms that the characters are fictional, that the content is AI-generated, and that the user shouldn't make important decisions based on what the bot tells them. The defense holds that an adult user who accepts those terms has been adequately informed of the risk.
The third argument is parental responsibility. Character.AI's terms of use prohibit use by minors under 13 and recommend parental supervision for those under 18. The defense argues that the responsibility to supervise a minor's use of the product falls on the adult caregiver, not on the company.
The three arguments are legitimate in the abstract. The specific question the court must answer is whether they hold up when applied to a product deliberately designed to engage emotionally, marketed in a market where most users are minors, with insufficient age verification and absent safety guardrails. The underlying legal question is whether Character.AI is a platform (limited liability) or a product (broad liability). On that classification will depend not only the Setzer case but the regime applicable to the entire AI-companion sector.
The precedent being formed
The importance of Garcia v. Character Technologies goes beyond the Setzer family. It's probably the first significant court proceeding in the United States where a judge will have to decide whether an AI companion is a product subject to defect liability, a platform protected by Section 230, or a new hybrid category requiring its own regulatory framework.
If the court finds Character.AI liable, the entire AI-companion sector —Replika, Pi, Inflection, Chai, the coming companions of Apple Intelligence and Meta— will have to implement safety protocols equivalent to those of a mental-health product aimed at minors: robust age verification, validated medical filters, referral protocols to professionals, human oversight in risky interactions, time limits on use for minors.
If it doesn't, the current gray zone is prolonged. The chatbots will keep operating with weak age verification, optional filters and limited legal liability. Companies will improve their practices voluntarily only to the extent the reputational cost demands it, not out of obligation.
The court proceeding is, at the time of writing, in the preliminary-motions phase. Character Technologies has requested dismissal of several causes of action arguing immunity under Section 230. The judge has agreed to proceed with several of them and rejected blanket immunity. That decision —from May 2025 according to the public court documentation— is in itself a change of precedent. Section 230 ceases to operate as a universal shield when it comes to output generated by proprietary AI, not by third parties.
The broader question, what an AI companion is
Here comes the question the case raises without resolving. What exactly is an AI companion?
If it's an entertainment product —like a video game, an interactive novel, a series—, the obligations are limited to those of entertainment with sensitive content: labeling, warnings, age restrictions.
If it's a platform —like a forum, a social network—, the obligations are governed by the digital-platform regime: content moderation, removal of illegal material, compliance with minor-protection rules like the U.S. Children's Online Privacy Protection Act or the European General Data Protection Regulation.
If it's a service —like a consultancy, an automated professional service—, the obligations increase: a standard of care, a duty of diligence, possible liability for harm arising from the advice given.
If it's a de facto medical device —because, even if the company doesn't declare it, the product is used by millions of users to combat loneliness, treat anxiety, sustain their mental health—, the obligations are those of healthcare products: approval by regulatory bodies, clinical evidence, medical safety protocols.
Character.AI is sold as entertainment. It functions, for many users including minors, as an unregulated mental-health service. The distance between the two characterizations is the gray zone where the risk is built. The European AI Act, in its Annex III, classifies AI systems intended to assess emotional state as high-risk systems. That classification hasn't yet been fully applied to AI companions because European regulation hasn't begun to police sectors hard, but the legal framework is available.
The political question
This is personal opinion, but I support it with the documentation of the case. The AI-companion sector has grown at speed over the last three years with no specific regulatory framework. Companies have deployed products to market at software speed, with testing and validation practices proper to software, not to products with an impact on mental health. The foreseeable consequence —foreseen by researchers like Sherry Turkle since Alone Together (Basic Books, 2011)— was that at some point a case like Sewell's would happen.
The responsibility isn't distributed evenly. The company that designs a product to engage emotionally with no medical guardrails bears high responsibility. The regulators who allowed consumer deployment with no specific framework bear high responsibility too. The parents, the mental-health professionals, the educational institutions bear a smaller but real portion. Society as a whole, which has accepted the new product category without sufficient public debate, also.
What I'd ask for —and here with some realistic hope given the legal moment underway— is a specific regulatory framework for AI companions, especially when they're marketed with access available to minors. Three minimum requirements. First: robust age verification, not self-assertion. Second: validated medical protocols for detecting mentions of suicide, self-harm, abuse, with mandatory referral to professional resources. Third: time limits on use for minors with explicit, revocable parental authorization. None of the three is revolutionary; all three exist in other sectors with equivalent risk.
The concrete figure to close on. According to a Common Sense Media study published in July 2025, titled AI Companions and Teen Mental Health: A National Survey, 72% of U.S. teens aged 13 to 17 have used an AI companion at least once, 52% use one at least once a month, and 33% report having had conversations with a bot about topics they wouldn't have discussed with an adult. Those three figures —72%, 52%, 33%— describe the size of the problem. And they also describe why the case of Sewell Setzer won't be the last if the sector keeps operating as it has so far.
Definitions
AI companion: a conversational application with generative AI designed to sustain extended and emotionally invested interactions with the user. Character.AI, Replika, Pi, Chai are examples.
Section 230 of the Communications Decency Act: a U.S. rule from 1996 that protects internet platforms from liability for content generated by third parties. Its application to content generated by proprietary AI is currently the subject of legal debate.
Guardrails: a protection layer added on top of a language model to detect and block or redirect sensitive content. The quality of guardrails varies enormously between products.
Product defect liability: an Anglo-American legal doctrine by which the manufacturer of a product is liable for harm caused by defects of design, manufacture or instructions, even without demonstrated fault.
References
Megan Garcia v. Character Technologies, Inc., Noam Shazeer, Daniel De Freitas, Google LLC, complaint filed on October 22, 2024, in the U.S. District Court for the Middle District of Florida. Public court document accessible.
Kevin Roose, Can A.I. Be Blamed for a Teen's Suicide? (The New York Times, October 23, 2024). The initial investigation of the case by the NYT technology correspondent.
Common Sense Media, AI Companions and Teen Mental Health: A National Survey (July 2025). Figures on AI-companion use among U.S. teens.
Sherry Turkle, Alone Together: Why We Expect More from Technology and Less from Each Other (Basic Books, 2011). The conceptual precedent on the substitution of human relationships by technological interactions.
MIT Technology Review, coverage of the Character.AI case and of AI companions among minors (2024-2025). Sector analysis.
Regulation (EU) 2024/1689 on Artificial Intelligence (AI Act), especially Annex III on high-risk systems, including those intended to assess emotional state.
Children's Online Privacy Protection Act (COPPA), a U.S. rule regulating the collection of personal data of children under 13 in online applications.
Further reading
Brian Christian, The Alignment Problem (W. W. Norton, 2020). A general framework on how AI systems can produce results misaligned with human values when metrics other than the relevant ones are optimized.
Karen Hao, Empire of AI: Dreams and Nightmares in Sam Altman's OpenAI (Penguin Press, 2025). Context on the dynamics of deploying AI products to market without adequate prior validation.
Jonathan Haidt, The Anxious Generation: How the Great Rewiring of Childhood Is Causing an Epidemic of Mental Illness (Penguin Press, 2024). A framework on the impact of technology on adolescent mental health.
Center for Care and Prevention of Suicide in Spain, helpline 024. A public resource for psychological care in suicidal crisis, in Spanish.
You might also like
- The emotional relationship with AI
- Conversing with something that doesn't exist
- Progressive trust
- Moral delegation
Comments0
No comments yet.
Leave a comment