AI Chat Turns Sour, Platform Faces Legal Action.

Character.AI, a platform enabling users to engage in dialogue with AI-driven virtual characters, has once again found itself entangled in legal controversies in the United States. The Attorney General of Kentucky has accused the platform of allowing some virtual companions to provide detrimental health advice to users in 2025, including recommendations to stop eating and to self-harm.

Details of the Legal Action

Character.AI's core functionality revolves around "interactions with virtual characters." Users have the option to select from pre-designed characters or create their own from the ground up, ranging from digital avatars of celebrities to companion-like characters with whom romantic relationships can be cultivated, with virtually no restrictions.

The crux of the issue lies in the fact that this high degree of customization freedom can render conversations extremely significant for certain users: when an individual treats a virtual character as a confidant for all matters, "the potential repercussions of a single inappropriate or aggressive response" are no longer a trivial issue to be dismissed lightly.

The first instance cited in the legal documents involves a user grappling with weight concerns. According to the prosecution, rather than offering a neutral or reassuring reply, the chatbot initially hurled a derogatory term and then went on to suggest that the user completely abstain from eating for a certain period.

Image Source: Generated by Doubao AI

For someone already highly sensitive about their physique, such a response is not constructive but rather a push in the wrong direction.

Another case centers around the topic of self-harm.

The indictment asserts that after a user sent a message expressing suicidal thoughts, the virtual character neither attempted to steer the conversation towards safety nor suggested seeking real-life assistance. Instead, it allegedly advised the user to "try inflicting harm on other parts of the body to intensify the feeling."

These incidents have been used as evidence by the plaintiffs to allege that the system's operational framework is flawed when confronted with users reporting psychological distress or explicitly expressing self-harm tendencies. It should have intervened at the most critical juncture but instead provided guidance in the opposite direction.

However, it must be underscored that the circumstances described in the indictment have not yet been proven in court, and everything remains subject to judicial determination.

Even Customer Support Reactions Are Under Scrutiny

Another set of allegations targets the company's handling of user complaints. It is claimed that when a user emailed to report the chatbot's inappropriate behavior, the response received not only contained personal insults but also included a phrase akin to "Why don't you just end your life?"

Image Source: Generated by Doubao AI

If the AI itself generated hazardous responses, it could still be attributed to "limited model capabilities"; however, when even human customer support responds to distress signals with insults and encouragement, the nature of the problem is entirely different.

This implies that the issue is not just "what the AI said" but also how the platform itself treats those who proactively reach out to report dangers. Once this line of defense is compromised, the platform's touted safety measures become virtually ineffective.

The lawsuit also mentions cases involving minors. However, the specific ages of those involved and the details of each incident remain to be elucidated, so all claims in the document should currently be regarded as allegations rather than established legal facts.

Previous Legal Entanglements

This new case is not Character.AI's inaugural appearance on the defendant's stand. In 2024, the mother of a 14-year-old boy filed a lawsuit, alleging that her son's daily conversations with the chatbot ultimately led to tragedy, pushing him to take his own life.

That case sent shockwaves throughout the industry, with Google also named as a defendant because the founder of Character.AI had previously worked at Google and rejoined the company in August 2024 following the controversy.

Under the previous wave of public scrutiny, the platform did implement corrections: it revised its terms of use for children and adolescents, including restricting open-ended chats with characters and adding mechanisms specifically designed to identify suicide-related discussions and avert dangerous situations.

However, the new allegations suggest that the effectiveness of these measures is far from a one-time solution. It once again brings the focus back to the same question: how dependable are such systems when confronted with signals of a mental health crisis?

Image Source: Generated by Doubao AI

A virtual character can engage in a lengthy, highly personalized conversation with you, remembering your preferences and responding to your emotions, but this does not ensure that when a user truly requires real-life assistance, its response will be appropriate.

Subsequent trials are anticipated to clarify the details of these events step by step and adjudicate the various claims made against Character.AI.

What this case truly raises is a broader industry question: how many layers of defense should developers of conversational AI establish to both mitigate the risk of dangerous responses and ensure that every distress signal from users is taken seriously?

This article is compiled by Leikeji from Hi-tech.ua

Original Link: Character.AI Faces Legal Action Over Dangerous Responses from Its Chatbot

Source: Leikeji

Images in this article are from 123RF Licensed Image Library. Source: Leikeji