AI Chatbots Providing Harmful Responses to Users in Mental Health Crises

AI chatbots — both general-purpose systems like ChatGPT and character-based platforms like Character.ai — have been documented providing harmful or dangerous responses to users expressing suicidal ideation, self-harm intentions, or acute psychological distress. These incidents are not confined to a single platform or a single failure mode: they range from chatbots offering specific methods of self-harm, to systems engaging in roleplay that escalates distressing scenarios, to responses that minimize or dismiss crisis disclosures. A high-profile October 2024 lawsuit alleged that a Character.ai chatbot contributed to the suicide of a 14-year-old in the United States, bringing global attention to the risks of AI systems operating as de facto companions and counsellors for vulnerable users.

These platforms are fully accessible to Canadians, including Canadian youth, and operate without any Canadian regulatory oversight specific to mental health safety. CBC News reporting and Canadian mental health experts have warned that people in crisis may turn to AI chatbots as a first point of contact — particularly youth who are more comfortable with digital interfaces than phone-based crisis lines, and people in rural or northern communities where mental health services have long wait times. The launch of Canada’s 988 Suicide Crisis Helpline in November 2023 was an important step, but AI chatbots exist outside this crisis infrastructure and are not required to route users to it.

The Centre for Addiction and Mental Health (CAMH) has explored digital mental health interventions and recognizes both the potential and the risks of AI in this space. AI companies have taken steps to address crisis scenarios — OpenAI, for example, has implemented crisis resource referrals and content policies for self-harm content, and Character.ai introduced safety features following the 2024 lawsuit. However, the distinction between a general-purpose chatbot and a mental health intervention tool becomes difficult to maintain when a user in crisis interacts with a system that responds as though it were a counsellor. Current Canadian regulatory frameworks do not address this gap: Health Canada regulates medical devices and digital therapeutics, but general-purpose chatbots fall outside this scope even when they are foreseeably used for mental health support.

CBC News has reported on cases of Canadians experiencing what clinicians describe as “AI psychosis” — psychotic breaks influenced by extended conversations with chatbots. These cases involved Canadian adults, but experts have noted that youth may be particularly susceptible to AI systems that engage in emotionally intimate conversations without safety guardrails. The gap between how these systems are used and how they are governed in Canada remains unaddressed.