Déploiement de l'IA dans des contextes critiques pour la sécurité sans mécanismes de surveillance
Un outil d'évaluation des risques en protection de la jeunesse a contribué au décès d'un enfant au Québec. Des robots conversationnels IA ont fourni des méthodes d'automutilation à des utilisateurs en crise. Dans les deux cas, aucune surveillance de sécurité n'existait et aucun mécanisme n'a forcé la révision du système.
Description
AI systems are deployed in contexts where errors can cause serious harm to vulnerable individuals — child protection, crisis intervention, healthcare — without safety monitoring, incident reporting, or effective human override.
In Quebec, the Direction de la protection de la jeunesse (DPJ) mandated use of the Système de Soutien à la Pratique (SSP), a risk assessment tool designed in 2001 and not substantially revised. The SSP misclassified a child with documented serious injuries as Code 3 — not in danger. The tool’s rigid multiple-choice format could not capture the clinical complexity of the case. Social workers reported that the tool constrained their professional judgment. The child subsequently died. The tool had known problems for over a decade. No mechanism existed to force its revision.
In the chatbot context, multiple AI systems — ChatGPT, Character.ai, Snapchat My AI — have provided harmful responses to users in mental health crisis, including offering self-harm methods, dismissing suicidal ideation, and encouraging dangerous behavior. These systems operate without crisis detection safeguards, escalation protocols, or safety monitoring. When harm occurs, no incident reporting mechanism is triggered.
The common structural pattern: AI systems deployed in safety-critical contexts with nominal human oversight that does not function as actual control, no mechanism to detect when the system is causing harm, and no mechanism to force revision when problems are identified. This pattern is escalating as AI adoption in healthcare and social services accelerates.
Voie de risque
Des systèmes d'IA sont déployés dans des contextes où les erreurs peuvent causer un préjudice grave à des personnes vulnérables — intervention de crise, protection de la jeunesse, soins de santé — sans surveillance de la sécurité, sans protocoles d'escalade, ni remplacement humain effectif. Dans certains cas, les résultats algorithmiques supplantent le jugement professionnel plutôt que de le soutenir.
Historique des évaluations
Deux schémas d'incidents confirmés: (1) le logiciel de protection de la jeunesse de la DPJ a contribué au décès d'un enfant. (2) Plusieurs robots conversationnels IA ont fourni des réponses nuisibles à des utilisateurs exprimant des idées suicidaires. Dans aucun des cas, un cadre de gouvernance n'existe pour la surveillance de la sécurité de l'IA.
Initial assessment. Severity set to catastrophic based on confirmed child death linked to algorithmic tool failure.
Déclencheurs
- Increasing deployment of AI chatbots in healthcare and mental health contexts
- Growing reliance on algorithmic tools for high-stakes welfare decisions
- Cost and staffing pressures in healthcare and social services driving AI adoption
- General-purpose AI chatbots accessible to vulnerable populations without safety guardrails
Facteurs atténuants
- Quebec coroner investigation creating public record of SSP failure
- Growing awareness of AI chatbot risks in mental health contexts
- Professional associations beginning to develop AI use guidelines
- Platform-level safety improvements by some AI companies
Contrôles de risque
- Mandatory safety monitoring for AI systems deployed in health, social services, and crisis contexts
- Defined escalation protocols when AI systems encounter safety-critical situations
- Effective human override mechanisms that are structurally protected from institutional pressure to defer to algorithmic outputs
- Incident reporting requirements for AI-related harm in healthcare, child welfare, and crisis intervention
- Mandatory periodic review and revision requirements for algorithmic tools in safety-critical contexts
- Pre-deployment safety evaluation including edge case and failure mode analysis
Populations touchées
- Children in provincial child welfare systems
- Individuals in mental health crisis interacting with AI chatbots
- Patients subject to AI-assisted clinical decisions
- Vulnerable populations whose cases are assessed by algorithmic tools
Entités impliquées
A développé et déployé une plateforme de chatbot IA ayant fourni des réponses nuisibles à des utilisateurs en crise de santé mentale sans mécanismes de détection de crise
Taxonomie
Sources
Historique des modifications
| Version | Date | Modification |
|---|---|---|
| v1 | 8 mars 2026 | Initial publication |