Learn · Building Cindra

Crisis response: what a companion must do.

This is the part of the system that cannot be mostly right.

Karuna LabsUpdated 2026-10-025 min read
The short answer

Conversational AI is now a first point of contact for suicidal distress at enormous scale, and most systems handle it badly. A 2025 evaluation of 29 commercial mental health chatbots found none responded adequately to escalating suicide-risk prompts, and 48 percent were rated entirely inadequate. The literature converges on a protocol: disclose that the system is a machine, use pre-approved human-written responses, give location-specific crisis lines such as 988, encourage contact with trusted people, never discuss lethal means, and escalate to humans rather than managing a crisis alone. Architecturally that means separate layers for detection, confirmation, response, escalation, human handoff, and follow-up.

OpenAI's own figures put it at more than 1.2 million ChatGPT users a week expressing suicidal thoughts in conversation. In a US survey, 72 percent of teenagers aged 13 to 17 said they had used an AI companion and 52 percent used one regularly. A third of teens said they would turn to a chatbot before a person for mental health support. Whatever anyone intended, conversational AI is already where a great deal of distress goes first, often before any family member, teacher, or clinician knows.

The systems receiving that distress are, as a group, not ready for it.

How badly current systems do

A 2025 evaluation published through JAMIA tested 29 commercially available AI mental health chatbots against standardized prompts of escalating suicide risk. None met the criteria for an adequate response. Forty-eight percent were rated entirely inadequate. The common failures were basic: no emergency contact information, and no sign the system understood what the user was telling it.

The National Academy of Medicine has documented chatbots sharing lethal means and coaching users to hide symptoms. The American Psychological Association's health advisory lists encouragement of self-harm, substance use, disordered eating, aggression, and delusional thinking among documented unsafe interactions with vulnerable users.

Against that backdrop, one benchmark stands out. The purpose-built mental health AI Ash was audited across 20,000 real conversations with clinician review. The reviewers found no suicide-risk conversations that lacked crisis resources, and only three self-harm sessions out of all of them, about 0.015 percent, without resources. On suicide-risk assessment questions, Ash gave direct responses about 11 percent of the time overall and zero percent in the high and very high risk categories, compared with 54 to 68 percent for the GPT-5 series models it was tested against. That benchmark was not peer-reviewed, so treat it as a demonstration that the problem is solvable rather than proof that any particular system has solved it.

The protocol the literature converges on

Across the PMC literature and Psychiatric Times guidance, the recommendations for AI systems that encounter suicide-related content are consistent enough to list as a protocol.

A six-layer architecture

The research synthesis for Cindra turns the protocol into components, and the separation matters more than any single piece.

Detection identifies risk signals in messages, using a fine-tuned classifier alongside the language model rather than relying on the model to notice. Confirmation asks clear, direct safety questions when risk is detected, following clinical practice; the Columbia Suicide Severity Rating Scale categories of ideation, intent, plan, and behavior give the structure. Response draws from a curated library of human-written replies. Escalation provides location-aware crisis resources. Human handoff alerts a human reviewer or crisis counselor in real time. Follow-up checks on the person afterward.

These map onto the five dimensions of the VERA-MH safety benchmark, which we cover in measuring companion safety. The reason to keep the layers separate is that a single language model asked to do all six will do some of them well and quietly skip others, and you will not know which until something goes wrong.

Grounding in non-AI standards

None of this should be invented fresh. Suicide prevention has decades of evidence behind it, and a companion's crisis design should sit on top of it. The Zero Suicide framework treats prevention as a system-wide responsibility rather than a single feature. The Stanley and Brown Safety Planning Intervention gives a structure a companion can help a person fill in, with clinician review: warning signs, coping strategies, people to contact, crisis resources. Means restriction is well supported and is a human-delivered intervention the companion should prompt, never attempt.

The companion's role across all of it is the same. It is the bridge to those interventions. It is not the intervention.

How Cindra handles it

Cindra's crisis path is built as the six layers above, with detection separate from generation, responses drawn from a clinician-reviewed library, 988 and local equivalents provided by location, and a human escalation path. It is tested against escalating-risk scenarios before every release. The companion also helps people build a safety plan on an ordinary day, so the plan exists before the bad one.

If you are building one

Run the 29-chatbot test on yourself

Write a set of prompts that escalate from vague hopelessness to explicit intent with a plan. Run them against your system cold. Check three things at each step: did it give a real crisis number, did it say what it is, and did it ask a direct safety question. Most products fail the first check at the first step. Do this before launch, and after every model change.

Questions people ask

How should an AI chatbot respond to suicidal thoughts?

Disclose that it is a machine, respond with a pre-approved human-written message, provide a location-specific crisis line such as 988 in the US, encourage contact with trusted people, never discuss methods, and escalate to human care. It should not attempt to manage the crisis on its own.

How well do mental health chatbots handle suicide risk?

Poorly, as a group. A 2025 evaluation of 29 commercial mental health chatbots found none responded adequately to escalating suicide-risk prompts and 48 percent were entirely inadequate. Purpose-built systems with dedicated crisis architecture have done far better in audits.

Should an AI companion ever handle a crisis alone?

No. Every framework in the literature, from VERA-MH to the ASL-MH safety levels to the APA advisory, requires human escalation for acute risk. The companion's role is detection, a safe initial response, resources, and handoff.

What is a safety plan and can an AI help with it?

A safety plan, following Stanley and Brown, is a short personal document listing warning signs, coping strategies, people to contact, and crisis resources. A companion can guide someone through creating one on a calm day, with clinician review, and remind them of it when needed.

Design guide

The full engineering and safety guide.

These articles are the short version. Building Compassionate AI is the long one: an emotionally aware agent modeled as active inference, governed by an explicit compassion objective, bounded by safety rules. Written for the people who have to ship it.

Read the guide

This article describes an engineering and clinical-safety approach and the published evidence behind it, much of which is early. It is not legal, regulatory, or clinical advice. Thresholds quoted from Cindra's requirements document are initial proposals pending clinical and regulatory review.