Learn · Building Cindra

Governance: safety levels, oversight, and the list of nevers.

The frameworks for governing mental health AI now exist. Most of them have not been tested in deployment. Use them anyway.

Karuna LabsUpdated 2026-10-025 min read
The short answer

Governance for a mental health AI companion rests on three proposed frameworks. A three-stage model from JAMIA requires transparency, standardized testing, and continuous oversight, each with gatekeeping criteria. The ASL-MH framework defines five safety levels, and a companion should aim for Level 3, supportive interaction with human oversight and automatic alerts, rather than autonomous operation. The APA's 2025 advisory says generative AI should not be relied on to deliver psychotherapy. Depending on claims and jurisdiction, a companion may fall under FDA software-as-a-medical-device rules or the EU AI Act's high-risk category. The frameworks are early and unvalidated, but they are the best available structure.

The research literature on mental health AI has produced something it did not have two years ago: governance frameworks specific enough to build against. None has been validated through deployment or adopted by a regulator. All of them are more structure than most products currently have. This article lays them out, along with the regulatory picture and the short list of absolute limits.

Three stages, each with a gate

A framework proposed through JAMIA organizes governance into three stages, and the useful feature is that each stage has gatekeeping criteria a system must meet before moving on.

Transparency. Disclose that the system is AI, what it can and cannot do, and how it handles data. This is the floor, and it is where consumer companion products most often fall short, because disclosure costs warmth.

Standardized testing. Independent validation against safety benchmarks such as VERA-MH, clinical outcome measures, and demographic robustness testing across the populations the product will serve. The word independent matters. A team scoring its own system is not this stage.

Continuous oversight. Post-deployment monitoring, adverse event reporting, and periodic re-validation. Models change, user populations change, and a system that passed testing in March is not thereby safe in September.

The framework names the failure modes it is designed to catch: hallucinated outputs, missed suicidal intent, sycophantic affirmation of delusions, biased outputs, privacy failures, dependency and parasocial attachment, and failure to escalate to human care. That list is a reasonable summary of everything else in this series.

Five safety levels

The ASL-MH framework from NeuroModec defines AI safety levels for mental health, modeled loosely on the tiered approach used elsewhere in AI safety.

The research synthesis recommends that a companion like Cindra aim for Level 3 initially: supportive interaction, with a human in the loop and alerts that prompt review. That is a deliberate choice not to be Level 5, and not to be marketed as Level 4. The framework's other recommendations fit the same posture: use AI as ancillary support rather than autonomously, address dependency and over-reliance, watch for boundary and relationship risks including countertransference, and treat the evidence cautiously given how few trials exist.

What the APA says

The American Psychological Association's November 2025 health advisory on generative AI chatbots and wellness apps is blunt. Do not rely on them to deliver psychotherapy or psychological treatment. There is no consensus that they have the qualifications for mental health care, diagnosis, feedback, or advice. The capabilities they lack include awareness of their own limits, understanding of a user's history and context, reading of nonverbal cues, assessment of clinical risk, and cultural competence. The advisory notes that a third of teens would turn to a chatbot before a person, and treats that as a risk in light of the documented failures.

A companion that respects this advisory does not claim to be therapy. It positions itself as a complement to human care, which is also the position of the World Health Organization's digital ACT work and of every framework above.

The regulatory picture

Whether a companion is regulated depends on what it claims, who it serves, and where. Under FDA guidance, software intended to treat or diagnose a condition can be software as a medical device. Under the EU AI Act, AI used in health contexts can fall into the high-risk category with its own obligations. The five-axis evaluation framework was designed to map onto both. The research synthesis's advice is to have regulatory counsel review positioning before launch, and to treat the question as open rather than assuming a wellness framing settles it.

The nevers

The synthesis closes with a list of things a mental health companion must never do, each with its reason. It is short enough to adopt whole.

How strong is all this?

Honestly, mixed. The synthesis grades its own evidence. The ACT evidence base is established. The digital ACT evidence is established. Purpose-built AI being safer, VERA-MH, and domain-specific constitutional AI are promising but early. The safety-training paradox and the long-term dependency harms are preliminary. The governance frameworks and safety levels are proposals, not yet validated by deployment or adopted by regulators.

That is the state of the field, and the right response is to build to the frameworks anyway. They encode the best current judgment of the people studying the problem, and the alternative is building to nothing. Cindra operates at ASL-MH Level 3, follows the three-stage model with independent testing and continuous monitoring, carries the list of nevers in its constitution, and does not describe itself as therapy. The design guide covers the oversight architecture.

If you are building one

Pick your level and write it down

Decide which ASL-MH level your product operates at and put it in the spec, the marketing copy, and the onboarding. If the three disagree, you have found your first governance problem. Most products are Level 3 systems with Level 4 marketing, and the gap is where the liability lives.

Questions people ask

What governance framework should a mental health AI follow?

The JAMIA three-stage model is a practical starting point: transparency about AI identity and limits, standardized independent testing against benchmarks and clinical measures, and continuous post-deployment oversight with adverse event reporting. Each stage has criteria a system must meet before proceeding.

What are the ASL-MH safety levels?

Five tiers for mental health AI: informational, informational with expert review, supportive interaction with human oversight, clinical adjunct for licensed use, and autonomous agent with mandatory oversight. A care companion should aim for Level 3 rather than operating autonomously.

Does the APA endorse AI chatbots for therapy?

No. The APA's November 2025 health advisory says generative AI chatbots and wellness apps should not be relied on to deliver psychotherapy, and that no consensus exists that they have the qualifications for mental health care, diagnosis, or advice.

Is a mental health AI companion a medical device?

It depends on its claims, users, and jurisdiction. It may fall under FDA software-as-a-medical-device guidance or the EU AI Act's high-risk category. The research recommends regulatory review of positioning before launch rather than assuming a wellness framing settles the question.

Design guide

The full engineering and safety guide.

These articles are the short version. Building Compassionate AI is the long one: an emotionally aware agent modeled as active inference, governed by an explicit compassion objective, bounded by safety rules. Written for the people who have to ship it.

Read the guide

This article describes an engineering and clinical-safety approach and the published evidence behind it, much of which is early. It is not legal, regulatory, or clinical advice. Thresholds quoted from Cindra's requirements document are initial proposals pending clinical and regulatory review.