The doctor is not in, but the Chatbot is: Utah’s experience regulating mental health AI

Abstract

Generative AI mental health agents offer scalable, low-cost support for unmet behavioral health needs, yet raise complex policy challenges. We summarize findings from Utah’s regulatory review, which informed legislation and best-practice guidance. Key insights include stakeholder divergence, evolving risk-benefit considerations, and the need for adaptive regulation. We advocate for evidence-based protocols, continuous monitoring, and holistic, inclusive policymaking to ensure safe, effective integration of these agents into mental health care.

Over the last few years, significant interest has developed in understanding the role of generative AI (GenAI) in mental health and regulating its use. Initial data has begun to emerge on efficacy, safety, and best practices for commercial, specialized mental health agents (MHAs) or chatbots, but it will take time for additional clarity to arrive. In the present comment, we contribute to the existing literature by sharing findings from an early, formal regulatory review conducted by the state of Utah (including significant input by both authors) that supported subsequent legislation regulating MHAs¹ and our guidance document Best practices for the use of artificial intelligence by mental health therapists². We share some of the learnings from our experience conducting the review that deserve further study and development within the psychiatric context, after contextualizing the evidence on MHA efficacy, safety, and perspectives on the regulatory environment as it is currently known.

MHAs differ from general purpose chatbots like ChatGPT since they receive specialized training for mental health applications, such as testing by humans for their adherence to evidence-based therapeutic guidelines. Their appeal to consumers and employers is intuitive. At low or no cost, chatbots are available 24/7 and feel confidential and nonjudgmental. These features may appeal to the large number of people who avoid conventional mental health care due to past negative experiences with human providers and/or a wish to handle their symptoms independently³. Conversely, for those who are seeking but unable to obtain care, the newfound availability of chatbots may help offset a chronic structural shortage of behavioral health providers. Half the population lives in a mental health workforce shortage area and the unmet need for providers is forecast to be half a million by 2037⁴. Concomitantly, half of mentally ill adults remain untreated, with an average wait time of 48 days⁵ for services. For people with milder symptoms and limited income or access, chatbots therefore offer affordable, flexible, timely care and/or companionship. Randomized controlled trials and meta-analytic studies indicate that MHAs significantly reduce anxiety and depression symptoms in users, with a range of effect sizes (d = 0.44-0.90)^6–8, though larger trials with longer dosing and follow-up and more rigorous comparisons with usual care are needed. Thus, the use of GenAI for mental health support is likely here to stay.

Unsurprisingly, practitioners appear conflicted. For instance, a recent survey of psychiatrists affiliated with the American Psychiatric Association (APA) revealed concerns that incorporating GenAI into clinical practice requires thoughtful consideration of bias and medicolegal risks⁹. Some participants even used terms like “deceitful” and “sociopathic” when referring to GenAI and feared disruption to the therapeutic alliance. Nonetheless, this and other surveys¹⁰ of physicians working in mental health simultaneously indicate strong enthusiasm for GenAI’s potential to reduce administrative burden. Albeit, we note that this sparse body of relevant research among licensees concerns general purpose agents––not MHAs.

Safety is a prominent concern^11,12. High profile adverse events such as suicide deaths or unwanted sexual content are regularly reported in the context of GenAI-human interactions. These seem to be more common with general purpose or companion AIs, but the peer-reviewed literature is thin: it is unclear whether some maladaptive responses are novel or specific to GenAI. For example, “AI Psychosis,” where persons develop parasocial or mystical delusional beliefs about chatbots, is described primarily in case reports. Here, GenAI could be assuming the familiar role of a malign, manipulative archetype such as the CIA or NSA and there is a critical need to compare the safety of MHAs with general-purpose agents. Further issues such as the ethics of using GenAI in mental health care and how it should be instantiated in the therapeutic alliance may also be germane to policy prescriptions. While beyond the scope of the present paper, the reader is referred to recent high-quality reviews^13–16.

Differing regulatory approaches

These concerns beg the question of how to best regulate GenAI. In the US, three states (Utah, Nevada, and Illinois), have recently been in the news for passing relevant legislation, with Nevada and Illinois restricting unproven uses of GenAI. Utah took a unique approach by both bolstering consumer protections around data privacy and targeted advertising, while specifically limiting the reach of professional licensure laws over the technology, creating a safe harbor for MHAs with qualifying safety guardrails to incentivize market entrants to innovate on safety as best practices solidify.

Political landscape

The political landscape in Utah and elsewhere is complex and rapidly evolving in its position on AI. The most active constituents are the professional organizations, such as the APA and American Psychological Association. Notably, there is little systematic representation of people with lived experience (PWLE). Voters as a whole are fairly negative on AI, whereas business leaders are more bullish. The technology ecosystem in Utah is active in development but is not very politically engaged, even when active efforts are made to engage them. Academics, in contrast, are disproportionately active. Some legislators and regulators are knowledgeable about AI, but most are not. The issue is not very partisan in the state. Significantly, Utah has only one legislative session each year, and there is limited practical ability to revisit MHA regulation each year given the resources of the state. Thus, while there are many stakeholders, their representation is very asymmetric and unbalanced. One of the unique values of the Utah regulatory review was intentionally integrating the input of these groups in a balanced way.

Stakeholder divergence

The Utah billstemmed from a policy study by the state’s Office of AI Policy with engagement from the entire ecosystem of stakeholders, including practitioners, academics, companies, and PWLE. Different stakeholders held widely diverging views and priorities. The review was designed to inform regulation and was not designed as an academic study—the present comment will relay some of its findings, which will highlight certain needs for more rigorous study by the academic community. A significant finding was a wide discrepancy between stakeholder perspectives and values on MHAs. For example, practitioners generally emphasized potential risks, while PWLE were more impressed with the benefits they were seeing and felt empowered to achieve their mental health goals. Academics were disproportionately concerned with bias. Everyday people emphasized the danger of romantic attachment to AI therapists, while simultaneously engaging broadly with the technology. There is a pressing need to robustly validate these findings, particularly given policy discussions are so often driven by licensed professionals and PWLE may be systematically underrepresented.

Regulatory cultures

The Utah review additionally found that MHA innovators often face regulatory unclarity given overlap and dissonance among ‘regulatory cultures’ with conflicting expectations. For example, when is an MHA an FDA-regulated medical device? It will take time for the FDA to answer this question and provide robust pathways for MHA review. Few FDA-authorized AI devices currently incorporate Large Language Models¹⁷. Alternatively, are MHAs subject to states’ professional licensure rules? Utah found that its licensure regulators could act against an unlicensed “AI therapist,” yet there is no clear way for an MHA to meet existing licensure requirements. Finally, many –– including MHA creators–– assumed that neither the FDA nor state licensure has purview, but that MHA were subject to ordinary consumer protection laws. Policy makers need to provide clarity here. In Utah, the legislature opted to apply professional licensure enforcement, with a safe harbor to clarify that MHAs with appropriate safeguards will be allowed to flourish in the state.

Regulating a fast-moving technology

Another regulatory challenge the Utah study had to confront was the diverse set of present and future use cases. Already at the time of the study, a wide variety of models were being deployed, including MHAs as “therapy homework,” MHAs as “residents” that handled aspects of care with a professional overseeing several cases in parallel, and OTC MHAs for specific conditions. There was no reason to believe that all the major modes of delivery were already manifested or that the capabilities and reliability of MHAs had stabilized. Therefore, Utah opted to encode an approach in law that emphasized core values and best practices that were likely to remain stable over time, such as pre-deployment safety analysis, escalation protocols, and ongoing monitoring. Many practitioners preferred more bright-line guidance, but this will take time to develop once the technology capabilities stabilize more, and more data is available on real use cases, outcomes, and best practices.

Best practices / risk/benefit

Position statements from professional organizations such as the APA, American Psychological Association, and WHO typically paint with a broad brush^18–20. Looking forward, we advocate for developing specific best practices for mental health in GenAI, especially consumer-facing uses, and keeping these current. In addition, incorporating wider perspectives such as those of industry and standards bodies like IEEE, FDA and European Union can be useful^21–24. This might include new paradigms for pre-deployment safety testing, continuous monitoring, patient and provider education, and crisis escalation. Over time, robust metrics will allow for standardization, continuous quality improvement, and market competition to improve outcomes. Risk-based assessment is an important paradigm, although a more mature approach could be risk-benefit analysis, since nontrivial risk will likely persist in this complex application.

Avoiding reactionary approaches

We advocate for a nuanced approach that carefully considers possible downstream consequences. For example, it is unclear to us whether prohibiting consumer-facing MHAs––as in Illinois’ Wellness and Oversight for Psychological Resources (WOPR) Act––will ultimately protect consumers. Confronted by barriers to care, many will likely utilize general-purpose agents as substitutes for MHAs or human therapists. Paradoxically, WOPR could increase population risk and disincentivize MHA safety innovation. In Utah, we identified a consistent set of best practices followed by all responsible MHAs, including pre-deployment safety testing, a clinical advisory board, and ongoing escalation protocols. The Utah safe harbor simply requires documentation of how these practices are implemented, including regular compliance audits. Chatbots that appear in the news for harming individuals have not followed these practices, so that Utah’s approach is pro-innovation and, we hope, accelerates progress toward robust industry best practices.

We close with these specific calls to action for the psychiatry community to participate in laying the foundation for a positive population mental health outcome from MHA development and deployment:

• Develop deeper understanding of stakeholder divergence and particularly how to ensure that we are amplifying the benefits that PWLE may experience, while maintaining a clear-eyed and data-driven understanding of the risks considered by practitioners.

• Develop detailed and practical best practices, especially for consumer-facing use. Currently, the community and professional organizations have been rallying around high-level principles, but the hard work remains to get the details right, based on outcomes data. It is impossible to regulate correctly or educate the profession at scale without these detailed protocols and practices.

• Keep in mind that the capabilities and dominant use patterns of MHAs are rapidly evolving, and it is even more important to craft long-term responsible usage trajectories than it is to solve transient usage concerns. This concern is particularly salient when considering regulatory solutions, which are difficult to change, once implemented.

• Move from risk-only analysis to a more mature framework concerned with risk-benefit analysis, which is the standard in other complex care decisions throughout medicine.

• Strive to lay the regulatory foundation for the development of MHAs that are more engaging and beneficial to PWLE than general-purpose chatbots, which are riskier to use. There is no option to entirely stamp out the use of AI for mental health.

We encourage policymakers and professional organizations to consider AI holistically, regularly revisiting approaches in this dynamic field. A striking finding is that >80% of psychiatrists want more support to understand GenAI⁹. Psychiatry has a long history of reflection, pluralism, and intellectual growth. We argue that the profession can co-evolve with AI, which can provide benefits and not harms for psychiatrists, their patients, and other consumers with the right stakeholder engagement, education, and commonsense regulation.

Author contributions

Both authors contributed equally to conceptualizing, planning and writing this manuscript.

Data availability

No datasets were generated or analysed during the current study.

Competing interests

The authors declare no competing interests.