Artificial Intelligence can transform mental health, but not necessarily for the better

Abstract

Importance:

The potential of tools employing artificial intelligence (AI) to address the many challenges in delivery of mental health care has been widely discussed. However, the possible negative consequences of AI for such care have received less attention.

Observations:

Integrating AI with mental health care has the potential to expand access and improve quality of care. It may also contribute to improvements in diagnosis, risk stratification, and development of novel therapeutics. At the same time, availability of AI chatbots and stratification algorithms may diminish access to human-delivered care. Reliance on AI tools may have other unanticipated adverse consequences on clinical practice, including diminished human clinician skill. The probabilistic nature of many of these tools, including large language models, makes their capacity to cause harm difficult to determine.

Conclusions and Relevance:

The likely benefits of AI for psychiatric care delivery must be balanced against substantial risks. Strategies to mitigate this risk may require regulation to enhance transparency and systematically evaluate the impact of AI in practice, as well as clinician training to make optimal use of these emerging methods.

Introduction

Discussions of artificial intelligence in health invariably describe it as transformative. While much of this impact remains aspirational, accumulating evidence suggests the capacity of this technology to profoundly change how health care is delivered, at every level¹.

But transformative does not necessarily ensure that these changes will be for the better, particularly as they relate to mental health care. Despite advances in genomics and neuroscience that have impacted other areas of medicine, the care of people with psychiatric disorders has changed little. Those with these illnesses continue to face tremendous challenges: limited access to high-quality care; inconsistencies in diagnosis; variability in care; and inherent limitations of the treatments themselves. While AI is typically invoked as a strategy for addressing these limitations, such progress cannot be assumed – and cannot be assumed to be entirely beneficial. Alongside the potential benefits, this review therefore considers the risks that widespread adoption of AI might pose to mental health care, and suggests strategies that may help to mitigate them.

Benefits

Broadly speaking, application of AI could broaden access to care and contribute to more uniform care. Two applications illustrate such benefits. The first is application of language models to drive chatbots delivering evidence-based therapies². At present, access to cognitive-behavioral therapy, for example, can be challenging. There is also no assurance that a given provider is actually providing evidence-based care, and that the care is of sufficient quality. Incorporating chatbots could relieve some of this demand – for example, providing an initial introduction to CBT techniques that might be sufficient for some individuals – and allow the limited supply of CBT therapists to focus on more severely affected patients. A recent pilot randomized wait-list controlled study suggested benefit in a heterogeneous group of disorders³.

Automating evidence-based therapies could also help to accelerate treatment innovation, making it simpler to disseminate and study potential treatments. In the same way websites routinely apply A/B testing to examine strategies to increase engagement, CBT apps could (with consent) rapidly investigate whether a new module or method improves outcomes. This application of AI could help to accelerate validation of a burgeoning set of targeted CBT interventions that acknowledge the heterogeneity of even individual disorders and subpopulations⁴.

The second application is clinical decision support. Initial work in mood disorders indicates that large language models, integrated with treatment guidelines using an approach called retrieval augmented generation, can identify appropriate next-step treatments at rates similar to clinical experts, and exceeding community clinicians⁵. As important, rates of identifying poor next-step treatments are significantly lower than community clinicians, suggesting an opportunity to avoid adverse outcomes from inappropriate medication selection. In principle, these strategies should allow any clinician to deliver expert-level care in a primary care setting, limiting the need for psychopharmacologic referral.

While clinical interventions are more investigated, in the near term, administrative applications may have the greatest impact by streamlining care⁶. Such applications include ambient scribes that document clinical visits, patient-driven intake processes, and automated prior authorizations, and generation of referrals, for example. And over the longer term, AI may more directly improve diagnosis and monitoring (for example, by parsing passive data measures). It may help to minimize adverse treatment outcomes by identifying higher-risk individuals requiring greater intensity of treatment. By more systematically monitoring and analyzing outcomes, AI may inform development of novel systems of care, allowing more rapid innovation – as with apps, allowing real-time trials of new strategies. Ultimately AI should also accelerate the science of treatment development both for psychotherapies and pharmacologic or other somatic treatments. In the latter case, AI scientists that review relevant literature and raw data and nominate novel therapeutics for investigation have already shown promise for non-psychiatric disease⁷.

Risks

Diminished Access to Human Care

Some of the risks associated with AI are not inherent in the technology itself, but reflect the way in which it may be deployed to alter care delivery in ways that are harmful for patients. For example, chatbots could become a required first-line outpatient treatment for many psychiatric disorders, a new tier in the treatment approval process. It is easy to imagine an insurer requiring evidence that a patient has tried to complete a full course of app-based CBT before authorizing any further treatment. Telehealth companies have already demonstrated an enthusiasm for diverting patients to such lower-cost interventions. In one such controversial decision, a telehealth provider unilaterally elected to triage all depressed individuals with moderate symptom severity away from individual psychotherapy.

Done thoughtfully and carefully, it is likely that some individuals would benefit sufficiently from an automated first-line therapy. However, as a mandated first step, chatbot-based therapy could delay effective treatment for many patients who could benefit from it.

To date, most chatbot studies compare these interventions to wait-list controls³. This design may reflect the reality that for many people, delays to see a human psychotherapist may be long. As a recent commentary noted⁸, it is important to compare AI-based interventions to current standards, not solely aspirational ones. In the long run, well-powered non-inferiority studies should compare such interventions to current best practices. Compelling evidence of non-inferiority should be a prerequisite for payers to require initial AI-delivered treatment. (And ultimately, investigators as well as clinicians should pursue AI-delivered therapy that exceeds human therapist performance.)

Unanticipated Changes in Clinical Practice

There are likely to be other unanticipated impacts of AI on care delivery. In one example, use of AI scribes in primary care was associated with changes in patterns of depression diagnosis, treatment, and referral (Perlis, in review). While not necessarily negative per se, these results highlight the importance of understanding such changes and not simply assuming that all AI deployment is beneficial. More broadly, while it has been assumed that AI scribes would improve clinician efficiency and could diminish burnout, it bears noting that not all studies find this to be the case.^9,10,11,12 Most notably, the impact of scribe use on patient care has not been well-characterized, particularly in terms of mental health.

The use of scribes highlights the extent to which technology deployment in health care has been accelerated by a flood of investment. In mental health, new technology has often been embraced as a means of addressing persistent underfunding and lack of parity. However, the pressure to grow quickly may outstrip efforts to fully grasp the consequences of new technologies. The proliferation of telemental health companies provides a cautionary tale: while undoubtedly a valuable strategy for improving access, quantity may increase at the cost of quality. For example, one telehealth company was pursued by the US Department of Justice for “Business Practices that Encouraged the Unauthorized Distribution of Controlled Substances”.¹³

In fact, any new technology that impacts mental health care may have both positive and negative consequences. The use of patient-reported outcomes (PROs) has been hailed as critical to facilitate measurement-based care, quantifying symptoms to ensure adequate treatment. On the other hand, primary care notes incorporating PHQ-9 as a measure of depression severity displayed a marked paucity of symptom documentation otherwise – perhaps because clinicians assumed that mental health symptoms were adequately captured by the self-report scale¹⁴. The lack of documentation made it nearly impossible to infer depression severity from narrative text. Again, while not necessarily negative, the impact of diminished documentation of individual symptoms on patient care will require further study to understand its significance.

These hints of changes in clinician behavior also raise the possibility that AI may inadvertently diminish the quality of care provided by humans. Anecdote suggests that medical trainees are among the most enthusiastic embracers of automated reference tools like OpenEvidence. These tools allow a medical student to rapidly access the accumulated knowledge of experts, integrated and summarized in a manner that differentiates these tools from simply searching the internet.

But the long term consequences of this instant knowledge are difficult to predict. In other disciplines, automation has brought both benefits and harms. The canonical example is use of autopilot and associated warning tools in aviation. Undoubtedly these technologies prevent more accidents than they cause, allowing pilots to focus on particular aspects of flying and preventing them from initiating movements that could exceed a plane’s limits. At the same time, the tendency for pilots to become dependent on these tools – and less able to fly planes independently – was recognized decades ago by the US FAA, responsible for regulating pilot proficiency. In 2013, they issued a safety alert noting, “continuous use of autoflight systems could lead to degradation of the pilot’s ability to quickly recover the aircraft from an undesired state.”¹⁵ Whether AI use will similarly degrade physicians’ clinical skills, particularly in more novel situations, merits further study. In one prior study, we found that providing erroneous recommendations was more likely to cause clinicians to choose incorrect treatments – particularly those clinicians who noted less familiarity with artificial intelligence.¹⁶

Misaligned Models Causing Harm

These sorts of risks are not specific to AI; any new technology can and likely will be applied by payers in an effort to lower costs and increase system efficiency. However, AI itself carries technology-specific risks that require attention as it is deployed for treatment. One substantial risk is misalignment, the notion that AI may guide treatment decisions, or provide talk therapy, that is not in the individual patient’s best interest. A decision tool may be developed that optimizes for fewer referrals to a specialist, or less use of costly medications, particularly when outcomes that matter most to patients (e.g., functioning and quality of life) are difficult to measure and therefore difficult to use for reinforcement learning or other tuning. While beneficial to the system as a whole, such a model will not be aligned to the individual patient’s needs. Current systems of care also apply algorithms that may not be optimal for patient care, but incorporation of AI may further obfuscate these motivations.

Misaligned AI also carries risks to individual patients when guardrails – the rules intended to keep an AI from behaving in a certain way – fail, or when AI provides otherwise unanticipated outputs. A well-publicized illustration of such a failure was an eating disorder chatbot that provided weight loss advice to its users¹⁷. While this particular mode of failure reflected a lack of experience in language model deployment by the developer, it indicates the broader risk that models can behave in unanticipated ways with widespread deployment. Unlike prior chatbots, which are deterministic such that their behavior can be inferred prior to deployment, large language models are by definition probabilistic, rendering them more difficult to fully test. Indeed, the US FDA has struggled to identify appropriate means of regulating tools incorporating such models¹⁸.

Multiple models, in multiple contexts, have provided advice or encouragement for suicide attempts¹⁹. Such models may provide other advice that is harmful to users’ health, including advising users to ‘eat one small rock a day’²⁰. And a nascent evidence base suggests that frequent and intensive use of chatbots in vulnerable individuals may contribute to depressive symptoms²¹ and delusions²².

Societal Consequences

Outside of health care, misaligned AI – or adversely-aligned AI – may also cause harm. An abundant literature supports the association between social media use and depression, anxiety, and irritability^23-26. While much of this is correlative and cross-sectional, some longitudinal data and a small number of randomized trials suggest a causal relationship, at least in part. It is likely that increased use of AI either to generate social media content, or to promote other engaging online interactions, will increase exposure to this kind of media. In this context, AI is tuned to increase engagement, not necessarily in the best interest of the user. While there are strategies to mitigate some of this risk – screen time reminders, encouragement to pursue other activities – it seems unlikely that companies who profit from greater engagement will invest in reducing engagement. Whether or not these behaviors are labelled internet addiction²⁷, they do appear to be consequential for many users²¹.

A particular societal risk with AI use may be loss of social connections and social skills. While largely theoretical at this point, it seems likely that greater use of AI for parasocial activity (e.g., chatting online) will displace at least some social interaction. Some amount of this activity may be beneficial, particularly for individuals who otherwise lack social outlets. Still, it seems unlikely that these activities can fully replace the benefits of social interactions, particularly face-to-face interactions. Recent work indicates high rates of loneliness in the US, including among older adults²⁸; AI may prove to be a 2-edged sword in addressing what the US Surgeon General has labeled an epidemic²⁹. Over the longer term, as AI becomes more embodied (i.e., in robots or even machines that closely resemble humans), and more ingrained in daily life, its impact on mental health may be even more challenging to predict.

A more insidious risk with AI may be manipulation. Both advertisers and political campaigns have employed AI to design more targeted and persuasive online campaigns. The brief appearance of a commercial chatbot absolutely determined to discuss a misleading view of South African politics³⁰, while humorous in its heavy-handedness, hints at the potential for language models to be used to change minds – and not always in a beneficial way. This potential could be managed by relying on models where the sponsor guarantees neutrality in some way – but the proliferation of chatbots and models suggests many people will not limit themselves to such curated models. TikTok may not be a mainstream news source, but it remains a news source for many US adults³¹.

A final risk of misalignment arises from the capacity of language models to help users complete tasks that would previously have required far more knowledge and resources. The rise of ‘vibe coding’ – that is, use of language models to develop software by individuals who otherwise are not adept at writing code – hints at the potential of these models. These tools are also available to those who wish to use them to do harm – for example, to create computer viruses or steal data. Beyond computer science, the application of models to facilitate development of nuclear or biological weapons is increasingly easy to envision. These risks have been explicitly cited by the creators of some of the frontier models, with recent indications that frontier models are now sufficiently knowledgeable to pose real risk³².

In parallel, these models – especially when integrated to form agents that can interact to complete real-world tasks – are likely to provoke substantial shifts in the labor market. Many entry-level jobs are susceptible to automation, and the capability of AI to replace human workers will only increase. In fact, some economic data suggest some of this displacement may already be occurring³³.

Together, these two potential consequences of AI – widespread destruction, or at least widespread job loss – are also new sources of anxiety. Whether these worries will simply replace older ones, or increase the overall global burden, remains to be seen. In both scenarios, it is also worth considering the possibility that AI will yield solutions – more efficient vaccination, e.g., or ways to redistribute income so that increased productivity benefits those who are displaced.

Solutions

No single strategy can address all of these concerns. One near-term solution may be regulation to address specific risks. The US FDA is limited in its ability to regulate AI in health by the 21st Century Cures Act, which restricts their scope to software used for diagnosis and treatment. Previous FDA leadership issued guidance statements (for AI in general, and for language models) but these seem unlikely to guide policy in the current political environment³⁴, particularly given the rapid pace of change in the technologies being deployed. Congress can also regulate AI, but has shown little inclination to do so. Individual states have stepped in to issue regulations – 31 by the end of 2024 – but these regulations and their impact on health in particular vary widely. Among the areas of regulation have been mandating strategies to manage risk, assess AI impact, and ensure algorithmic fairness³⁴. A patchwork state-by-state approach to AI regulation may inhibit applications of AI for health, and lead AI technology developers to simply avoid particular states – including some states that have contributed disproportionately to AI development. On the other hand, absent Federal leadership, action at the state level may be the only realistic means of addressing potential harms in the US.

Regardless of the source of regulation, one key need is for transparency. Requiring that details of the models being applied be made public, perhaps along with performance on particular benchmarks, would help health systems, clinicians, and patients and families make informed decisions about technology use. Payers could be required to disclose how and when they apply AI to guide mental health treatment.

As the impact of new tools becomes more apparent, it may be necessary to restrict certain applications for mental health, specifically to ensure that they do not become gatekeepers to more standard methods of care delivery. Even if an AI technology is cost-effective, a society may decide that the particular cost-cutting strategy it embodies should not be applied. If, for example, chatbots are not to be mandated first-line treatment by some insurers, this may need to be accomplished with legislation.

Whether FDA regulation is appropriate to ensure the safety of AI-driven chatbots is open to debate. A recent clearance of a depression coaching application³⁵ arguably muddied the waters, emphasizing a lack of evidence of harm but acknowledging a lack of benefit as well. Warranting safety in this way could at least ensure patients are directed to interventions that will not harm them per se (although, as noted, the substitution of such applications for effective treatments could indirectly cause harm). FDA staff have noted that they do not regulate psychotherapy; quality is principally assured by professional organizations themselves, though the extent to which these controls are effective is also debatable.

Another strategy to reduce harm is to ensure that the next generation of physicians is specifically educated to apply AI-based tools as part of clinical practice³⁶. Being the human in the loop, rather than just a human in the loop, entails a deeper understanding of the loop itself. This education is easier said than done, insofar as technology is rapidly developing and often the most junior member of a care team is the most adept at applying new technology. At minimum, clinicians can benefit from an understanding of the ways in which AI tools may fail, and the way they may introduce bias. Physicians in training may also benefit from the equivalent of manual flying time in aviation: patient care in settings without access to AI and other automation technologies. Simulations (perhaps powered by AI) can help teach such manual flying, as they do for pilots, but may not be sufficient.

Conclusion

Tools incorporating AI could dramatically improve access to mental health care, and enhance the safety and quality of that care. However, these tools also pose substantial and underappreciated risks. Considering the multitude of ways in which AI may adversely impact mental health, from direct impacts on care to broader societal consequences, may enable development of strategies to mitigate these risks.