Is Generative AI Increasing the Risk for Technology‐Mediated Trauma Among Vulnerable Populations?

ABSTRACT

The proliferation of Generative Artificial Intelligence (Generative AI) has led to an increased reliance on AI‐generated content for designing and deploying digital health interventions. While generative AI has the potential to facilitate and automate healthcare, there are concerns that AI‐generated content and AI‐generated health advice could trigger, perpetuate, or exacerbate prior traumatic experiences among vulnerable populations. In this discussion article, I examined how generative‐AI‐powered digital health interventions could trigger, perpetuate, or exacerbate emotional trauma among vulnerable populations who rely on digital health interventions as complementary or alternative sources of seeking health services or information. I then proposed actionable strategies for mitigating AI‐generated trauma in the context of digital health interventions. The arguments raised in this article are expected to shift the focus of AI practitioners against prioritizing dominant narratives in AI algorithms into seriously considering the needs of vulnerable minority groups who are at the greatest risk for trauma but are often invisible in AI data sets, AI algorithms, and their resultant technologies.

1. Introduction

1.1. What Is Generative AI

Generative AI, encompassing models like ChatGPT, Google's Bard, and others, is an emerging sub‐field of AI that uses machine learning models to generate high‐quality text, images, and other content based on the data they are trained on (IBM Research ²⁰²¹). While earlier AI models were limited to analyzing, interpreting, and predicting scenarios based on existing data, generative AI can create new content (Houssami et al. 2019). Generative AI automatically learns patterns and structures from texts, images, sounds, animations, models, or other media inputs to produce new ones with similar characteristics. The introduction of generative AI has brought accelerated advancements and is gaining prominence in various fields including healthcare (Raza, Venkatesh, and Kvedar ²⁰²⁴; Reddy ²⁰²⁴). The ability to generate new content, plus the accessibility and ease of use of generative AI, has led to a surge in use by both patients and providers (Carrie 2024).

1.2. Generative AI in Healthcare

The proliferation of Generative AI has led to an increased reliance on AI‐assisted information seeking, clinical decision‐making, diagnosis, routine monitoring, treatment recommendations, and post‐treatment monitoring among healthcare providers and patients (Carrie 2024). Generative AI is also used in the design, development, and deployment of digital health interventions (Raza, Venkatesh, and Kvedar ²⁰²⁴). According to the WHO, generative AI can be used for creating content and images for digital health user interfaces, improving the quality of existing ones, or enhancing the resolution of images in digital health platforms (WHO ²⁰²⁴). Generative AI has the potential to streamline the development of digital health interventions by facilitating the coding process and improving the development of computational solutions for patients. The potential of generative AI in digital health was demonstrated in a study that used ChatGPT to develop a digital health intervention for supporting patient engagement and adherence to digital diabetes prevention programs (Rodriguez et al. 2024). In this project, ChatGPT was not only used in developing the codes for the intervention but also supported the conceptualization of the product, content development, software requirement generation, and software design (Rodriguez et al. 2024). Beyond assisting with the design and development of digital health, generative AI can also help patients become more autonomous by facilitating individualized assessment of data and symptoms at a faster pace than humans can do (Meskó and Topol 2023). For instance, chatbots can provide patients with 24‐h access to health information including symptom assessment, medication reminders, counseling information, and appointment scheduling (Clark and Bailey 2024).

While generative AI has great promise in the development of digital health and delivery of healthcare, there are concerns that advancements in this technology could increase the risk of technology‐mediated trauma among technology users, particularly vulnerable populations. In this article, vulnerable populations are used to refer to people and groups including those with intellectual disabilities, sex workers, 2SLGBTQI+ people, racialized people, indigenous, youth as well as elderly patients, and religious groups who are most at risk of technology‐facilitated abuse. This definition is adapted from the government of Canada's 2018 definition of vulnerable population as encompassing marginalized identities including immigrants, children, Indigenous, newcomers, members of LGBTQ2+ communities, language minority communities, people living with mental illness, people living with homelessness, persons with disabilities, etc. (Government of Canada ²⁰¹⁸). While all technology users, including the privileged, might be faced with technology‐facilitated abuse, it is important to state that vulnerable populations may be more impacted (Chen et al. 2022; Venkatasubramanian and Ranalli ²⁰²²). The risk of technology‐mediated trauma among vulnerable populations is heightened because of prior traumatic experiences at the personal, structural, and systemic levels in the healthcare systems (Haimson et al. 2020; Matthews et al. ²⁰¹⁷; Simko et al. ²⁰¹⁸).

Technology‐based interventions can trigger or exacerbate such prior traumatic experiences if used inappropriately. For instance, various technology‐based interventions such as mobile applications, spyware tools, and the Internet of Things could be used to cause harm that may result in traumatic symptoms including anxiety, panic attacks, fear, anger, and irritability (MacLure and Jones ²⁰²¹). Other technology‐facilitated cycles of abuse can take the form of online sexual harassment, cyberstalking, image‐based exploitation, and the use of manipulative technologies services to coerce people into unwanted sexual acts (Henry and Powell 2018). Given that vulnerable populations may have a disproportionate rate of trauma (Newland et al. 2022), any negative effects of digital technologies could trigger or exacerbate any prior traumatic experiences. This emerging threat of technology‐mediated trauma is particularly concerning for people who rely on AI‐powered digital health interventions as alternative or complementary sources of seeking health information or services. To contextualize the discussions and understand how generative AI can trigger, perpetuate, and/or exacerbate trauma, I first provided an overview of trauma and how trauma occurs in the context of digital health technologies.

1.3. Trauma

In this article, trauma is used to describe the challenging emotional and psychological consequences associated with an individual after going through a distressful event (Dupont 1998). This definition of trauma is specific to emotional and psychological trauma, which is different from physical trauma (i.e., injuries or body wounds produced by sudden physical injury from impact, violence, or accident). Available evidence suggests that approximately 70% of people across the globe will experience some form of trauma during their lifetime (Knipscheer et al. 2020). This number is estimated to be high among vulnerable populations who are already faced with systematic injustices and inequitable access to healthcare services (Tebes et al. 2019). Trauma resulting from the use of digital health technologies (herein referred to as technology‐mediated trauma) is an emerging threat that has a profound impact on the lives of technology users, particularly among vulnerable groups (Chen et al. 2022). Given these high rates of trauma and the concomitant increase in the use of digital health technologies, it means many technology users might be survivors of trauma (Wong et al. 2021). For vulnerable populations who might have experienced prior traumatic events, any additional negative effects of health technologies might trigger, perpetuate, or exacerbate prior traumatic experiences – further exacerbating health inequities for those with the greatest need.

1.4. Technology‐Mediated Trauma in Digital Health

While there is not yet a universally accepted definition of technology‐mediated trauma, it is generally conceptualized as technology‐enabled activities, interface appearances, or features that leave a person in emotional distress following the use of digital technology (Chen et al. 2022). Technology‐mediated trauma may occur inadvertently through interface features that remind users of prior traumatic circumstances (Holloway et al. 2014; Musetti et al. ²⁰²¹). For instance, technologies that expose people to online scams, sextortion, identity theft, and cyberbullying could result in short‐term or long‐term emotional trauma, and sometimes, suicidal thoughts for people (Kraft 2006; Schüz and Urban ²⁰²⁰). For survivors of trauma like victims of childhood sexual abuse, viewing certain sexually explicit content that they consider to be inappropriate can re‐traumatize them and further create technology‐facilitated cycles of trauma or re‐traumatization each time the content is viewed (Regehr, Birze, and Regehr ²⁰²²). Even seemingly neutral user interface design elements developed in line with established design guidelines (e.g., the use of different colors and images) can trigger emotional trauma in people with negative prior experiences or a negative attitude toward the said elements (Shaughnessy et al. 2022). Bonell et al. (2015) termed these sorts of occurrences as the “dark logic” of public health interventions while Ziebland, Hyde, and Powell (2021) also described it as the “paradoxical consequences of technologies” where interventions designed for treating a particular condition inadvertently deliver a contrary outcome.

Aside from the inadvertent effects of technology on trauma, trauma can also be orchestrated intentionally through negative online activities. For instance, Digital technologies can trigger emotional trauma through interface features that can be used to harass, coerce, or stalk someone (Freed et al. 2018). These sorts of abuses can range from intimate image abuse, and the use of geolocation features for stalking, to the use of Internet of Things devices to manipulate people into doubting their sanity (Regehr, Birze, and Regehr ²⁰²²). Perpetrators of intimate partner abuse and domestic violence can install or place Global Positioning Systems (GPS) devices to track their victims and possibly cause harm (Straw and Tanczer 2023). Cyberbullying perpetrated via online social networking sites such as negative forum posts and comments has been shown to result in emotional trauma among patients with prior histories of trauma (Habib et al. 2021; Hong et al. ²⁰²⁰). Other forms of communication channels on digital platforms including chatrooms, user subscription channels, and forums/bulletin boards may be used unscrupulously to victimize and expose users in a manner that could result in emotional trauma (Henry, Flynn, and Powell ²⁰²⁰; Machimbarrena et al. ²⁰¹⁸).

The preceding evidence suggests that technology‐mediated trauma is not a new phenomenon but a set of events that are associated with technology use. With the innovative and widespread use of generative AI in social media (Sheikh and Rogers 2024), I argue that technology‐mediated abuse may not only be on the increase but could also be perpetrated by lay and less sophisticated people with ill motives. For instance, generative AI can exacerbate trauma by introducing new, highly personalized, and invasive forms of harassment (UK Council for Internet Safety ²⁰¹⁹). Also, biased AI models can enhance technology‐facilitated abuse by enabling the creation of highly realistic and deceptive digital content, such as deepfake videos, images, and audio recordings, which can be used to manipulate, impersonate, or defame victims (Stonard et al. 2017). This advanced capability can significantly intensify the victim's sense of helplessness and violation, as the fabricated content can be indistinguishable from reality, making it harder to disprove and stop its spread. The sophistication and reach of generative AI can amplify the psychological impact, leading to severe anxiety, fear, and trust issues (Lucas 2022). The use of AI‐enabled deepfake videos has led to unprecedented growth in image‐based intimate sexual abuse with profound emotional and social impacts on the victims' lives (Lucas 2022). A recent study shows that deepfake videos can be used to enact sexual violence and such acts could be particularly harmful and could result in endless cycles of abuse for the victims (Rousay Victoria ²⁰²³). The enduring and replicable nature of AI‐generated content means that the abuse can persist indefinitely, further entrenching the trauma and its long‐term psychological effects, such as PTSD, depression, and a diminished sense of personal security (Regehr, Birze, and Regehr ²⁰²²; Sheikh and Rogers ²⁰²⁴). For vulnerable populations who already experience trauma at the personal, structural, and systemic levels in the healthcare systems (Haimson et al. 2020; Matthews et al. ²⁰¹⁷; Simko et al. ²⁰¹⁸), such abuses can cause long‐lasting or permanent emotional trauma that could reoccur anytime such videos or images are brought to their attention. In such situations, victims may experience an overwhelming sense of loss of control over their digital and real‐world identities (Rousay Victoria ²⁰²³). With the increasing use of AI‐powered technologies among vulnerable populations including survivors of trauma (Venkatasubramanian and Ranalli 2022), any potential negative consequence of such technologies could exacerbate trauma and widen health inequities among vulnerable populations who have the most needs but face the greatest barriers in access to care. In the rest of this article, I explained how AI algorithms and their resultant technologies can trigger or perpetuate harmful consequences including trauma among historically vulnerable populations. I then proposed actionable strategies for ensuring safer and trauma‐informed AI‐powered digital health technology interventions.

2. How Generative AI Increases the Risk for Technology‐Mediated Trauma

Generative AI is seen as a promising tool that can speed up the design and use of digital health interventions and facilitate autonomous decision‐making for patients (Rodriguez et al. 2024). The opportunities presented by generative AI for automating healthcare are both exciting and problematic at the same time. Despite its vast potential, there are concerns that the use of generative AI can increase the risk for technology‐mediated trauma – posing a risk to people with prior traumatic experiences who rely on digital health for information and services. Traumatic experiences via generative AI could occur in several ways including the use of underrepresented and biased data sets used in training AI algorithms, the provision of inaccurate health information, and the intentional creation of malicious AI models for spreading disinformation.

2.1. Underrepresented, Biased, and Inaccurate Data Sets Used in Training AI algorithms

First of all, it is important to understand that current AI algorithms rely largely on data from Western, Educated, Industrialized, Rich, and Democratic (WEIRD) societies (Henrich, Heine, and Norenzayan ²⁰¹⁰). Vulnerable minority groups who are susceptible to trauma are often underrepresented in data sets that are used in training AI models (Shanklin et al. 2022). With such underrepresentation, AI algorithms may often not recognize minority and marginalized groups whose concerns are often perceived to occur outside the dominant narratives. For instance, one study found that a widely used patient prioritization AI algorithm was more likely to prioritize white patients over black patients with the same level of health needs (Obermeyer et al. 2019). Another study on the deployment of AI‐enabled medical imaging was found to exacerbate existing biases among Hispanic female patients and could potentially lead to unequal access to medical care for underserved populations like women (Seyyed‐Kalantari et al. ²⁰²¹). Such biases may not only perpetuate existing health inequities but also downplay or exacerbate concerns that could trigger or perpetuate trauma among marginalized populations (Larasati 2023).

The lack of recognition of different marginalized identities in AI algorithms and AI‐powered digital technologies may reinforce a feeling of lack of belongingness and create harmful consequences including anxiety, anger, lack of control, and self‐inadequacy among marginalized populations who use AI‐powered technology interventions (Holmes 2018). Also, biased online information on marginalized populations could lead to problematic representation of minority groups while amplifying harmful social biases and negative stereotypes in generative AI‐powered technologies (Hastings 2024). Also, AI algorithms that are trained on data sets that contain inadequate data on gender, racial, or ethnic representation may exhibit the same racist and gender biases and reinforce harmful stereotypes as their source (Obermeyer et al. 2019; Lamensch ²⁰²⁴). For instance, a predictive generative AI algorithm that predominantly contains images of white skin was found to produce biased and inaccurate images of dark‐skinned people (Adamson and Smith 2018). Technologies resulting from such algorithms can either exacerbate prior trauma or introduce new forms of trauma by preventing people from seeking help with the intervention (Eggleston and Noel 2024). Digital health interventions that use biased AI‐generated content and images may also appear as the exclusion of people who experience marginalization and a lack of attention to the safety and care of these populations (Holmes 2018).

In addition to the biases in AI algorithms, generative AI may also use data sets that may contain inappropriate, or harmful information about certain minority groups. While some AI companies try to validate their data sources and remove inappropriate data sources (Biessmann et al. 2021), those data sources might still contain information that could be harmful or at best inappropriate for specific user groups. Therefore, AI‐powered technologies resulting from inaccurate data sources might result in user interface content that could be traumatizing for vulnerable minority groups who find such content as either undesirable, reinforcing negative stereotypes, or reminding them of prior traumatic events (Regehr, Birze, and Regehr ²⁰²²). For instance, AI algorithms that are trained on internet‐based information may use “naked images” of people to create or replicate hypersexualized images that could be harmful to sexual minority groups. These harmful misrepresentations are more common in sexual health‐related technologies where hypersexualized data from the internet are used to create and propagate negative images of minority identities including 2SLGBTQI, blacks, and religious minority groups (Lamensch 2024). For instance, a recent study found ChatGPT‐4 to depict problematic representation and harmful social biases that carry considerable risks of triggering, exacerbating, and perpetuating harmful consequences among minority groups including blacks, 2SLGBTQI people, indigenous, racial, and religious minorities (Zack et al. 2024). These AI‐mediated cultural misrepresentations are especially concerning when AI algorithms cannot distinguish between appropriate cultural appropriation and negative cultural stereotyping. Cultural appropriation is the representation of cultural practices or experiences by “outsiders,” while negative cultural stereotyping refers to widely held beliefs (mostly in a negative light) about a particular social group (Kashima 2008; Matthes ²⁰¹⁶). In addition to the biases and inaccurate representation of minority groups in AI algorithms, there is also a lack of diversity within the teams that design and deploy AI algorithms (Green, Murphy, and Robinson ²⁰²⁴). Research from the AI Now Institute (https://ainowinstitute.org/) highlighted that the majority of AI developers and researchers were white cis‐gender males (AI Now Institute ²⁰¹⁸). The powerful and often non‐diverse technology industry tends to hold a narrow view of marginalized populations or may not even understand trauma. As a result, such teams may not understand the implications of design decisions on marginalized populations and how well‐intentioned design decisions could inadvertently re‐traumatize minority identities.

2.2. Inappropriate Health Advice via AI‐Powered Chatbots

One other area in which generative AI can foment trauma is through AI‐powered technologies such as chatbots used in delivering healthcare (Boucher et al. 2021). There has been a recent increase in the use of chatbots to counsel people, including survivors of abuse, and also to connect people experiencing high rates of trauma and psychological distress with healthcare providers (Ngũnjiri et al. 2023). While such chatbots might be easily accessible and convenient to use, the inaccuracies and underrepresentations in AI data sets and AI algorithms might lead to situations where AI‐powered chatbots use words, and statements or give health advice that reminds people of prior traumatic events, thereby re‐traumatizing rather than helping them (Coghlan et al. 2023; McMahon and McMahon ²⁰²⁴).

Empathy, which is considered the ability to recognize people's thoughts and feelings and to respond with appropriate emotions, is an essential element for survivors of trauma (Greenberg et al. 2018). For survivors of trauma, empathy and compassionate engagement are necessary for fostering connection, building resilience, empowerment, and gaining coping strategies after going through a distressful event (Nugent, Sumner, and Amstadter ²⁰¹⁴). While recent studies have demonstrated the ability of ChatGPT to recognize human emotions and produce empathetic responses to queries (Elyoseph et al. 2023), generative AI does not yet fully possess the human capabilities for conveying patient‐specific emotional demands and other traits that require critical thinking and emotional intelligence. Even when AI models can communicate with empathy, the algorithms may not understand the patients' context or may not contain background information to be able to tailor responses or distinguish between what could be helpful or traumatizing to each patient's circumstances. For instance, ChatGPT may not understand the traumatic past of a patient who uses an AI‐powered chatbot to seek information on abortion. In healthcare and nursing, in particular, each individual is unique, and generative AI in its current state may not be able to respond to the uniqueness of each patient (Katims 1995). Other crucial human communication traits such as active listening, eye contact, and portraying a sense of connectedness with patients are important for trauma survivors but not inherent in current generative AI models. Without the ability to convey such traits, AI‐powered technologies could end up being distressful to vulnerable populations that urgently need emotional connection and empathetic communication to be able to deal with their traumatic past.

2.3. Intentional Use of Malicious AI Models

Another way that generative AI increases the risks of technology‐mediated trauma is through the intentional development of malicious AI algorithms that can be used to target vulnerable populations including survivors of trauma. Despite the proactive steps by companies like Open AI and Google Bard to mitigate the nefarious use of generative AI, malicious AI models have been developed and will continue to be created (Ferrara 2024). These adversarial AI models highlight the potential threats generative AI poses to the health and well‐being of vulnerable populations. Malicious AI models such as WormGPT and PoisonGPT, for instance, were created as “black hat” alternatives to standard ChatGPT (McGowan ²⁰²³). These AI models can be used to produce deceptive narratives, fake images, and deepfake videos that could trigger and perpetuate prior traumatic events with harmful consequences (Maras and Alexandrou 2019). Deepfakes are videos that have been convincingly manipulated to represent someone and spread misinformation (Maras and Alexandrou 2019). For instance, WormGPT can be leveraged to create malicious content, images, or information targeting vulnerable populations – triggering and perpetuating prior traumatic experiences. Another concerning scenario is the potential modification of existing AI algorithms to spread misinformation and disinformation through open‐source coding. For instance, generative models such as PoisonGPT can be used to generate images, videos, and audio content to vividly and persuasively convey misinformation about a particular group or individual (Xu, Fan, and Kankanhalli ²⁰²³). This threat, in particular, is real because of the use of unregulated and unaudited internet information/data that are used in training these adversarial AI models (Martínez et al. 2024). While AI‐generated misinformation can be used to target people with any disease condition, they are more likely to be targeted at people with sensitive and potentially embarrassing health conditions or people seeking politically sensitive health services (e.g., abortion) (McMahon and McMahon ²⁰²⁴; Xu, Fan, and Kankanhalli ²⁰²³). For instance, ChatGPT was found to provide abortion‐related information that overstated the risk associated with self‐managed abortion in a manner that contradicts current evidence, perpetuates misinformation, and amplifies the stigma associated with abortion (McMahon and McMahon ²⁰²⁴).

Given the increasing adoption of generative AI in designing digital health technologies and automating patient decision‐making, the issues raised in this article should be concerning to AI researchers and AI practitioners. It is also important to note that generative AI in healthcare is still in its infancy and, therefore, its effects on trauma might not have been widely noticed or reported yet. Nevertheless, AI researchers and AI practitioners must be aware of these concerns and take actionable strategies to prevent traumatic consequences in future AI algorithms and their resultant technologies.

3. Potential Actionable Strategies Toward Trauma‐Informed AI Technologies

Given the increasing threats of generative AI, researchers and AI practitioners are exploring various ways of designing safer, trustworthy, and responsible AI. For instance, the European Commission developed ethical guidelines for developing responsible and trustworthy AI (Lemonne 2018). Also, tech companies such as Google and agencies like the Responsible AI Institute developed guidelines for creating, procuring, and deploying safe and trustworthy AI systems (Responsible AI Institute ²⁰²⁰). These guidelines are generally meant to promote technical robustness, safety, transparency, and diversity, ensure nondiscrimination, and enhance fairness and accountability in AI technologies. While these guidelines might be gravitating toward a trustworthy and safer AI, it must be noted that they are largely high‐level recommendations, not specific to the needs of marginalized populations, and are therefore inadequate in mitigating technology‐mediated trauma in AI‐powered digital health interventions.

To design AI‐powered digital health technologies with the least chance of triggering or perpetuating trauma, health‐related AI researchers and industry partners should undergo trauma‐informed training to have a better understanding of trauma, the possible trigger factors, and its effects on technology users, particularly vulnerable populations. While trauma‐informed care practices are currently being integrated into agile software development and virtual reality programs (Anton Skornyakov ²⁰²³; Matt ²⁰²⁴), it is not evident if software engineers in the technology industry, particularly those who design health‐related AI models are undergoing any training opportunities before developing AI algorithms, particularly for populations who face existing vulnerabilities in society. To make AI‐powered technologies less traumatizing to the end user, I propose that industry partners and AI practitioners must engage with trauma experts and possibly survivors of trauma during the design and development phases to ensure a deep understanding of the needs of trauma survivors. Such engagement would not only help to audit, identify, and address data sources that can foment trauma but also ensure a sense of ownership of AI technology by end‐user patients.

While patient engagement is crucial in mitigating technology‐mediated trauma, it is important to state that some engagement approaches could re‐traumatize some vulnerable groups (Knipscheer et al. 2020). Also, some vulnerable patients may not be in the position or have the capacity to communicate their needs or indicate what could be traumatizing to them. In such instances, health professionals like nurses could advocate for the needs of the patient population to be addressed in AI algorithms and ensure that vulnerable patients are not re‐traumatized in an attempt to help them (Nsiah, Siakwa, and Ninnoni ²⁰¹⁹). In addition to patient engagement, AI practitioners also need some standardized guidelines to stay abreast with information on their patient population and current issues that could re‐traumatize them. This could be like an extension of existing design guidelines to allow for the development of technologies to meet the needs of marginalized populations (Avellan, Sharma, and Turunen ²⁰²⁰; Tengkawan, Agnihotri, and Minhas ²⁰²²). Furthermore, inclusive design guidelines like integrating exit buttons, having simple fonts, reducing cognitive load, limiting the number of navigational options, and providing options for support could be applied in developing AI‐powered technologies to limit the potential for distress caused by difficult‐to‐use interfaces (Kelly, Lauren, and Nguyen ²⁰²¹; Venkatasubramanian and Ranalli ²⁰²²). Indeed, domain‐specific design guidelines have been developed for other equally difficult topics like stigma (Abdulai et al. 2021, ²⁰²³). Therefore, it is only proper that technology design guidelines tailored to trauma for vulnerable populations are also developed. Such guidelines would not only enhance the user experience for trauma survivors but also promote inclusivity and accessibility – making technology interventions more effective and empathetic for a diverse range of users.

Another concern that needs to be addressed to prevent technology‐facilitated abuse is diversity. It has been suggested that AI researchers and developers are mostly white men (AI Now Institute ²⁰¹⁸). I contend that it could be challenging to increase minority groups' representation in AI research and development teams in the short term. In the meantime, AI developers should recognize the diversity of potential users, appreciate their lived experiences, and develop AI models to accurately reflect differences in sex, gender, race, education, religion, skills, and disability (Ulnicane 2024). Such appreciation might help AI developers develop models that adapt and respond to the queries of different patient populations without introducing biases or reinforcing stereotypes, thereby addressing the generality of AI‐generated responses. For instance, AI researchers at Stanford University developed an AI algorithm that was able to discern people's sexual orientation on the dating site with remarkable accuracy (Wang and Kosinski 2018). These sorts of algorithms could prevent technology‐mediated trauma often caused by misrepresentation or non‐representation of sexual minority groups (Eggleston and Noel 2024). Recognizing and integrating diversity is important because non‐diverse AI algorithms could result in technologies that essentially reject nonnormative practices and rather project beliefs that could be oppressive, traumatizing, and negatively impact the safety of minority groups who face other vulnerabilities in life (Nicki 2001).

Research shows that AI‐powered technologies like Chatbots could result in technology‐mediated trauma if the advice offered reinforces prior traumatic experiences (Bendig et al. 2019). As chatbots are being integrated with electronic health records (Clark and Bailey 2024), urgent steps are needed to evaluate the clinical utility of these technologies and how they can adapt to the specific circumstances of patients, particularly vulnerable groups who are prone to trauma. For example, EPIC and Cerner (Electronic Health Records companies) partnered with Microsoft Open AI to integrate chatbots into their Electronic Health Records – enabling EHRs to auto‐draft responses to patient queries, order medications, and schedule follow‐up appointments (Microsoft News Center ²⁰²³). Given the potential risk that the misuse of these technologies poses to vulnerable groups, appropriate actions must be taken to evaluate their clinical value for minority groups before being implemented for patient care. Such evaluation could adopt the framework developed by Wornow et al. ( 2023) for evaluating AI models in healthcare settings. The framework comprises six criteria including predictive performance, data labeling, model development, emergent clinical applications, multimodality, and novel human–AI user interfaces.

Beyond establishing clinical utility for minority groups, I propose that AI algorithms should also contain context‐specific information and human values. This is consistent with prior and recent calls for integrating human and ethical values in developing technology interventions (Barnard and Sandelowski 2001; Duarte and Baranauskas ²⁰¹⁶). It has already been established that ChatGPT can recognize human emotions and produce empathetic responses to queries (Elyoseph et al. 2023). Digital empathy can be enhanced by having tech developers work closely with nurses in developing AI‐powered technologies including chatbots. Nurses are particularly skilled in empathy and individualized care and could help in developing AI‐powered chatbots that not only communicate with empathy but also provide individualized and context‐specific responses (Rey Velasco et al. 2024). Also, given the inherent variability in what people perceive as correct responses and the fact that medical AI algorithms are often unclear about the target population, settings, and the handling of missed data (Collins et al. 2024; Wu et al. ²⁰²¹; Wynants et al. ²⁰²⁰), further efforts should be made to contextualize and provide individual‐specific and human‐centric responses. Recent developments in AI research show that this is possible as there are growing body of work to make AI more human and individual‐centered (Dhamala et al. 2021; Nadeem, Bethke, and Reddy ²⁰²¹; Ouyang et al. ²⁰²⁴). These efforts should be primarily aimed at identifying, integrating, and aligning human values in AI models. Furthermore, AI‐powered chatbots should provide real‐time responses by relying on the input and context of the patient (Meskó and Topol 2023). Such AI‐enabled adaptive systems have been successfully used in education and learning systems and could prove useful for health‐related technologies (Kabudi, Pappas, and Olsen ²⁰²¹).

In addition to the lower‐level actions toward mitigating technology‐mediated trauma, I propose that governments, the private sector, industry leaders, and academic researchers on AI should partner to discuss governance mechanisms that mitigate the risks and potential harm of AI while harnessing its potential for vulnerable groups. Such governance mechanisms could include collaborative efforts at building transparency, fairness, and accountability in the design, deployment, and clinical validation of AI algorithms (Xu, Bradford, and Garg ²⁰²³). There has been a strong and consistent call for transparent and ethical AI to promote equitable use of technology but such practices are not necessarily enforced (Basu, Faghmous, and Doupe ²⁰²⁰; Larsson and Heintz ²⁰²⁰). For instance, while some generative AI models such as Adobe Firely (i.e., a generative AI for creating images) have seen adhering to transparency guidelines by publishing information on the data/images used in training their model, others such as Dall‐E, have been accused of lacking transparency in their training data (Marr 2024). This lack of transparency makes it difficult to assess the risks of AI models used in generating images for AI‐powered digital health technologies (Marr 2024). Publishing information on the AI development process, including the data sources, model choices, and the algorithms used in designing AI technologies would provide enough information to inform the public about the safety and potential risks of AI algorithms (Fehr et al. 2024). For already existing AI‐powered technologies like chatbots, developers should publish regular transparency reports that highlight the systems' performance, accuracy rate, biases, and error rates as well as disclose the data of population sub‐groups that are underrepresented in AI algorithms (Xu, Bradford, and Garg ²⁰²³). There is a greater chance that generative AI would improve people's health and well‐being if people trust the AI technologies to behave safely, securely, and understandably.

4. Conclusion

Technology‐mediated trauma is an emerging threat that requires urgent attention when developing AI‐powered digital health technologies. As generative AI is gradually adopted in healthcare, it is important to assess the potential effects of the technology on trauma and take actionable strategies to mitigate technology‐mediated trauma among vulnerable populations including survivors of trauma. In this discussion article, I examined various ways in which AI algorithms and their resultant digital health technologies could result in traumatic consequences and proposed actionable strategies for mitigating technology‐mediated trauma among vulnerable populations with prior experiences of trauma. Addressing technology‐mediated trauma is important because if we ignore the issue of trauma in designing and deploying AI algorithms, we risk re‐traumatizing vulnerable populations who will use such interventions, or at best being unhelpful to them. Addressing the needs of vulnerable populations is crucial because current AI algorithms almost always focus on addressing the needs of 80% of the population while neglecting what they consider “edge cases” or situations that are considered to occur outside the dominant narrative (Hastings 2024). I expect that the arguments raised in this article will sensitize AI practitioners against prioritizing dominant narratives in AI algorithms to prioritizing minority groups, or the so‐called “edge cases” in designing and deploying AI algorithms. After all, a design that addresses the needs of vulnerable minority groups would inevitably address the needs of dominant groups. As patients increasingly use AI‐powered technologies, we also have a moral obligation to ensure that such interventions are not only mitigating the unintended effects of trauma on users but also empowering users to deal with traumatic situations.

Ethics Statement

This study does not require ethics approval.

Conflicts of Interest

The author declares no conflicts of interest.

Data Availability Statement

No new data were generated for this manuscript.