Automated Health Care Messages and Unexpected Patient Responses

Key Points

Question

How do patients respond to automated messages from their health care system beyond simple opt out commands and what are the implications of these responses for improving health system automated messaging?

Findings

In a qualitative content analysis of 743 patient text responses within an integrated health system, patients used messages in attempts to manage appointments, request help or information, correct inaccurate data, express preferences, and voice frustration. These responses extended far beyond anticipated opt-out or opt-in requests, highlighting opportunities for care improvement.

Meaning

In this study, patients used automated messages as opportunities for bidirectional communication; and evaluating these unexpected responses can inform the design of more transparent, responsive systems that improve patient engagement and care.

This qualitative study examines the range of patient responses to automated text messages from their health care system and identifies considerations for improving health system communication design.

Abstract

Importance

Automated text messaging systems are crucial for health care communication, but text message software is often programmed to recognize only a limited range of responses, such as requests to stop messaging. However, patients may send replies that go beyond these expected responses. Systems may disregard or misclassify such replies, leading to accidental opt-outs or unaddressed needs. Identifying unexpected patient responses to automated messages might identify areas for improvement in communication systems.

Objective

To develop a framework representing the range of patient responses to automated text messages and to identify considerations for improving health system communication design.

Design, Setting, and Participants

This qualitative study examines all patient responses to automated text messages from January 1, 2022, to December 31, 2023, at Kaiser Permanente Colorado, an integrated health system serving more than 500 000 patients. Data were analyzed from January to November 2024.

Main Outcomes and Measures

Patient response themes and domains were identified through inductive qualitative content analysis and interpretive system-level considerations informed by identified response patterns. The focus was on the diversity and implications of patient replies rather than their prevalence.

Results

Among 28 456 patients who responded to 38 395 automated text messages (16 268 female [57.2%]; 11 692 [41.1%] aged 65 years or older), 743 unique responses were analyzed. Sixteen message types were identified that extended beyond the system’s recognized commands, grouped into 8 overarching domains: opt-out or opt-in requests, appointment management, requests for help or information, communication preferences, corrections of inaccurate data, comments on usefulness and usability, expressions of frustration, and uninterpretable responses. Replies reflected expectations for bidirectional communication, highlighted errors in patient information, or pointed to usability barriers. These communications underscored limitations of exclusively automated processing and revealed opportunities to make communication systems more responsive.

Conclusions and Relevance

In this qualitative study of patient responses to automated text messages, patients used these systems as an opportunity for bidirectional communication, revealing needs that extend beyond current system capabilities. Addressing these responses could reduce communication errors, improve appointment management, and strengthen trust in health care communication.

Introduction

Health care systems increasingly use automated text messages, interactive voice response (IVR) calls, and emails to deliver standardized information to large patient populations.¹ Automated messaging is widely used for appointment reminders, preventive outreach, and population health management, and can streamline processes and improve patient care outcomes.^2,3

However, current automated messaging platforms are typically programmed to recognize only a narrow set of replies,⁴ most commonly simple commands to stop or start messages that are in compliance with regulations mandating opt-out options.⁵ In contrast, patients may view outreach texts as invitations to converse with their clinician or health care system, and may respond in unanticipated ways that fall outside these limited reply options. In a study⁶ of a posthospital discharge text-messaging program, some patients replied with unprompted texts, triggering an error message from the software program (eg, “I don’t understand that response. Valid choices are: Y, Yes, N, or No”). Misinterpretation could also occur in a case where a patient might reply to an automated lab reminder with a question like, “Can I stop by without an appt?” which, because it contains the word “Stop,” could prevent future texts.

Automated processing of replies may lead to accidental removal from messaging lists, unaddressed needs, and patient frustration. These unexpected responses also provide insight into how patients interpret automated communication. From a sociotechnical perspective, these responses reflect the interaction between system design and patient behavior and may signal gaps in how systems engage patients as active participants in their care.⁷

Prior research on automated messaging in health care has largely focused on the effectiveness of reminders and outreach on outcomes, such as appointment attendance, medication adherence, and postdischarge utilization.^8,9,10 Other work has examined bidirectional messaging systems or methods to classify patient responses,^11,12 and qualitative studies have explored their implementation within clinical workflows.¹³ However, these approaches generally rely on structured response pathways or focus on system implementation rather than the content of patient communication itself. As a result, little prior research has characterized patient responses to automated messages when systems are not designed to accommodate unstructured input or how these responses reflect broader expectations for care.

To address this gap, we conducted a qualitative content analysis of patient messages in reply to automated text messages within an integrated health care delivery system. Our goal was to characterize the range and diversity of patient communications and inform more responsive message handling. By capturing the full spectrum of patient messages, including infrequent but important communications (eg, notification from a family member that a patient has died), we aimed to identify opportunities for automated systems to respond more effectively.

Methods

Study Design and Setting

We performed a qualitative content analysis of patient text message replies sent to an automated outreach system at Kaiser Permanente Colorado (KPCO), an integrated health system serving more than 500 000 members. Automated texts are used for appointment reminders and preventive care outreach.¹ Within this system, approximately 4.4 million automated text messages are sent annually to more than 500 000 patients, twice the volume of emails sent. The study period spanned January 1, 2022, through December 31, 2023. All messages included opt-out instructions (eg, “Text STOP to stop”) and were sent to all adult members who had not previously opted out.

The KPCO institutional review board designated this project as quality improvement not involving human participants research; informed consent was not required. Because the data were deidentified, consent for publication or message excerpts was also not required. This study followed Standards for Reporting Qualitative Research (SRQR) reporting guideline.¹⁴

Automated Messaging

Incoming patient replies were processed into 1 of 4 categories using predefined keywords: “stop” (including variants, such as “wrong,” “end,” “quit,” and “unsubscribe”), “start” (ie, resume messaging), “cancel” (ie, responses to appointment routed to clinical departments), and no action (ie, responses not recognized as valid commands). For behavioral health appointment reminders, patients were instructed that replying “cancel” would cancel their appointment. Messages not recognized by the system received an automated response directing patients to the call center.

Data Collection

We retrieved all patient replies to automated messages during the study period, along with the system’s automated classification. Identical messages sent by multiple patients were deduplicated and treated each as a single unique message for coding purposes. For example, the reply “wrong number” appeared frequently in the dataset but was included once in our qualitative sample. All identifying information was removed. Messages from other systems (eg, national Kaiser marketing, pharmacy) were excluded because they were generated from a separate automated system and corresponding response data were not accessible.

Content Analysis

We conducted an inductive qualitative content analysis to identify themes and categories of patient communications.¹⁵ Throughout 2024, a subset of the study team (S.M. and C.K.) conducted primary coding and thematic analysis, with input from study team members with expertise in health services research (J.F.S. and C.A.S.) and clinical operations (A.N.D. and J.J.). The team reviewed a subset of messages to develop inductive codes reflecting diverse patient communications.

Coding was conducted iteratively in batches. Coders reviewed subsets of messages and met regularly to compare interpretations, resolve discrepancies, and refine the codebook. Multiple codes were assigned when messages contained more than 1 theme. Through iterative team discussions, we finalized a framework of patient response categories to encompass all identified themes and organize our results.

Sampling and Saturation

We used a purposeful sampling strategy to maximize variation in response type rather than representativeness to ensure that messages were coded and themes were derived across a wide range of responses. Deduplicated messages were stratified by the 4 system response groups (“stop,” “start,” “cancel,” and “no action”) and reviewed in random order within each group. Coding continued until inductive thematic saturation was reached, defined as the point when no new response types or themes were identified in subsequent groups of analyzed messages.¹⁶ In total, 743 unique patient messages were analyzed (Table 1).

Table 1. Number of Messages Received and Health Care System Responses.

Health care system automated response to message	Messages, No. (%)
	Total messages (N = 38 395)	Unique responses (n = 8630)	Unique responses coded to saturation (n = 743)a
Stop	13 687 (36)	470 (5)	264 (56)
Start	2910 (8)	52 (1)	52 (100)
Cancel	5335 (14)	781 (9)	86 (11)
No action	16 463 (43)	7327 (85)	341 (5)

^a Denominator for this column is number of unique responses.

Our sampling method was designed to capture the diversity of patient responses and themes rather than to estimate prevalence,¹⁷ as even a single response could identify important improvements to the automated system.^18,19 This approach enabled identification of both frequent and infrequent but operationally consequential response types.

Development of System Level Considerations

After identifying response types, the study team developed potential considerations to guide system improvements. These were generated through iterative discussions among a multidisciplinary team with expertise in qualitative research (S.M. and C.K.), health services research (S.M., C.A.S., and J.F.S.), population health (A.N.D.), health care operations (A.N.D.), and the manager of the automated messaging systems (J.J.). These discussions were informed by more than a decade of collaborative research and operational work in automated communication strategies within integrated health care systems, including randomized trials of outreach and reminder systems, studies of modality choice and adherence, analyses of opt-out behavior, and conceptual work on patient-centered communication ecosystems.^1,9,10,20,21 Additional insights were drawn from prior qualitative interviews, patient experience surveys, and experience within the health care system.

Proposed considerations were further refined through discussions with KPCO operational leaders and front-line staff involved in communication strategy, implementation, and system monitoring. This process was interpretive and intended to contextualize findings for operational implementation.

Data Analysis

We used electronic health record data to summarize the characteristics of patients who sent at least 1 reply (N = 28 456), including demographics, clinical comorbidities,²² and prior health care utilization. Race and ethnicity were self-reported by patients in the electronic health record. Data were analyzed from January to November using Microsoft Excel (Microsoft 365).

Results

Among 28 456 patients who responded to automated text messages between January 1, 2022, and December 31, 2023 (16 268 female [57.2%]; 11 692 aged 65 years or older [41.1%]), we received 38 395 total messages. As shown in Table 1, patient responses spanned multiple system defined categories, with the majority classified as “no action,” indicating responses that were not recognized by the automated system. Table 2 provides descriptives for patients who sent at least 1 reply to an automated text message. Respondents spanned a wide age range, with higher representation among adults aged 65 years and older and included patients with diverse insurance coverage, racial or ethnic backgrounds, and active portal accounts.

Table 2. Demographic Characteristics of Patients Who Sent Messages From January 1, 2022, to December 31, 2023 (N = 28 456).

Characteristics	Messages, No. (%)
Age, y
18-34	4465 (15.7)
35-54	6988 (24.6)
55-64	5089 (17.9)
65-79	8762 (30.8)
80 or older	2930 (10.3)
Missing	222 (0.8)
Sex
Female	16 268 (57.2)
Male	11 962 (42.0)
Missing	226 (0.8)
Race or ethnicity
Asian	920 (3.2)
Black	1329 (4.7)
Hispanic	4040 (14.2)
Multiple race or ethnicity	452 (1.6)
Pacific Islander	91 (0.3)
White	18 289 (64.3)
Missing	222 (0.8)
Unknown	3020 (10.6)
Insurance payer
Deductible or coinsurance	6385 (22.4)
High deductible	1861 (6.5)
Traditional HMO	953 (3.4)
Medicaid	2094 (7.4)
Medicare	12 059 (42.4)
Other	4854 (17.1)
Not enrolled in Kaiser Permanente	250 (0.9)
Online patient portal account
Yes (active)	24 034 (84.5)
No (not active)	1710 (6.0)
No patient portal activity found	2712 (9.5)

Abbreviation: HMO, health maintenance organization.

We analyzed 743 unique text responses. While some replies were anticipated, such as requests to stop or restart messaging, many reflected unsolicited communications that extended beyond the system’s programmed recognition. Through inductive analysis, we developed a framework of 16 patient responses in 8 overarching domains (Table 3). Because responses were deduplicated and sampled to capture variation, the distribution of themes in the analytic sample does not reflect the underlying prevalence of response types.

Table 3. Framework of Message Domains and Types, Definitions, and Representative Examples.

Message domain or theme and definition	Definition	Representative message
Opt-out or opt-in: requests to stop or start receiving messages	Stop: a simple request to stop future messages, sometimes indicating desire to opt-out of specific types of messages	Stop texts.
	Start: a simple request to start messages. Certain instances requested a restart after an accidental opt-out	Start. I meant to stop the message today but not altogether.
Appointment management: confirming, canceling, or rescheduling upcoming care events	Confirm: attempting to confirm appointment referenced in appointment reminder	Stop will be there
	Cancel or reschedule: request to cancel or reschedule upcoming appointment	I will need to cancel my appointment. I have the flu.
		Cancel unable to attend thank you; I like to rescheduled [sic] please.
Request for help and information	Requesting care assistance: patients asking about additional care needs, such as medication refill or information about a procedure	I also need a lung xray, can I get that too?
	Requesting additional information about care: patients ask questions about upcoming visits, care instructions, or more information about care	I received a appt reminder but don’t know what it’s about. Please help me. Thank you.
		Can you send me the address?
Preference requests for message language, mode, and frequency	Expresses language preference	Please send messages in English.
	Expresses preference for message to be delivered through different communication channel	Can you send email? I cannot find this on phone.
	Expresses desire for a different number of messages, often fewer	Six messages are too many to remind me of my appointment. Please limit text messages.
Obtain and use personal information correctly: information misused or outdated	Not personalized: members assume health care system knows and uses up-to-date information about them to send correct and appropriate messaging	Should have been cancelled; I canceled I had to go to urgent care.
	Data incorrect: attempt to correct a data error issue, such as a wrong number or name misspelling, or message sent to person who is no longer a member	Not the correct number; wrong customer
		I am no longer employee and will have to cancel
	Deceased: family member of deceased member writing back to notify health care system they have died and to stop messages	Can you please stop the text he’s no longer with me he passed away thank you
Usefulness: utility and usability issues related to the usefulness of a message	Usability: technical errors prevent action, or messages cannot be acted on because of individual circumstances	That number doesn’t work, it just hangs up
		I am not going to make this appt, I’m to far now, to go without money for transportation, so need to cancel this
	Utility: question the value of content in the message	COVID? Those don’t work! Stop pushing useless experimental vaccines!
Expression of frustration	Dissatisfaction with health system or care experience	Do not want spend any more money with you people you’re program socks [sic] some was good but most was not
Uninterpretable	Messages that are uninterpretable: auto replies, reactions (eg, emojis), or unrelated messages	[Loved] ‘you are due for cervical cancer screening...’
		Good morning …happy Easter to you and your family!!!!

Opt-Out or Opt-In

Many messages involved requests to manage future communication, often expressed with single-word commands (“stop” or “start”). However, some members attempted to tailor their preferences to specific campaigns or situations, such as opting out of reminders for one day rather than ending all outreach: “Start. I meant to stop the message today but not altogether.”

Appointment Management

Patients responded to automated messages to confirm, cancel, or reschedule their appointments. Some patients included additional information about the reasons for cancellation: “I will need to cancel my appointment. I have the flu.”

Request for Help and Information

Members asked for further assistance or information. These messages varied from general pleas (“please help me”) to specific requests (“I also need a lung xray, can I get that too?”). Some replies indicated that system generated messages did not have enough information for members to act on. Members asked for additional details about their care appointments, such as location, content of the visit, or the necessity of an appointment.

Communication Preferences

Members shared comments that highlighted an expectation that the system could provide personalized messages. Specifically, members noted their preferences for specific wording, the channel through which they wanted to receive messages, and the desired number of messages they desire for different topics.

Accuracy and Use of Personal Information

Members replied to communications with information they assumed the health care system should have but was missing or used incorrectly. These responses included communications to deceased members, incorrect contact information (phone numbers, misspelled names), or messages to individuals who were no longer insured by the health care system. Additionally, replies challenged the accuracy and timeliness of the data used to trigger messages. For example, replying “she got her vaccines already” in response to a vaccine reminder message.

Utility and Usability Issues

Members commented on the usefulness of the messages. Respondents noted disinterest in receiving more messaging about influenza or COVID-19 vaccination. Members also highlighted technical issues as barriers to follow through on the message, such as broken links or incorrect contact numbers. They also described instances when social determinants of health, like finances or transportation, limited their ability to act on a message.

Expression of Frustration

A subset of messages conveyed broader frustration or dissatisfaction with service costs, negative experiences with care or customer service, or specific topics, such as COVID-19 vaccines. In some instances, members expressed anger without further elaborating on their experience.

Uninterpretable

There were also replies where we could not categorize the sender’s intentions. Example messages include those generated by automated phone settings for unavailable members (“I’m driving—sent from my car”) or visual symbols (emojis), such as “love” or “dislike” that recipients used to show how they felt about messages they received. Finally, some replies were unintelligible (“you would have been out here already”) or contained irrelevant information (“red just finished for today”). Together, these domains illustrate the breadth of patient responses and informed a set of system-level considerations for improving automated messaging, summarized in Table 4.

Table 4. Potential System-Level Considerations and Barriers or Implementation Challenges Based on Thematic Area.

Thematic area	Potential system-level considerations	Potential barriers or implementation challenges
Enhance personalization and usability	Implement campaign- or topic-specific opt-outs to avoid accidental universal opt-outs.	Coordinating nuanced opt-out mechanisms across multiple campaigns and vendors.
	Gather and honor member preferences for message frequency, channel, and content.	Balancing individualized preferences with large-scale communication campaigns.
	Regularly review message content for clarity, tone, and technical accuracy (eg, links, phone numbers).	Ongoing usability testing and feedback collection require staff time and operational resources.
	Incorporate patient testing and feedback into message design.
Ensure data accuracy and trust	Regularly verify and update patient contact information, appointment data, and health record links.	Real-time synchronization across multiple data systems is technically complex and resource intensive.
	Integrate with death records and external data sources to prevent distressing or inaccurate messages.	Delays in external record updates (eg, hospitalization records, death records) may still produce outdated messages.
	Strengthen internal processes for error reporting and correction across communication systems.	Requires sustained investment in data governance and IT infrastructure.
Build responsive systems	Increase system responsiveness and escalation pathways.	Developing and maintaining AI/NLP tools is costly and requires continual oversight to ensure accuracy and patient safety.
	Add direct links for appointment rescheduling or confirmation.	Integration with scheduling and clinical systems demands coordination across departments.
	Implement AI-assisted chatbot to interpret patient messages and route to appropriate resources or staff.	Manual review processes add workload and may delay responses.
	Flag frustrated or negative replies for review by member experience teams.	Crafting a default message that is empathetic, clear, and universally applicable without sounding dismissive or generic requires careful design and testing.
	Provide a default general response message that acknowledges potential system errors and includes a link to a resource space for additional support.
	Combine automated and human oversight to ensure timely, empathetic responses.

Abbreviations: AI, artificial intelligence; IT, information technology; NLP, natural language processing.

Discussion

Our analysis suggests that patients often use automated messages as opportunities for bidirectional communication, sending replies that extend beyond expected opt-out commands. Given the widespread use of automated messaging, these patterns likely reflect broader patient-system interactions. In high-volume systems, even infrequent response types may represent substantial numbers of patient interactions and reflect recurring patterns rather than isolated cases.

Prior research has largely focused on the effectiveness of reminders and outreach in improving outcomes such as appointment attendance and medication adherence,^2,3,23 or on structured bidirectional messaging systems and classifications of patient SMS responses.^6,11 In contrast, our findings highlight how patients respond to automated message programs that were not designed to accommodate unstructured messages. Rather than providing predefined or easily classifiable responses, patients frequently used these messages to communicate diverse needs, preferences, and concerns, revealing a broader range of communication behaviors than had been described in prior work. These findings highlight automated messaging as a sociotechnical system shaped by the interaction between technical design and patient interpretations,⁷ and underscore the limited guidance available for interpreting or responding to these diverse communication patterns.

Enhance Personalization and Usability

Patients’ responses reflected a desire for more personalized and flexible communication. Unintended opt-outs were identified across multiple message types, including cases in which patients attempted to stop messages for a single appointment but instead stopped all communication. These patterns suggest that current opt-out mechanisms may not align with patient expectations and that more flexible options could reduce accidental removals and improve engagement.² One potential approach would be to incorporate manual review of messages containing keywords that could trigger accidental opt-outs. Implementing more flexible opt-out mechanisms would require coordination across multiple communication platforms and resources but may improve patient experience and system performance.

Patients also expressed preferences for the way messages were delivered and how often they received them. Some indicated frustration with texts they saw as irrelevant or excessive, while others requested alternative delivery channels, such as email. These comments reflect a broader expectation for adaptable communications, although implementing individualized preferences across a diverse population may present logistical challenges,²⁴ and personalized messaging systems have been associated with increased patient retention and adherence.^23,25

Finally, messages highlighted usability concerns: broken links, unclear instructions, or barriers to accessing the health care described in the messages touching on social determinants of health, such as transportation and cost. These findings suggest that incorporating user testing and feedback loops into message development and directing patients to appropriate resources may enhance patient comprehension, compliance, and satisfaction with health care instruction.²⁶

Ensure Data Accuracy and Trust

We observed that patients identified or corrected inaccuracies in automated messages. Replies revealed messages sent to former members, those with outdated information, such as prior appointments or incorrect names, and in a few tragic instances, deceased individuals. These errors undermine trust in the system and highlight the importance of accurate underlying clinical and administrative data used for automated outreach, consistent with patients’ reports on the potential harmful and traumatizing effects of routine automated messaging.²⁷ Although the messaging system draws from near real-time data feeds, inaccuracies may persist when source information is incomplete or outdated.

These findings highlight the importance of accurate data. Integration with death records and real-time updates from electronic health records could help reduce errors, although such changes may be constrained by limitations in external data source availability and system interoperability. Addressing these challenges may reduce errors and improve patient trust in automated outreach.

Build Responsive Systems

Patients used automated messages to ask questions, manage appointments, or request assistance, often expecting a response. Current automated systems are generally not designed to accommodate this type of interaction, resulting in unacknowledged or misclassified messages. Some responses included requests for rescheduling or confirmation, while others sought clinical guidance or additional information. These patterns suggest an unmet expectation for bidirectional communication and increased transparency.

Integrating more responsive system functions, such as rescheduling links or automated confirmations, may improve the user experience. Artificial intelligence (AI) driven chatbots may offer an approach to improving interaction quality by providing relevant answers or directing patients to appropriate resources. Prior research suggests that incorporating AI in health care systems can assist in patient communication, reduce the volume of messages clinicians need to address,²⁸ improve patient satisfaction, and reduce demand on clinician time by automating routine interactions.^29,30 However, these tools require oversight to ensure safety, accuracy, and data security.

These findings also relate to broader concerns about transparency and trust in patient-facing digital tools. Prior work has shown that patient acceptance of automated communication systems depends in part on how clearly systems convey their capabilities and limitations.³¹ Our findings suggest that similar concerns apply even to simpler automated messaging systems, where patients may assume that messages are reviewed by a human or that their responses will be acknowledged.

Patient responses that conveyed frustration, anger, or disengagement may signal gaps between system design and patient expectations that persist when unrecognized. These messages offer important feedback about patient experience. Analysis of message sentiment within automated systems may help identify patients whose negative experiences should be addressed. Although natural language processing tools may support this process,³² implementation is complex and requires ongoing oversight.

These considerations may also extend to access and equity. Automated text messaging may serve as a primary access point for some patients. The ability of systems to respond to unstructured input may therefore affect inclusion, as system design can either mitigate or exacerbate disparities depending on how well it accommodates diverse patient needs.³³

Limitations

This study has limitations. This study reflects a single integrated health system, limiting generalizability. Systems that outsource automated messaging to commercial vendors may have less access to underlying data and fewer opportunities to refine message content.

Because our sampling strategy prioritized variation over representativeness, the distribution of response types in the analytic sample is not reflective of their prevalence among all messages. As a result, we are unable to estimate prevalence or prioritize response types based on occurrence. Future work should quantify the prevalence and impact of these response types in representative datasets.

Given the scale of this qualitative analysis, we could not validate findings directly with patients. Future work should incorporate patient engagement approaches to confirm interpretations. Finally, the system-level considerations were developed through interpretive synthesis rather than a formal analytic method, which may limit their generalizability.

Conclusions

In this qualitative study of patient responses to automated text messages, patients used these systems as opportunities for bidirectional communication. These responses revealed unmet needs and data inaccuracies, suggesting that automated messaging systems may not fully align with patient expectations and should account for a broader range of patient inputs. Addressing these considerations may improve communication and reduce errors.

Supplement.

Data Sharing Statement