Abstract
Introduction:
Missed clinic appointments (“no-shows”) waste health system resources, decrease physician availability, and may worsen patient outcomes. Appointment reminders reduce no-shows, though evidence on the optimal number of reminders is limited and sending multiple reminders for every visit is costly. Risk prediction models can be used to target reminders for visits that are likely to be missed.
Methods:
We conducted a randomized quality improvement project at Kaiser Permanente Washington among patients with primary care and mental health visits with a high no-show risk comparing the effect of one text message reminder (sent 2 business days prior to the appointment) with 2 text message reminders (sent 2 and 3 days prior) on no-shows and same-day cancellations. We estimated the relative risk (RR) of an additional reminder using G-computation with logistic regression adjusted for no-show risk.
Results:
Between February 27, 2019 and September 23, 2019, a total of 125,076 primary care visits and 33,593 mental health visits were randomized to either 1 or 2 text message reminders. For primary care visits, an additional text message reduced the chance of no-show by 7% (RR = 0.93, 95% CI: 0.89–0.96) and same-day cancellations by 6% (RR = 0.94, 95% CI: 0.90–0.98). In mental health visits, an additional text message reduced the chance of no-show by 11% (RR = 0.89, 95% CI: 0.86–0.93) but did not impact same-day cancellations (RR = 1.02, 95% CI: 0.96–1.11). We did not find effect modification among subgroups defined by visit or patient characteristics.
Conclusion:
Study findings indicate that using a prediction model to target reminders may reduce no-shows and spend health care resources more efficiently.
Keywords: appointments, attendance, behavioral health, health care delivery, prediction, primary care, reminder systems
Introduction
Missed clinic appointments result in a waste of health system resources as well as a missed opportunity for patients to receive care. Missed visits also decrease the availability of health care physicians to see other patients, especially in settings where there are a limited number of available appointments. Thus, use of health system resources, as well as patient care, can be improved by reducing the number of missed clinic appointments (also frequently referred to as “no-shows”).1–5
Health system interventions may reduce missed appointments. Recent meta-analyses found evidence that reminders reduce no-shows for primary care visits.6,7 More specifically, text messages and phone calls (including live calls, automated messages, and interactive voice response calls) have been shown to reduce no-shows for primary care visits.6, 8–11 There is also limited evidence that 2 or more reminders are more effective than a single reminder in reducing missed visits.6,12
Rather than provide additional phone or text reminders for all visits, risk prediction models can be used to target interventions for visits that are most likely to be missed and, in so doing, reduce the total cost of reminder systems.6, 9, 11–13 For example, Steiner et al developed a 10-variable no-show prediction model that had high discrimination (area under the curve = 0.9) for visits at Kaiser Permanente Colorado.9 In a subsequent trial comparing 2 reminders (delivered via interactive voice response call or text message, per patient preference) to a single reminder, Steiner et al found that the additional reminder reduced missed appointments among visits with the highest no-show risk.12 Shah et al also conducted a randomized trial to reduce no-show rates at the Massachusetts General Hospital primary care clinics among visits predicted to be at high chance of no-show by a prediction model estimated with earlier visits at the same practices.13 They found that reminder phone calls from patient service representatives reduced no-shows compared to automated reminder phone calls.
Epic, the developer of a widely-used electronic health record system, has developed a proprietary algorithm for predicting no-shows that health systems can use to guide interventions to reduce the chance of no-shows or to assist in overbooking and maximize the use of clinical resources.14 We conducted a randomized quality improvement project at Kaiser Permanente Washington (KPWA) to compare the effect of sending 1 versus 2 text message reminders to patients with visits predicted to be at high chance of no-show by the Epic algorithm. Outcomes were no-shows (defined as appointments that the patient did not attend and did not cancel prior to the visit) and same-day cancellations (defined as when the patient canceled the visit on the day it was scheduled to occur.) We examined the effect of an additional text message reminder on missed appointments and same-day cancellations in both primary care and mental health visits.
Methods
Study Setting
The quality improvement study took place at KPWA, a health care system with more than 700,000 members in the state of Washington. About two-thirds of members receive comprehensive care at Kaiser Permanente medical facilities. KPWA has 33 care locations across the state which offer primary and/or specialty care, plus 4 additional facilities that offer specialty care such as eye care and mental health. Members can schedule visits over the phone with assistance from member access representatives or online through KPWA’s patient portal.
At the time of the study, KPWA members received a single text reminder for all scheduled clinic visits 2 business days before the appointment. Members may opt out of receiving text messages at any time. This study examined whether sending an additional text message 3 business days before the appointment for visits that are estimated to have a higher risk for no-show would reduce the rate of same-day cancellations and no-shows for outpatient primary care and mental health visits. No-show risk was estimated using a prediction model provided by Epic, the electronic health record software used by KPWA. No-show predictions are generated every night within Epic for all visits scheduled for the following week, and predictions are generated using the Epic proprietary model which takes as input the members’ demographic characteristics, the number of previous no-shows in the prior year, and visit characteristics (including how many days in advance the visit was scheduled and whether the visit is with the patient’s designated primary care practitioner). The study was conducted in partnership with health system leaders in the hopes of identifying effective ways to decrease unused primary care and mental health appointments.
Intervention Design
We conducted a 1:1 pragmatic randomized study for visits scheduled between February 27, 2019, and September 23, 2019, to assess whether an additional text message improved same-day cancellation and no-show rates for mental health and primary care appointments. Visits met the inclusion criteria if they were scheduled 4 business days or more in advance and were classified as “high-risk” for no-show, defined as an Epic no-show risk in the top 40% of risk predictions for that type of visit (because the distribution of no-show risk was different between primary care and mental health visits).
Operational partners at KPWA chose to focus the study on visits in the top 40% of risk based on budgetary considerations and satisfactory prediction model performance. At the 40th percentile, sensitivity was 64% and positive predictive value was 9% in 364,940 primary care visits from May 1, 2018 to October 20, 2018. In 76,577 mental health visits from the same time, sensitivity was 62% and positive prediction was 20% at the 40th percentile. The 40th percentile thresholds were defined in this retrospective sample of visits. Visits with no-show risk above 5.1% for primary care visits and 21.1% for mental health visits were classified as high risk.
High-risk visits were randomized to either receive a text message 2 business days before the appointment (standard of care) or receive a text message 3 business days before the visit, in addition to the usual text message reminder 2 business days before the visit (intervention). All text messages were sent in English. For both groups, patients who had opted out of the reminder system did not receive any text messages; the study team did not have access to data on which members opted out of text message reminders. Upon receipt of the text message, members were asked to confirm the visit, or if they were unable to attend, to cancel by phone or online. Members with visits in the intervention arm who confirmed following the first text message reminder (3 business days before the appointment) but prior to the standard text message reminder (2 business days before the appointment) did not receive the standard text message reminder. Because randomization took place at the visit level, members with multiple eligible visits during the study period may have had visits randomized to both study arms. The KPWA Institutional Review Board determined that this project was a quality improvement (not research) project and, therefore, did not require Institutional Review Board oversight.
Statistical Methods
The primary and secondary outcomes of interest were the rate of no-shows and same-day cancellations, respectively. We estimated the intervention’s effect on each outcome as follows. First, we estimated the conditional effect of the intervention using a logistic regression model. To increase precision, logistic regression models were also adjusted for each visit’s predicted no-show risk, as estimated by the Epic prediction model.15 We used generalized estimating equations with a working independent correlation structure to account for the clustering of visits within individuals in our study.16 To obtain an unconditional estimate of the intervention’s effect (rather than effect conditional on predicted no-show risk), we used G-computation to obtain a marginal relative risk estimate of the intervention’s effect on each outcome (averaged over the distribution of the no-show risk predictions for all visits in the study).17 The advantage of the marginal estimate is that the interpretation of the parameter of interest is unconditional with respect to the no-show risk and has more precision than an unconditional estimated relative risk.15 Finally, we obtained the corresponding 95% confidence intervals of the marginal effects via bootstrapping the observations at the visit level to reflect the expected variability in the population of visits if implementing the prediction model in a clinical setting.18
Additionally, we conducted exploratory analyses to assess whether the intervention had different effects within subgroups, also known as heterogeneity of treatment effects, defined by patient or visit characteristics. For these analyses, we examined patient and visit characteristics that prior research suggests might be associated with missed appointments such as the day and time of the scheduled visit, how far in advance the visit was scheduled, and patient characteristics, such as patient age.9 Subgroups that had a prevalence of less than 1% were removed from this analysis. To estimate the subgroup effect on each outcome, we adjusted for the intervention arm, the Epic risk prediction, the subgroup variable, and the interaction of the intervention with the subgroup in a generalized estimating equation model with logistic link. We tested for effect modification at the 0.05 type I error level using an analysis of variance test. Using G-computation we estimated the marginal effect of the intervention within subgroups and obtained their 95% confidence intervals via the bootstrap.
Results
A total of 390,064 visits were scheduled over the 7-month study period, including 302,689 primary care visits and 87,375 mental health visits. A total of 125,076 primary care visits (41.3%) had a predicted no-show risk at or above 5.1% (40th percentile cut-off) and were randomly assigned 1 text message (n = 62,519) or 2 reminder text messages (n = 62,557). A total of 33,593 mental health visits (38.4%) had a predicted no-show risk at or above 21.1% and were assigned for randomization during the study period; 16,830 mental health visits were sent only one text message reminder and 16,763 were sent an additional text message. The percentage of visits of each type included in the study deviated slightly from 40% due to variation in the distribution of risk predictions between visits in the retrospective sample used to select cut-offs and those during the study period.
Table 1 shows the patient and visit characteristics for primary care and mental health visits included in the study. As expected, due to randomization, characteristics were balanced across the study arms in both primary care and mental health visits. Overall, demographic characteristics were similar across mental health and primary care visits, although mental health visits had a higher proportion of patients aged 18–30 and a lower proportion of patients with Asian or Asian American race indicated in their medical record. Visit characteristics such as the time or day of the week the appointment took place were also similar between the two visit types. However, lead time was higher in mental health visits, with 53% of the visits being scheduled more than a month in advance, in contrast to 19% of primary care visits.
Table 1.
Characteristics of primary care and mental health visits
| Variable | Category | Primary Care Visits | Mental Health Visits | ||
|---|---|---|---|---|---|
| Intervention | Control | Intervention | Control | ||
| 62,557 (50%) |
62,519 (50%) |
16,763 (50%) |
16,830 (50%) |
||
| Age (years) | < 18 | 15,133 (24%) | 15,318 (25%) | 2968 (18%) | 2925 (17%) |
| 18 – 29 | 11,830 (19%) | 11,717 (19%) | 5694 (34%) | 5612 (33%) | |
| 30 – 49 | 17,853 (29%) | 17,917 (29%) | 5391 (32%) | 5496 (33%) | |
| 49 – 64 | 12,637 (20%) | 12,678 (20%) | 2366 (14%) | 2443 (15%) | |
| > 64 | 5104 (8%) | 4889 (8%) | 344 (2%) | 354 (2%) | |
| Sex | Female | 38,212 (61%) | 38,320 (61%) | 10,906 (65%) | 10,881 (65%) |
| Male | 24,344 (39%) | 24,196 (39%) | 5857 (35%) | 5949 (35%) | |
| Neither female nor male indicated | 1 (0%) | 3 (0%) | NA | NA | |
| Hispanic Ethnicity | Hispanic | 7208 (12%) | 71,65 (11%) | 2102 (13%) | 2002 (12%) |
| Non-Hispanic | 51,263 (82%) | 51,265 (82%) | 13,709 (82%) | 13,899 (83%) | |
| Missing, Declined, or Do Not Know | 4086 (7%) | 4089 (7%) | 952 (6%) | 929 (6%) | |
| Race | American Indian or Alaskan Native | 1168 (2%) | 1,217 (2%) | 380 (2%) | 409 (2%) |
| Asian or Asian American | 6222 (10%) | 6360 (10%) | 894 (5%) | 915 (5%) | |
| Black | 5537 (9%) | 5593 (9%) | 1267 (8%) | 1233 (7%) | |
| Native Hawaiian or Pacific Islander | 831 (1%) | 810 (1%) | 100 (1%) | 78 (0%) | |
| White | 40,435 (65%) | 40,141 (64%) | 12,275 (73%) | 12,407 (74%) | |
| Other | 4225 (7%) | 4326 (7%) | 912 (5%) | 817 (5%) | |
| Missing, Declined or Do Not Know | 4139 (7%) | 4072 (7%) | 935 (6%) | 971 (6%) | |
| Copay Due | No copay | 42,417 (68%) | 42,180 (67%) | 10,979 (65%) | 10,909 (65%) |
| $1 – $15 | 8018 (13%) | 7964 (13%) | 2208 (13%) | 2206 (13%) | |
| $16 – $60 | 11,849 (19%) | 12,138 (19%) | 3525 (21%) | 3677 (22%) | |
| Missing | 273 (0%) | 237 (0%) | 51 (0%) | 38 (0%) | |
| Epic No-Show Risk Score | 5% – 10% | 39,676 (63%) | 39,654 (63%) | 0 (0%) | 0 (0%) |
| 11% – 30% | 19,875 (32%) | 19,797 (32%) | 6455 (39%) | 6505 (39%) | |
| 31% – 51% | 2446 (4%) | 2447 (4%) | 6887 (41%) | 6811 (40%) | |
| 51% – 96% | 560 (1%) | 621 (1%) | 3421 (20%) | 3514 (21%) | |
| Number of No-Shows in Year Prior to Visit | No missed visits | 31,205 (50%) | 31,312 (50%) | 4459 (27%) | 4459 (26%) |
| 1 or 2 missed visits | 15,070 (24%) | 14,849 (24%) | 5,872 (35%) | 5,950 (35%) | |
| 3–10 missed visits | 5100 (8%) | 5086 (8%) | 5312 (32%) | 5233 (31%) | |
| 10 or more missed visits | 494 (1%) | 446 (1%) | 959 (6%) | 1027 (6%) | |
| No visits in previous year | 10,688 (17%) | 10,826 (17%) | 161 (1%) | 161 (1%) | |
| Number of Same-Day Cancellations in Year Prior to Visit | No same-day cancellations | 32,039 (51%) | 32,489 (52%) | 5495 (33%) | 5533 (33%) |
| 1 same-day cancellation | 10,720 (17%) | 10,522 (17%) | 3805 (23%) | 3775 (22%) | |
| 2 same-day cancellations | 4210 (7%) | 4039 (6%) | 2424 (14%) | 2407 (14%) | |
| 3 or more same-day cancellations | 4900 (8%) | 4643 (7%) | 4878 (29%) | 4954 (29%) | |
| No visits in previous year | 10,688 (17%) | 10,826 (17%) | 161 (1%) | 161 (1%) | |
| Appointment Day | Monday | 14,092 (23%) | 14,047 (22%) | 3734 (22%) | 3772 (22%) |
| Tuesday | 13,573 (22%) | 13,580 (22%) | 3513 (21%) | 3544 (21%) | |
| Wednesday | 11,302 (18%) | 11,315 (18%) | 3432 (20%) | 3434 (20%) | |
| Thursday | 10,527 (17%) | 10,500 (17%) | 3258 (19%) | 3266 (19%) | |
| Friday | 11,666 (19%) | 11,658 (19%) | 2757 (16%) | 2755 (16%) | |
| Saturday | 1397 (2%) | 1418 (2%) | 69 (< 1%) | 59 (< 1%) | |
| Sunday | 0 (0%) | 1 (<1%%) | 0 (0%) | 0 (0%) | |
| Appointment Hour of Day | Morning (7AM–11:59 AM) | 32,934 (53%) | 32,877 (53%) | 6880 (41%) | 6950 (41%) |
| Midday (12 PM–3:59 PM) | 21,184 (34%) | 21,310 (34%) | 5965 (36%) | 5991 (36%) | |
| Evening (4 PM–6:00 PM) | 8439 (13%) | 8332 (13%) | 3918 (23%) | 3889 (23%) | |
| Appointment Lead time in Days a | Less than 8 | 14,339 (23%) | 14,280 (23%) | 1068 (6%) | 1055 (6%) |
| 8 to 14 | 15,464 (25%) | 15,442 (25%) | 2145 (13%) | 2138 (13%) | |
| 15 to 30 | 20,858 (33%) | 20,860 (33%) | 4937 (29%) | 4984 (30%) | |
| More than 30 | 11,896 (19%) | 11,937 (19%) | 8613 (51%) | 8653 (51%) | |
| PHQ-9 Scoreb (past year) | No PHQ-9 recorded | 34,739 (56%) | 35,033 (56%) | 1610 (10%) | 1682 (10%) |
| PHQ-9 <10 | 20,543 (33%) | 20,280 (32%) | 6433 (38%) | 6557 (39%) | |
| PHQ-9 ≥ 10 | 7275 (12%) | 7206 (12%) | 8720 (52%) | 8591 (51%) | |
| Had Urgent Care Visit in Past 30 Days | No | 59,233 (95%) | 59,242 (95%) | 15,872 (95%) | 15,899 (94%) |
| Yes | 3324 (5%) | 3277 (5%) | 891 (5%) | 931 (6%) | |
| Has MyChart c | No | 31,269 (50%) | 31,410 (50%) | 5526 (33%) | 5457 (32%) |
| Yes | 31,288 (50%) | 31,109 (50%) | 11,237 (67%) | 11,373 (68%) | |
Number of days in advance that the visit was scheduled.
Patient Health Questionnaire (PHQ-9) scores of 10, 15, 20 represent moderate, moderately severe, and severe depression, respectively.13
MyChart is Epic’s online patient portal.
Results (shown in Table 2) indicate that the intervention reduced the chance of no-shows in both primary care and mental health visits and reduced the chance of same-day cancellation in primary care visits. The estimated relative risk (RR) of no-show between the control and the intervention arm in primary care visits was 0.93 (95% confidence interval, CI, between 0.89 and 0.96). Thus, we estimate that the no-show risk for that visit will decrease by 7% if an additional text message is sent for any given primary care visit with a high estimated no-show risk. Sending an additional text message reminder also reduced same-day cancellations in primary care; the estimated RR was 0.94 (95% CI: 0.90–0.98). Among mental health visits, the estimated RR between intervention and control was 0.89 (95% CI: 0.86–0.93) for no-shows, indicating that an additional reminder reduced no-shows, and 1.02 (95% CI: 0.96–1.11) for same-day cancellations, indicating that the additional reminder had no impact.
Table 2.
No-show and same-day cancellation rates across study arms and relative risk
| Setting | Outcome | 1 text message | 2 text messages | Estimated Relative Risk (95% CI) |
|---|---|---|---|---|
| Primary Care N = 125,076 |
No-Show | 5683 (9.1%) | 5269 (8.4%) | 0.93 (0.89, 0.96) |
| Same-day Cancellation | 4202 (6.7%) | 3946 (6.3%) | 0.94 (0.90, 0.98) | |
| Mental Health N = 33,593 |
No-Show | 3386 (20.1%) | 3007 (17.9%) | 0.89 (0.86, 0.93) |
| Same-day Cancellation | 1640 (9.7%) | 1674 (10.0%) | 1.02 (0.96, 1.11) |
Finally, for each variable listed in Table 1, we found no evidence of heterogeneous treatment effects of the intervention on no-show rates in either primary care or mental health visits, that is, p > 0.05 for all the subgroup effect modification analysis of variance tests. For example, for the time of day of a scheduled visit, the estimated marginal relative chance of the additional text message reminder’s effect on missed visits for primary care were 0.93 (95% CI: 0.89–0.97) for visits in the morning, 0.94 (95% C: 0.89–1.00) for midday visits, and 0.89 (95% CI: 0.81–0.98) for evening visits. Table 3 shows the marginal relative risk estimates and their corresponding 95% CI for each of the subgroup analyses. We note that, because characteristics associated with no-show risk were selected for subgroup analyses, distribution of these characteristics among visits in the top 40% of risk vary from their distribution in the entire population of primary care and mental health visits at KPWA. Lack of heterogeneous treatment effects among high-risk visits does not imply that there would be no subgroup effect modification if additional text messages were sent to all visits.
Table 3.
Results of subgroup analyses for effect modification in no-show outcome
| Variable | Category | Primary Care Relative Risk (95% CI) |
Mental Health Relative Risk (95% CI) |
|---|---|---|---|
| Age | < 18 | 0.91 (0.83, 0.99) | 0.91 (0.80, 1.05) |
| 18–29 | 0.93 (0.86, 0.99) | 0.94 (0.87, 1.00) | |
| 30–49 | 0.99 (0.93, 1.05) | 0.82 (0.76, 0.89) | |
| 49–64 | 0.88 (0.81, 0.96) | 0.91 (0.82, 1.00) | |
| > 64 | 0.82 (0.69, 0.98) | 0.95 (0.72, 1.30) | |
| Sex | Female | 0.92 (0.88, 0.96) | 0.9 (0.85, 0.96) |
| Male | 0.94 (0.89, 1.00) | 0.87 (0.80, 0.94) | |
| Hispanic Ethnicity | Hispanic | 0.88 (0.80, 0.96) | 0.94 (0.83, 1.04) |
| Non-Hispanic | 0.93 (0.90, 0.96) | 0.88 (0.83, 0.93) | |
| Missing, Declined, or Do Not Know | 0.99 (0.87, 1.12) | 0.90 (0.75, 1.08) | |
| Race | American Indian or Alaska Native | 1.09 (0.90, 1.37) | 0.84 (0.66, 1.06) |
| Asian or Asian American | 1.00 (0.88, 1.13) | 0.89 (0.74, 1.09) | |
| Black | 0.89 (0.82, 0.98) | 0.88 (0.77, 1.02) | |
| White | 0.91 (0.87, 0.95) | 0.89 (0.85, 0.94) | |
| Other | 0.91 (0.82, 1.01) | 0.89 (0.77, 1.10) | |
| Missing, Declined, or Do Not Know | 0.99 (0.89, 1.13) | 0.93 (0.77, 1.15) | |
| Copay Due | No Copay | 0.93 (0.89, 0.96) | 0.88 (0.84, 0.93) |
| $1–$15 | 0.91 (0.84, 1.01) | 0.93 (0.79, 1.06) | |
| $16–$60 | 0.94 (0.86, 1.01) | 0.89 (0.80, 0.99) | |
| Epic No-Show Risk a | Lower risk | 0.91 (0.86, 0.98) | 0.87 (0.80, 0.93) |
| Higher risk | 0.93 (0.89, 0.97) | 0.91 (0.86, 0.95) | |
| Number of No-Shows in Year {rior to Visit | No missed visits | 0.91 (0.86, 0.96) | 0.85 (0.72, 0.96) |
| 1 or 2 missed visits | 0.93 (0.88, 0.98) | 0.89 (0.83, 0.96) | |
| 3–10 missed visits | 0.94 (0.86, 1.01) | 0.88 (0.82, 0.93) | |
| 10 or more missed visits | 0.82 (0.65, 0.99) | 0.99 (0.90, 1.10) | |
| No visits in previous year | 0.96 (0.87, 1.07) | 1.64 (1.03, 2.93) | |
| Number of Same-Day Cancellations in Year Prior to Visit | No same-day cancellations | 0.89 (0.85, 0.94) | 0.87 (0.79, 0.95) |
| 1 same-day cancellation | 1.00 (0.92, 1.09) | 0.93 (0.85, 1.00) | |
| 2 same-day cancellations | 0.94 (0.84, 1.06) | 0.89 (0.79, 1.00) | |
| 3 or more same-day cancellations | 0.87 (0.78, 0.96) | 0.87 (0.81, 0.94) | |
| No visits in previous year | 0.96 (0.87, 1.07) | 1.64 (1.01, 3.42) | |
| Appointment Day b | Monday | 0.92 (0.86, 0.99) | 0.89 (0.83, 0.97) |
| Tuesday | 0.92 (0.85, 1.00) | 0.88 (0.79, 0.97) | |
| Wednesday | 0.94 (0.86, 1.02) | 0.86 (0.78, 0.94) | |
| Thursday | 0.96 (0.88, 1.03) | 0.91 (0.82, 1.01) | |
| Friday | 0.91 (0.84, 0.98) | 0.92 (0.84, 1.04) | |
| Appointment Hour of Day | Morning (7 AM–11:59 AM) | 0.93 (0.89, 0.97) | 0.93 (0.85, 1.01) |
| Midday (12 PM–3:59 PM) | 0.94 (0.89, 1.00) | 0.90 (0.83, 0.96) | |
| Evening (4 PM–6:00 PM) | 0.89 (0.81, 0.98) | 0.83 (0.78, 0.90) | |
| Appointment Lead Time in Days c | Less than 8 | 0.96 (0.89, 1.04) | 1.01 (0.85, 1.20) |
| 8 to 14 | 0.93 (0.87, 1.01) | 0.83 (0.73, 0.94) | |
| 15 to 30 | 0.93 (0.88, 0.99) | 0.91 (0.85, 0.98) | |
| More than 30 | 0.89 (0.82, 0.97) | 0.88 (0.82, 0.93) | |
| PHQ-9 score ≥ 10 in past year d | 0 | 0.94 (0.89, 1.00) | 0.87 (0.81, 0.94) |
| 1 | 0.91 (0.83, 0.99) | 0.89 (0.83, 0.94) | |
| No PHQ-9 recorded | 0.92 (0.88, 0.97) | 1.01 (0.86, 1.19) | |
| Had Urgent Care Visit in Past 30 Days | No | 0.93 (0.89, 0.96) | 0.88 (0.84, 0.92) |
| Yes | 0.95 (0.83, 1.07) | 1.05 (0.88, 1.30) | |
| Has MyChart e | No | 0.95 (0.91, 1.01) | 0.89 (0.83, 0.95) |
| Yes | 0.89 (0.84, 0.94) | 0.89 (0.84, 0.94) |
Lower risk primary care visits had a predicted no-show risk from 5%–9%, and lower risk mental health visits had a predicted no-show risk form 21%–35%. Visits at a higher risk for no-show had predicted risks above 10% for primary care visits and 36% for mental health visits.
Saturday and Sunday were not included in subgroup analyses because less than 1% of visits were scheduled for those days.
Number of days in advance that the visit was scheduled.
Patient Health Questionnaire (PHQ-9) scores of 10, 15, 20 represent moderate, moderately severe, and severe depression, respectively. Source: 1. Kroenke K, Spitzer RL. The PHQ-9: a new depression diagnostic and severity measure. Psychiatric Annals, 2002; 32(9):509–515. DOI: https://doi.org/10.3928/0048-5713-20020901-06
MyChart is Epic’s online patient portal.
Discussion
An additional text message in advance of visits at high chance of being missed was effective in reducing no-shows in primary care and mental health visits, and in reducing same-day cancellations of primary care visits. Our findings suggest that if an additional text message was sent to 18,250 primary care visits with Epic no-show risk predictions in the top 40%—the approximate number of high-risk primary care visits per month at KPWA—about 126 fewer missed visits would occur than if higher risk visits received only a single text message. Similarly, the additional text message is estimated to result in an additional 72 fewer same-day cancellations of primary care visits per month. In mental health, among approximately 4900 monthly visits in the top 40% of risk, we estimate that 106 fewer no-shows would occur per month if an additional text message was sent to high-risk visits.
Results from the randomized study indicate that using a risk prediction model to target reminders may promote efficient use of health care resources. By sending an additional text message reminder to high-risk visits we achieved higher efficiency at lower costs to the health system (compared to sending a message to all visits) and limited unnecessary notifications to patients with a high likelihood of attending their scheduled visit.19 We note that one smaller study previously found patient satisfaction to be unaffected by sending an additional reminder.12 One limitation of this study is that we did not monitor for a change in KPWA member requests to opt out of text messages among members with visits in the intervention arm. Although a targeted intervention to reduce no-shows may save resources, this approach involves additional costs of implementing a risk prediction model within the system. Risk-based targeting of additional text message reminders can be further improved by developing a risk model tailored to a particular population. A more precise risk model could improve the effectiveness of the intervention and may also be used to inform more resource-intensive interventions (eg, live reminder phone calls by patient access representatives).
In this study, we used a no-show prediction algorithm provided within Epic and available to health systems with the applicable Epic licenses. The development and implementation of prediction models by electronic health system vendors have the potential to expand access to predictive analytic tools to health care physicians who do not have in-house analytic capabilities. But these algorithms are proprietary, which hinders transparent auditing of prediction model performance. We independently evaluated the performance of Epic’s model at KPWA before conducting this quality improvement project. One advantage of Epic’s no-show prediction model is that predictions can be calibrated within a health system to better fit the patient population; this feature is not available in many proprietary prediction models. Being able to independently assess an algorithm is crucial to ensuring that the implementation of prediction models provides accurate and equitable guidance in clinical settings.20
In this study, no-show risk was predicted for all KPWA primary care and mental health visits, and all visits identified as high risk were automatically randomized and included in the study. Because this study did not use additional exclusion criteria, findings in this study reflect the real-life impact of implementing this intervention for all high-risk visits. However, this study was performed in only a single health care system, and effectiveness in other health systems may differ. Nevertheless, we did not find substantial heterogeneity of treatment effects for the no-show outcome, thus, our results may indicate that the effect is generalizable to populations with different visit and patient characteristics. For example, an additional text message was found to decrease no-show risk at Kaiser Permanente Colorado as well as in Denver Health.12,21 Thus, our study complements previous results for a different integrated health system and future work should be carried out to examine additional health care systems and models of care.
Finally, it is worth noting that sending an additional text message does not address many of the barriers to health care access that may be driving no-shows and same-day cancellations. Interventions that address social determinants of health need to be further developed and studied. For example, future studies could explore ways to address the lack of transportation to an appointment or the need for backup childcare or eldercare.22 Sending an additional text message reminder may help complement more robust interventions to reduce missed visits. Text message reminders are a relatively inexpensive strategy to reduce no-shows; the value of targeting interventions to those at highest risk would be even greater for more resource-intensive interventions.
Conclusion
Missed clinic appointments can be reduced through appointment reminders sent via text messages. Our randomized study at KPWA examined whether sending an additional text message reminder could reduce the rate of missed appointments for visits with a high predicted no-show risk. We found that sending an additional reminder, compared to the current practice of a single reminder, reduced the chance of missed appointments among primary care and mental health visits with predicted high no-show risk. This study provides an example of how predictive analytics can be used to target interventions to improve health care delivery and more efficiently allocate health system resources to optimize outcomes.
Footnotes
Author Contributions: Ernesto Ulloa-Pérez, MS, participated in the analysis of data and drafting and critical review of the final manuscript. Paula R. Blasi, MPH, participated in the drafting and critical review of the final manuscript. Emily O. Westbrook, MHA, participated in the study design, acquisition and analysis of data, and critical review of the final manuscript. Paula Lozano, MD, MPH, participated in the study design and critical review of the final manuscript. Katie F. Coleman, MSPH, participated in the critical review of the final manuscript. R. Yates Coley, PhD, participated in the study design, acquisition and analysis of data, and drafting, critical review, and submission of the final manuscript.
Conflicts of Interest: None declared
Funding: This work was conducted as part of Kaiser Permanente Washington’s Learning Health System Program. Dr. Coley was also supported by the Agency for Healthcare Research and Quality (K12HS026369).
References
- 1.Daggy J, Lawley M, Willis D, et al. Using no-show modeling to improve clinic performance. Health Inform J. 2010;16((4)):246–259. DOI: 10.1177/1460458210380521. [DOI] [PubMed] [Google Scholar]
- 2.Moore CG, Wilson-Witherspoon P, Probst JC. Time and money: Effects of no-shows at a family practice residency clinic. Fam Med. 2001; 33((7)):522–527. [PubMed] [Google Scholar]
- 3.Karter AJ, Parker MM, Moffet HH, et al. Missed appointments and poor glycemic control: an opportunity to identify high-risk diabetic patients. Med Care. 2004; 42((2)):110–115. DOI: 10.1097/01.mlr.0000109023.64650.73. [DOI] [PubMed] [Google Scholar]
- 4.Horstmann E, Brown J, Islam F, Buck J, Agins BD. Retaining HIV-infected patients in care: where are we? Where do we go from here? Clin Infect Dis. 2010; 50((5)):752–761. DOI: 10.1086/649933. [DOI] [PubMed] [Google Scholar]
- 5.Sharp DJ, Hamilton W. Non-attendance at general practices and outpatient clinics. BMJ. 2001; 323((7321)):1081–1082. DOI: 10.1136/bmj.323.7321.1081. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Robotham D, Satkunanathan S, Reynolds J, Stahl D, Wykes T. Using digital notifications to improve attendance in clinic: systematic review and meta-analysis. BMJ Open. 2016; 6:e012116 DOI: 10.1136/bmjopen-2016-012116. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Peterson K, McCleery E, Anderson J, Waldrip K, Helfand M. Evidence brief: comparative effectiveness of appointment recall reminder procedures for follow-up appointments. In: VA Evidence Synthesis Program Evidence Briefs. [internet]. Washington (DC):: Department of Veterans Affairs (US);: 2011. [PubMed] [Google Scholar]
- 8.Gurol-Urganci I, de Jongh T, Vodopivec-Jamsek V, Atun R, Car J. Mobile phone messaging reminders for attendance at healthcare appointments. Cochrane Database Syst Rev. 2013; (12):Cd007458. DOI: 10.1002/14651858.CD007458.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Steiner JF, Shainline MR, Bishop MC, Xu S. Reducing missed primary care appointments in a learning health system: Two randomized trials and validation of a predictive model. Med Care. 2016; 54((7)):689–696. DOI: 10.1097/MLR.0000000000000543. [DOI] [PubMed] [Google Scholar]
- 10.Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of outpatient appointment reminder systems in reducing no-show rates. Am J Med. 2010; 123((6)):542–548. DOI: 10.1016/j.amjmed.2009.11.022. [DOI] [PubMed] [Google Scholar]
- 11.Percac-Lima S, Cronin PR, Ryan DP, Chabner BA, Daly, EA, Kimball AB. Patient navigation based on predictive modeling decreases no-show rates in cancer care. Cancer. 2015; 121((10)):1662–1670. DOI: 10.1002/cncr.29236. [DOI] [PubMed] [Google Scholar]
- 12.Steiner JF, Shainline MR, Dahlgren JZ, Kroll A, Xu S. Optimizing number and timing of appointment reminders: a randomized trial. Am J Manag Care. 2018; 24((8)):377–384. PMID: 30130032. [PubMed] [Google Scholar]
- 13.Shah SJ, Cronin P, Hong CS, et al. Targeted reminder phone calls to patients at high risk of no-show for primary care appointment: A randomized trial. J Gen Int Med. 2016; 31((12)):1460–1466. DOI: 10.1007/s11606-016-3813-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Murray SG, Wachter RM, Cucina RJ. Discrimination by artificial intelligence in a commercial electronic health record—a case study. Health Affairs Blog. 2020. Accessed April 8, 2020. https://www.healthaffairs.org/do/10.1377/forefront.20200128.626576/. [Google Scholar]
- 15.Steingrimsson, JA, Hanley DF, Rosenblum M. Improving precision by adjusting for prognostic baseline variables in randomized trials with binary outcomes, without regression model assumptions. Contemporary Clinical Trials. 2017; 54:18–24. DOI: 10.1016/j.cct.2016.12.026. [DOI] [PubMed] [Google Scholar]
- 16.Liang, K-Y, Zeger SL. Longitudinal data analysis using generalized linear models. Biometrika. 1986; 73((1)):13–22. DOI: 10.1093/biomet/73.1.13. [DOI] [Google Scholar]
- 17.Snowden JM, Rose S, Mortimer KM. Implementation of G-computation on a simulated data set: demonstration of a causal inference technique. Am J Epidemiol. 2011; 173((7)):731–738. DOI: 10.1093/aje/kwq472. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Efron B. The jackknife, the bootstrap and other resampling plans. Philadelphia: PA:: Society for Industrial and Applied Mathematics;; 1982. DOI: 10.1137/1.9781611970319. [DOI] [Google Scholar]
- 19.Steiner J, Zeng C, Comer AC, et al. Factors associated with opting out of automated text and telephone messages among adult members of an integrated health care system. JAMA Network Open. 2021; 4((3)): e213479. DOI: 10.1001/jamanetworkopen.2021.3479. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019; 366((6464)):447–453. DOI: 10.1126/science.aax2342. [DOI] [PubMed] [Google Scholar]
- 21.Fischer HH, Moore SL, Johnson TL, Everhart RM, Batal H, Davidsoni AJ. Appointment reminders by text message in a safety net health care system: a pragmatic investigation. eGEMs. 2017; 5((1)):20. DOI: 10.5334/egems.215. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Chaiyachati KH, Hubbard RA, Yeager A, et al. Association of rideshare-based transportation services and missed primary care appointments: A clinical trial. JAMA Intern Med. 2018;178((3)):383–389. DOI: 10.1001/jamainternmed.2017.8336. [DOI] [PubMed] [Google Scholar]