Abstract
Purpose
Between 23% and 34% of outpatient appointments are missed annually. Patients who frequently miss medical appointments have poorer health outcomes and are less likely to use preventive health care services. Missed appointments result in unnecessary costs and organizational inefficiencies. Appointment reminders may help reduce missed appointments; particular types may be more effective than other types. We used a survey with a discrete choice experiment (DCE) to learn why individuals miss appointments and to assess appointment reminder preferences.
Methods
We enrolled a national sample of adults from an online survey panel to complete demographic and appointment habit questions as well as a 16-task DCE designed in Sawtooth Software’s Discover tool. We assessed preferences for four reminder attributes – initial reminder type, arrival of initial reminder, reminder content, and number of reminders. We derived utilities and importance scores.
Results
We surveyed 251 adults nationally, with a mean age of 43 (range 18–83) years: 51% female, 84% White, and 8% African American. Twenty-three percent of individuals missed one or more appointments in the past 12 months. Two primary reasons given for missing an appointment include transportation problems (28%) and forgetfulness (26%). Participants indicated the initial reminder type (21%) was the most important attribute, followed by the number of reminders (10%). Overall, individuals indicated a preference for a single reminder, arriving via email, phone call, or text message, delivered less than 2 weeks prior to an appointment. Preferences for reminder content were less clear.
Conclusion
The number of missed appointments and reasons for missing appointments are consistent with prior research. Patient-centered appointment reminders may improve appointment attendance by addressing some of the reasons individuals report missing appointments and by meeting patients’ needs. Future research is necessary to determine if preferred reminders used in practice will result in improved appointment attendance in clinical settings.
Keywords: reminders, discrete choice experiment, no-show rates, text messaging, survey, preferences, appointment attendance, DNAs, conjoint analysis
Introduction
As the US health care system continues to focus on population-based, value-driven care, it is essential for primary care providers and health care organizations (HCOs) to get patients in the clinic doors. It is estimated between 23% and 34% of outpatient medical appointments in the US are missed annually.1–3 Patients who miss appointments do not receive necessary health care services, and prevent or delay other patients from being able to schedule appointments for treatment, follow-up, or preventive care. Furthermore, individuals who frequently miss medical appointments have poorer health outcomes4–9 and are less likely to utilize preventive health care services than individuals who keep medical appointments.10 In addition to clinical consequences for patients, missed appointments result in an underutilization of health care providers who have excess capacity to see patients, as well as underutilization of equipment, space, and staff. As a result, HCOs and providers experience a loss in revenue, which may be offset by charging patients’ fees or increasing overall charges to patients and payers.11 Moreover, provider and organizational compensation, as well as health system reimbursement models are often contingent upon patient populations meeting quality metrics.10 As such, missed appointments are also missed opportunities for providers and HCOs to improve health care quality by increasing the number of patients receiving preventive care and medical treatments.
Patients commonly give several reasons for missing appointments that include forgetfulness, confusion, or miscommunication over appointment information, feeling better, transportation issues, and difficulty leaving work or school.12 HCOs and providers have used different strategies to increase appointment attendance and to mitigate the impact of missed appointments.13 Strategies using appointment reminders have been generally well received by patients.3,14–16 Traditionally, HCOs have adopted a one-size-fits-all approach to appointment reminders, where one type of reminder is sent to all patients.17 However, research suggests that peoples’ preferences and attitudes impact their behavior.18–20 One study suggests patients who receive preferred reminders may be more likely to attend an appointment.17 Appointment reminder types that have been shown to be effective include mail, phone calls, email, and text messaging.11,21–25 Our objectives are to explore why patients miss appointments, to evaluate patient preferences for reminders, and to consider the use of patient-centered appointment reminders to reduce missed appointments. We hypothesize customizing appointment reminders to meet patients’ needs and preferences may improve appointment attendance.
One way of understanding the diversity of patients’ preferences toward appointment reminders is to elicit their preferences using a discrete choice experiment (DCE). DCEs are widely used in marketing to assess preferences and have gained prominence in the health care realm in recent years.26 DCEs use features of specific products or services to investigate the relative value individuals associate with a product or service. By understanding how individuals value features, products, or services, reminders can be designed to better meet the needs of patients. DCE methodology presumes that a product or service, such as a medical appointment reminder, can be described by attributes. These attributes can be organized into comparative choice tasks from which individuals choose their most preferred configuration.27 HCOs may be able to use reminder preferences to strategically design reminder systems to increase patient attendance, improve operational efficiency, augment revenue, and advance health care quality.
Methods
Online survey
We designed and fielded a survey based on the literature and pre-testing feedback.12,13,17 Participants were asked to respond to demographic questions and appointment reminder habit questions, and to complete a DCE. Participants provided informed consent online before beginning the survey. We used a survey panel company, Survey Sampling International (SSI), to obtain the participant sample. This research was approved by the Institutional Review Board at the University of North Carolina at Chapel Hill.
DCEs
In a DCE, participants make choices between two or more hypothetical alternatives that are described by a set of attributes and levels. The levels of each attribute systematically change in a series of choice tasks where participants select the option they prefer the most. Responses to the choices are analyzed to determine the relative value participants attach to the different levels of the attributes as well as to the attributes being considered.27
Selection of attributes and levels
We used the literature to select four salient appointment reminder attributes: initial reminder type, arrival of initial reminder, reminder content, and number of reminders. We developed plausible levels for each attribute. We presented the initial attributes and levels we selected to 12 members of a patient advisory council at a medical clinic for feedback. The group discussed each attribute and its associated level as well as alternative attributes and levels. Based on the group’s feedback, we refined the attributes and levels we planned to use in the questionnaire (Table 1).
Table 1.
Attributes and levels
| Attributes
|
||||
|---|---|---|---|---|
| Initial reminder type | Arrival of initial reminders | Reminder content | Number of reminders | |
| Levels | Postal mail | 1–6 days prior to appointment | Clinic location information (clinic address, directions and map) | 1 reminder |
| Phone call | 1–2 weeks prior to appointment | Rescheduling information (phone number or email) | 2 or 3 reminders–same type | |
| Text message | 3–4 weeks prior to appointment | Reason for visit | 2 or 3 reminders–different types | |
| >1 month prior to appointment | Clinic location information (clinic address, directions and map) and rescheduling information (phone number or email) | |||
| Social media | Clinic location information (clinic address, directions and map) and reason for visit | |||
| Electronic calendar (Outlook, Gmail, etc.) | Rescheduling information (phone number or email) and reason for visit Clinic location information (clinic address, directions and map), rescheduling information (phone number or email) and reason for visit | |||
We framed the initial reminder type attribute to include two broadly adopted reminder types (postal mail, phone), two commonly used but not universally adopted types (text message, email), and two emerging reminder types (social media, electronic calendar). Furthermore, we presented three general time frames in four levels for the arrival of initial reminder attribute: immediately preceding the appointment (1–6 days prior), advance notification (1–2 weeks prior, 3–4 weeks prior), and distant notice (more than 1 month prior). We did not include time frames over 1 month because the evidence suggests longer lead times can increase missed appointment rates.28 Moreover, we framed the reminder content attribute to include reminders with varying amounts of information: limited (clinic location only, rescheduling information only, reason for visit only) multicomponent (clinic location and rescheduling information, clinic location and reason for visit, rescheduling information and reason for visit), and comprehensive (clinic location, rescheduling information, and reason for visit). Finally, we represented the number of reminder attributes as being 1 reminder or 2–3 reminders with the option of having the same type of reminder each time or having different reminders. We considered offering more than three reminders, but the patient advisory council expressed concerns that more than two reminders would not be appreciated by most patients.
Development of the DCE
We used Sawtooth Software’s online tool, Discover, to create an efficient set of 16 choice tasks using the selected attributes and levels. Discover is an online software as a service tool that assists users in developing and fielding an efficient choice-based conjoint (CBC) questionnaire.29 The software is designed to be user-friendly for individuals who do not have significant knowledge of or experience with DCE development. As such, the software guides the user by making recommendations for an appropriate number of tasks and concepts per task to create an efficient design. The user is warned if the number of tasks is too few to achieve high-quality utility estimates.29 The recommended DCE design is based on prior parameters which are derived from several rating questions participants complete before beginning the DCE. The rating questions allow respondents to differentiate between levels using four preference categories: very desirable, desirable, extremely desirable, and no opinion.29 The data obtained from the rating questions serve as utility constraints to allow for robust individual utility estimation using logit with data augmentation employing empirical Bayes (EB). Additionally, the rating data provide individual-level preference information to avoid dominated concepts as the software creates on-the-fly experimental designs during data collection.29 All designs recommended by the software are near orthogonal, have a high relative D-efficiency, and are statistically efficient.29
Based on our attribute list and decision to have three active appointment reminder options, Discover recommended using 16 choice tasks. We have used 16 choice tasks in the past, and we found the number to be feasible for participants to complete.30,31 We decided not to include an opt-out choice, indicating a preference for not receiving a reminder. We wanted to force participants to make a choice because in practice, it is uncommon for patients to be offered an opportunity to opt out of appointment reminders. Figure 1 is a sample choice task. We followed the International Society for Pharmacoeconomics and Outcomes Research Guidelines for DCE design.32
Figure 1.
Example discrete choice experiment task.
We used Johnson’s formula to determine that a sample size of 73 participants or more would yield reasonably precise estimates of utility levels, given the use of 16 choice tasks, three active alternatives, and a maximum number of levels within a single attribute of seven.27 Based on our pretest results and previous experience with online surveys, we thought it was feasible to enroll at least 200 participants.
Pretesting
The online survey along with seven embedded usability questions was pretested in a random sample of 200 adults from September 9, 2015 to September 13, 2015. Participants were recruited using a mass informational email that was sent to subscribers of a listserv at the University of North Carolina at Chapel Hill. We aimed to recruit up to 250 participants, the maximum number of completed surveys our Discover license allows (in some instances, the software will allow more completed surveys if participants complete surveys simultaneously). Participants were screened online; exclusionary criteria consisted of being younger than 18 years or residing outside of the US. Individuals who completed the survey were eligible for one of four $25 gift certificates. Based on participant feedback from an embedded usability questionnaire and the participants’ responses, we made three changes. First, for the arrival of initial reminder attribute, we changed two levels indicating time in terms of number of days to number of weeks. Then, we modified the levels for the number of reminder attributes which used the phrase “multiple reminders” to 2–3 reminders because individuals stated they wanted to know a definitive number. We also changed the language stating “One reminder” level to “1 reminder”. Results from the pretest can be found in Tables S1 and S2.
Data collection and analysis
From September 25, 2015 to September 29, 2015, a stratified sample of survey panelists was obtained from an online survey panel company (SSI). To assure sufficient representation of each gender, we requested half of the sample to be female; exclusionary criteria included being younger than 18 years or residing outside of the US. The primary outcomes of interest were reasons for missing medical appointments and mean utilities of four attributes and overall attribute importance.
We used Sawtooth Software’s online Discover tool to perform descriptive analyses with mean values and proportions as well as to analyze the DCE data. Discover is a streamlined tool for individuals who want high-level, rapid results soon after the data have been collected. Discover uses EB methods to obtain individual-level utilities.29 EB methodology closely approximates the posterior mean values for individual-level utilities. The EB method involves computing an aggregate logit solution across all respondents using all of the data gleaned from the DCE. Then, each respondents’ answer to each CBC task is augmented by average population preferences to improve the quality of individual-level estimates so that results closely approximate those obtained using hierarchical Bayesian methods, which are considered the gold standard.29
Utilities are zero-centered numerical values that represent the relative desirability of the levels within each attribute. The higher the number, the more desirable the characteristic is to participants. Within Discover, utilities are used to calculate individual-level attribute importance scores, which are computed for each respondent by percentaging ranges for each attribute to determine mean importance scores.29 Attribute importance scores represent the relative importance or impact the four attributes have on the choice, given the range of levels used in the experiment.27 The attribute with the highest importance score is considered to be the most important attribute.
Results
We enrolled 251 adults nationally with a mean age of 43 (range 18–83) years, and 51% were female. Most were White (84%) and had a bachelor’s degree or higher (50%). Approximately half were employed full-time, and 38% had annual household incomes less than $45,000. About one-quarter (23%) reported missing one or more appointments in the past 12 months. Participants provided the following main reasons for missing appointments – transportation problems (28%), forgetfulness (26%), and confusion over time, date, or location (14%). Additional reasons given include problems leaving school/work (12%), feeling better (2%), and other (10%). Further participant characteristics are summarized in Table 2.
Table 2.
Discrete choice experiment results: medical appointment reminder preferences
| Participant characteristics | n=251 |
|---|---|
| Mean age (SD) | 43 (15) |
| Female | 51.2% |
| Race | |
| White | 84.1% |
| African American/Black | 7.6% |
| Asian | 5.2% |
| Native American | 0.4% |
| Other | 2.8% |
| Latino/Hispanic | 7.2% |
| Education | |
| Less than high school | 1.6% |
| High school graduate | 17.6% |
| Some college/trade school | 30.0% |
| Bachelor’s degree or higher | 50.8% |
| Household income | |
| <$45,000 | 37.8% |
| $45,000–$89,999 | 36.0% |
| $90,000+ | 25.9% |
| Employment | |
| Full-time | 49.2% |
| Part-time | 13.4% |
| Unemployed | 19.5% |
| Retired | 13.4% |
| Other | 4.5% |
| Appointments in 12 months | |
| Had 5+ | 35.9% |
| Had 2–4 | 41.4% |
| Had 1 | 11.6% |
| I do not recall how many I had | 0.0% |
| No appointments | 11.2% |
| Missed 1+ | 22.4% |
| Reasons for missed appointments | |
| I forgot | 26.0% |
| I had problems leaving work/school | 12.0% |
| I was confused about time, date, or location | 14.0% |
| I had transportation problems | 28.0% |
| I felt better | 8.0% |
| I was too sick | 2.0% |
| Other | 10.0% |
Abbreviation: SD, standard deviation.
Mean utility levels and importance scores are shown in Table 3. Attribute importance scores indicate the initial reminder type attribute was the most important attribute (44%). Among the other attributes, the reminder content attribute (24%) and arrival of initial reminder attribute (21%) were also relatively important. The number of reminder attributes (10.3%) was least important. Of the initial reminder type attribute levels, email and phone calls were the most preferred, followed by text message. Electronic calendar reminders and postal mail reminders were not preferred. The negative utility assigned to social media reminders suggests social media reminders are strongly disfavored. Preferences for the different levels of the reminder content attribute were not as distinct as preferences for other attributes’ levels. Some participants preferred reminders that only included clinic information, while others preferred reminders that contained all three types of information (location, rescheduling, and reason for visit) the most. Reminders containing only rescheduling information were not generally favored, and reason for visit was the least preferred level. Overall, for the arrival of the initial reminder attribute, participants preferred receiving reminders within 2 weeks of an appointment with reminders received between 1 and 6 days prior to appointment being most preferred. Appointment reminders received prior to 2 weeks before an appointment were least preferred. Additionally, participants preferred receiving one reminder over receiving two or three reminders even if multiple reminders were of different reminder types.
Table 3.
Medical appointment reminder attributes and levels with corresponding utility scores, n=251
| Attribute | Levels | Mean utilities | Lower 95% CI | Upper 95% CI | Mean attribute importance scores (CI) |
|---|---|---|---|---|---|
| Initial reminder type | Postal mail | −7.54 | −14.4 | −0.67 | 44% (42%–46%) |
| Phone call (personal or automated) | 32.88 | 26.26 | 39.5 | ||
| Text message | 21.24 | 14.86 | 27.62 | ||
| Email (from provider’s office or EHR) | 36.9 | 31.48 | 42.32 | ||
| Social media (Facebook, Twitter, etc.) | −78.91 | −87.15 | −70.68 | ||
| Electronic calendar (Outlook, Gmail, etc.) | −4.57 | −11.01 | 1.88 | ||
| Arrival of initial reminder | 1–6 days prior to appointment | 31.68 | 26.97 | 36.4 | 21% (19%–23%) |
| 1–2 weeks prior to appointment | 11.53 | 8.42 | 14.65 | ||
| 3–4 weeks prior to appointment | −14.75 | −18.73 | −10.78 | ||
| >1 month prior to appointment | −28.46 | −32.52 | −24.4 | ||
| Reminder content | Clinic location information (clinic address, directions and map) | 5.43 | 1.22 | 9.64 | 24% (23%–26%) |
| Rescheduling information (phone number or email) | −0.78 | −5.04 | 3.48 | ||
| Reason for visit | −4.19 | −9.18 | 0.81 | ||
| Clinic location information (clinic address, directions and map) and rescheduling information (phone number or email) | 0.37 | −3.63 | 4.37 | ||
| Clinic location information (clinic address, directions and map) and reason for visit | −3.11 | −7.42 | 1.19 | ||
| Rescheduling information (phone number or email) and reason for visit | −0.62 | −5.15 | 3.91 | ||
| Clinic location information (clinic address, directions and map), rescheduling information (phone number or email) and reason for visit | 2.9 | −1.71 | 7.51 | ||
| Number of reminders | 1 reminder | 9.1 | 5.69 | 12.52 | 10% (9%–11%) |
| 2 or 3 reminders–same type | −5.34 | −8.01 | −2.68 | ||
| 2 or 3 reminders–different types | −3.76 | −6.36 | −1.16 |
Abbreviations: CI, confidence interval; EHR, electronic health record.
Discussion
Consistent with the literature, we found 23% of individuals reported missing one or more appointments in the past 12 months.1–3 Among individuals who reported missing an appointment(s), two of the top three reasons given for missing appointments (forgetfulness and confusion over appointment time, date, or location) can be addressed through the use of appointment reminders. This finding suggests appointment reminders have the potential to increase appointment attendance. Consequently, improved patient appointment attendance may result in improved health outcomes for patients, improved clinic efficiency, as well as increased revenue. Additionally, using appointment reminders may increase opportunities for providers and HCOs to meet or exceed quality benchmarks for their patient populations.
Previous research evaluating patient preferences for appointment reminder strategies is relatively limited17,33 and may not reveal complex or novel reminder preferences. To our knowledge, this work is the first to use DCE to examine patient preferences for medical appointment reminders. Overall, we found participants preferred a single reminder that comes by email, phone, or text message, arriving 2 weeks or less prior to a scheduled appointment. Some individuals seem to prefer reminders that contain information about only the clinic location. In contrast, other individuals prefer reminders that provide all three pieces of information we evaluated in the DEC – clinic location, rescheduling information, and reason for visit.
Given initial reminder type was the most important attribute, sending patients their most preferred type of reminder may have the most impact on appointment attendance. HCOs should consider using one or more of the most preferred reminder types such as email, phone calls, or text messages. One recent study found three-quarters of individuals were somewhat or very willing to exchange or receive medical appointment reminders via a mobile device.34 Our results confirm this finding and may suggest patients prefer “mobile reminders” that can be received wherever an individual happens to be in the world to “static reminders”, which must be retrieved from a fixed location. Furthermore, HCOs should evaluate their use of postal mail, which we found to be relatively unpreferred, and consider using alternative reminder methods, which may be more effective. Moreover, electronic calendar reminders were somewhat unpreferred by participants, and social media reminders were strongly unpreferred. These results seem to contradict the popularity of electronic calendars and social media tools such as Facebook and Twitter. While we are not able to explain these findings within scope of this work, future work should explore whether these findings are due to unfamiliarity with using electronic calendar appointments or social media in this context, privacy or security concerns, or other possible worries about the delivery modality.
We found preferences for levels within the reminder content attribute were not as clear as preferences for levels within other attributes. This may be due to the use of overlapping levels within the attribute that could have confounded the results. Moreover, relatively little is known about how appointment reminder content influences appointment attendance.35 Consequently, with little context, our results are challenging to interpret and glean information to improve reminders for patients. However, as the literature base grows, researchers and HCOs should consider exploring and testing emerging information regarding reminder content. Of recent interest, preliminary research conducted in the UK National Health Service revealed that including appointment costs in text message reminders proved more effective at increasing appointment attendance than reminders without cost information.35
Limitations
Our results should be considered in light of several limitations. Foremost, we were only able to test a limited number of attributes and levels in the DCE while maintaining a valid experimental design. If we had used other attributes or levels, we may have gleaned different results. We believe that we selected relevant attributes and levels based on participant feedback, pretest results, and the existing literature. However, the use of overlapping levels for the attribute “reminder content” likely yielded results that may confound the result as participants may not have been able to differentiate between levels enough to express meaningful preferences. Moreover, there are various ways to determine the relative attribute importance.36 We used the standard method used by Sawtooth Software, but there may be other methods that could better represent the relative impact of each of the attributes. Additionally, our participant sample was drawn from an online pool of individuals who are highly educated and mostly White. As such, results may not be generalizable to other populations. For example, the national sample yielded results that were somewhat different from the pretest results.
Furthermore, our results show individuals’ stated preferences for reminders. It is unclear if an individual’s actual appointment attendance will change due to receiving a “preferred reminder” as the psychology research suggests. Finally, we were unable to assess individuals’ health conditions and thus were not able to evaluate how the type or severity of illness impacts reminder preferences.
Conclusion
Ultimately, as communication patterns and mechanisms change over time, it is important for HCOs to continuously evaluate the effectiveness of appointment reminders to encourage attendance. Our findings may be used by HCOs seeking to improve appointment attendance to achieve a variety of goals such as improving patient outcomes or operational efficiency. Ultimately, future research and informal experimentation is necessary to determine whether patient-centered reminders will result in improved appointment attendance.
Supplementary materials
Table S1.
Pretest participants
| Participant characteristics | n=200 |
|---|---|
| Mean age (SD) | 41 (15.5) |
| Female | 85.0% |
| Race | |
| White | 80.0% |
| African American/Black | 10.5% |
| Asian | 6.0% |
| Native American | 0.5% |
| Other | 3.0% |
| Latino/Hispanic | 4.0% |
| Education | |
| Less than high school | 0.0% |
| High school graduate | 9.0% |
| Some college/trade school | 19.5% |
| Bachelor’s degree or higher | 72.0% |
| Household income | |
| <$45,000 | 24.7% |
| $45,000–$89,999 | 33.8% |
| $90,000+ | 41.1% |
| Employment | |
| Full-time | 73.0% |
| Part-time | 15.5% |
| Unemployed | 5.0% |
| Retired | 1.5% |
| Other | 5.0% |
| Appointments in 12 months | |
| Had 5+ | 35.9% |
| Had 2–4 | 41.4% |
| Had 1 | 11.6% |
| I do not recall how many I had | 0.0% |
| No appointments | 11.2% |
| Missed 1+ | 22.4% |
| Reasons for missed appointments | |
| I forgot | 47.8% |
| I had problems leaving work/school | 17.4% |
| I was confused about time, date, or location | 15.2% |
| I had transportation problems | 4.3% |
| I felt better | 4.3% |
| I was too sick | 0.0% |
| Other | 10.9% |
Abbreviation: SD, standard deviation.
Table S2.
Pretest discrete choice experiment results (n=200)
| Attribute | Levels | Mean utilities | Lower 95% CI | Upper 95% CI | Mean attribute importance scores (CI) |
|---|---|---|---|---|---|
| Initial reminder type | Postal mail | −17.64 | −24.87 | −10.41 | 53% (51%–55%) |
| Phone call (personal or automated) | 19.46 | 14.25 | 24.66 | ||
| Text message | 39.06 | 32.78 | 45.34 | ||
| Email (from provider’s office or EHR) | 61.32 | 56.36 | 66.28 | ||
| Social media (Facebook, Twitter, etc.) | −121.15 | −128.38 | −113.92 | ||
| Electronic calendar (Outlook, Gmail, etc.) | 18.95 | 12.82 | 25.08 | ||
| Arrival of initial reminder | 1–5 days prior to appointment | 19.88 | 16.13 | 23.63 | 16% (14%–17%) |
| 6–14 days prior to appointment | 11.23 | 8.61 | 13.84 | ||
| 15–30 days prior to appointment | −12.56 | −15.54 | −9.59 | ||
| >1 month prior | −18.55 | −22.28 | −14.81 | ||
| Reminder content | Clinic location information (clinic address, directions and map) | 5.62 | 1.07 | 10.17 | 23% (21%–24%) |
| Rescheduling information (phone number or email) | −4.89 | −9.71 | −0.08 | ||
| Reason for visit | −17.09 | −21.7 | −12.48 | ||
| Clinic location information (clinic address, directions and map) and rescheduling information (phone number or email) | 13.57 | 9.14 | 17.99 | ||
| Clinic location information (clinic address, directions and map) and reason for visit | 4.44 | 0.03 | 8.86 | ||
| Rescheduling information (phone number or email) and reason for visit | −5.79 | −9.68 | −1.9 | ||
| Clinic location information (clinic address, directions and map), rescheduling information (phone number or email) and reason for visit | 4.15 | 0.11 | 8.18 | ||
| Number of reminders | 1 reminder | 6.53 | 3.13 | 9.93 | 9% (8%–10%) |
| Multiple reminders–same type | −5.9 | −8.07 | −3.73 | ||
| Multiple reminders–different types | −0.63 | −3.33 | 2.06 |
Abbreviations: CI, confidence interval; EHR, electronic health record.
Acknowledgments
The project was supported by the National Center for Advancing Translational Sciences (TraCS), National Institutes of Health (NIH), through Grant Award Number UL1TR001111. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH.
Footnotes
Disclosure
The authors report no conflicts of interest in this work.
References
- 1.Geraghty M, Glynn F, Amin M, Kinsella J. Patient mobile telephone ‘text’ reminder: a novel way to reduce non-attendance at the ENT outpatient clinic. J Laryngol Otol. 2008;122(3):296–298. doi: 10.1017/S0022215107007906. [DOI] [PubMed] [Google Scholar]
- 2.Dreiher J, Froimovici M, Bibi Y, Vardy DA, Cicurel A, Cohen AD. Nonattendance in obstetrics and gynecology patients. Gynecol Obstet Invest. 2008;66(1):40–43. doi: 10.1159/000115844. [DOI] [PubMed] [Google Scholar]
- 3.Parikh A, Gupta K, Wilson AC, Fields K, Cosgrove NM, Kostis JB. The effectiveness of outpatient appointment reminder systems in reducing no-show rates. Am J Med. 2010;123(6):542–548. doi: 10.1016/j.amjmed.2009.11.022. [DOI] [PubMed] [Google Scholar]
- 4.Schectman JM, Schorling JB, Voss JD. Appointment adherence and disparities in outcomes among patients with diabetes. J Gen Intern Med. 2008;23(10):1685–1687. doi: 10.1007/s11606-008-0747-1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Nuti LA, Lawley M, Turkcan A, et al. No-shows to primary care appointments: subsequent acute care utilization among diabetic patients. BMC Health Serv Res. 2012;12:304. doi: 10.1186/1472-6963-12-304. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Walburn A, Swindells S, Fisher C, High R, Islam KM. Missed visits and decline in CD4 cell count among HIV-infected patients: a mixed method study. Int J Infect Dis. 2012;16(11):e779–e785. doi: 10.1016/j.ijid.2012.06.004. [DOI] [PubMed] [Google Scholar]
- 7.Mugavero MJ, Lin HY, Willig JH, et al. Missed visits and mortality among patients establishing initial outpatient HIV treatment. Clin Infect Dis. 2009;48(2):248–256. doi: 10.1086/595705. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Colubi MM, Perez-Elias MJ, Elias L, et al. SEAD Study Group Missing scheduled visits in the outpatient clinic as a marker of short-term admissions and death. HIV Clin Trials. 2012;13(5):289–295. doi: 10.1310/hct1305-289. [DOI] [PubMed] [Google Scholar]
- 9.Berg MB, Safren SA, Mimiaga MJ, Grasso C, Boswell S, Mayer KH. Nonadherence to medical appointments is associated with increased plasma HIV RNA and decreased CD4 cell counts in a community-based HIV primary care clinic. AIDS Care. 2005;17(7):902–907. doi: 10.1080/09540120500101658. [DOI] [PubMed] [Google Scholar]
- 10.Hwang AS, Atlas SJ, Cronin P, et al. Appointment “no-shows” are an independent predictor of subsequent quality of care and resource utilization outcomes. J Gen Intern Med. 2015;30(10):1426–1433. doi: 10.1007/s11606-015-3252-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.McLean SM, Booth A, Gee M, et al. Appointment reminder systems are effective but not optimal: results of a systematic review and evidence synthesis employing realist principles. Patient Prefer Adherence. 2016;10:479–499. doi: 10.2147/PPA.S93046. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Deyo RA, Inui TS. Dropouts and broken appointments. A literature review and agenda for future research. Med Care. 1980;18(11):1146–1157. doi: 10.1097/00005650-198011000-00006. [DOI] [PubMed] [Google Scholar]
- 13.Norris JB, Kumar C, Chand S, Moskowitz H, Shade SA, Willis DR. An empirical investigation into factors affecting patient cancellations and no-shows at outpatient clinics. Decis Support Syst. 2014;57:428–443. [Google Scholar]
- 14.van Baar JD, Joosten H, Car J, et al. Understanding reasons for asthma outpatient (non)-attendance and exploring the role of telephone and e-consulting in facilitating access to care: exploratory qualitative study. Qual Saf Health Care. 2006;15(3):191–195. doi: 10.1136/qshc.2004.013342. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.O’Connor MB, Bond U, Saunders JA, et al. The preferences for outpatient clinic appointment reminders among rheumatology patients. J Clin Rheumatol. 2009;15(5):258–259. doi: 10.1097/RHU.0b013e3181b126c1. [DOI] [PubMed] [Google Scholar]
- 16.Gauthier C, Lindwall E, Davis W, Quinet R. Spanning generations-appointment reminder preferences among patients with rheumatic diseases. J Clin Rheumatol. 2012;18(6):294–297. doi: 10.1097/RHU.0b013e3182676b6b. [DOI] [PubMed] [Google Scholar]
- 17.Finkelstein SR, Liu N, Jani B, Rosenthal D, Poghosyan L. Appointment reminder systems and patient preferences: patient technology usage and familiarity with other service providers as predictive variables. Health Informatics J. 2013;19(2):79–90. doi: 10.1177/1460458212458429. [DOI] [PubMed] [Google Scholar]
- 18.Bem DJ. Self-perception theory. Adv Exp Soc Psychol. 1972;6:1–62. [Google Scholar]
- 19.Festinger L. A Theory of Cognitive Dissonance. Vol. 2. Stanford, CA: Stanford University Press; 1962. [Google Scholar]
- 20.Petty RE, Cacioppo JT, Schumann D. Central and peripheral routes to advertising effectiveness: the moderating role of involvement. J Cons Res. 1983;10(2):135–146. [Google Scholar]
- 21.Guy R, Hocking J, Wand H, Stott S, Ali H, Kaldor J. How effective are short message service reminders at increasing clinic attendance? A meta-analysis and systematic review. Health Serv Res. 2012;47(2):614–632. doi: 10.1111/j.1475-6773.2011.01342.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Hashim MJ, Franks P, Fiscella K. Effectiveness of telephone reminders in improving rate of appointments kept at an outpatient clinic: a randomized controlled trial. J Am Board Fam Pract. 2001;14(3):193–196. [PubMed] [Google Scholar]
- 23.Hasvold PE, Wootton R. Use of telephone and SMS reminders to improve attendance at hospital appointments: a systematic review. J Telemed Telecare. 2011;17(7):358–364. doi: 10.1258/jtt.2011.110707. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Boksmati N, Butler-Henderson K, Anderson K, Sahama T. The effectiveness of SMS reminders on appointment attendance: a meta-analysis. J Med Syst. 2016;40(4):90. doi: 10.1007/s10916-016-0452-2. [DOI] [PubMed] [Google Scholar]
- 25.Perri-Moore S, Kapsandoy S, Doyon K, et al. Automated alerts and reminders targeting patients: a review of the literature. Patient Educ Couns. 2016;99(6):953–959. doi: 10.1016/j.pec.2015.12.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.de Bekker-Grob EW, Ryan M, Gerard K. Discrete choice experiments in health economics: a review of the literature. Health Econ. 2012;21(2):145–172. doi: 10.1002/hec.1697. [DOI] [PubMed] [Google Scholar]
- 27.Orme BK. Getting Started with Conjoint Analysis: Strategies for Product Design and Pricing Research. Glendale: Research Publishers; 2010. [Google Scholar]
- 28.McMullen MJ, Netland PA. Lead time for appointment and the no-show rate in an ophthalmology clinic. Clin Ophthalmol. 2015;9:513–516. doi: 10.2147/OPTH.S82151. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Orme BK. Discover-CBC: how and why it differs from SSI Web’s CBC Software. Sawtooth Software; 2014. [Accessed December 22, 2016]. Available from: http://www.sawtoothsoftware.com/download/SaaS_CBC_White_Paper.pdf. [Google Scholar]
- 30.Pignone MP, Crutchfield TM, Brown PM, et al. Using a discrete choice experiment to inform the design of programs to promote colon cancer screening for vulnerable populations in North Carolina. BMC Health Serv Res. 2014;14:611. doi: 10.1186/s12913-014-0611-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Kistler CE, Hess TM, Howard K, et al. Older adults’ preferences for colorectal cancer-screening test attributes and test choice. Patient Prefer Adherence. 2015;9:1005–1016. doi: 10.2147/PPA.S82203. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.Bridges JF, Hauber AB, Marshall D, et al. Conjoint analysis applications in health – a checklist: a report of the ISPOR Good Research Practices for Conjoint Analysis Task Force. Value Health. 2011;14(4):403–413. doi: 10.1016/j.jval.2010.11.013. [DOI] [PubMed] [Google Scholar]
- 33.Zallman L, Bearse A, West C, Bor D, McCormick D. Patient preferences and access to text messaging for health care reminders in a safety-net setting. Inform Health Soc Care. 2016:1–11. doi: 10.3109/17538157.2015.1113177. [DOI] [PubMed] [Google Scholar]
- 34.Serrano KJ, Yu M, Riley WT, et al. Willingness to exchange health information via mobile devices: findings from a population-based survey. Ann Fam Med. 2016;14(1):34–40. doi: 10.1370/afm.1888. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Hallsworth M, Berry D, Sanders M, et al. Correction: stating appointment costs in SMS reminders reduces missed hospital appointments: findings from two randomised controlled trials. PLoS One. 2015;10(10):e0141461. doi: 10.1371/journal.pone.0141461. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Lancsar E, Louviere J, Flynn T. Several methods to investigate relative attribute impact in stated preference experiments. Soc Sci Med. 2007;64(8):1738–1753. doi: 10.1016/j.socscimed.2006.12.007. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Table S1.
Pretest participants
| Participant characteristics | n=200 |
|---|---|
| Mean age (SD) | 41 (15.5) |
| Female | 85.0% |
| Race | |
| White | 80.0% |
| African American/Black | 10.5% |
| Asian | 6.0% |
| Native American | 0.5% |
| Other | 3.0% |
| Latino/Hispanic | 4.0% |
| Education | |
| Less than high school | 0.0% |
| High school graduate | 9.0% |
| Some college/trade school | 19.5% |
| Bachelor’s degree or higher | 72.0% |
| Household income | |
| <$45,000 | 24.7% |
| $45,000–$89,999 | 33.8% |
| $90,000+ | 41.1% |
| Employment | |
| Full-time | 73.0% |
| Part-time | 15.5% |
| Unemployed | 5.0% |
| Retired | 1.5% |
| Other | 5.0% |
| Appointments in 12 months | |
| Had 5+ | 35.9% |
| Had 2–4 | 41.4% |
| Had 1 | 11.6% |
| I do not recall how many I had | 0.0% |
| No appointments | 11.2% |
| Missed 1+ | 22.4% |
| Reasons for missed appointments | |
| I forgot | 47.8% |
| I had problems leaving work/school | 17.4% |
| I was confused about time, date, or location | 15.2% |
| I had transportation problems | 4.3% |
| I felt better | 4.3% |
| I was too sick | 0.0% |
| Other | 10.9% |
Abbreviation: SD, standard deviation.
Table S2.
Pretest discrete choice experiment results (n=200)
| Attribute | Levels | Mean utilities | Lower 95% CI | Upper 95% CI | Mean attribute importance scores (CI) |
|---|---|---|---|---|---|
| Initial reminder type | Postal mail | −17.64 | −24.87 | −10.41 | 53% (51%–55%) |
| Phone call (personal or automated) | 19.46 | 14.25 | 24.66 | ||
| Text message | 39.06 | 32.78 | 45.34 | ||
| Email (from provider’s office or EHR) | 61.32 | 56.36 | 66.28 | ||
| Social media (Facebook, Twitter, etc.) | −121.15 | −128.38 | −113.92 | ||
| Electronic calendar (Outlook, Gmail, etc.) | 18.95 | 12.82 | 25.08 | ||
| Arrival of initial reminder | 1–5 days prior to appointment | 19.88 | 16.13 | 23.63 | 16% (14%–17%) |
| 6–14 days prior to appointment | 11.23 | 8.61 | 13.84 | ||
| 15–30 days prior to appointment | −12.56 | −15.54 | −9.59 | ||
| >1 month prior | −18.55 | −22.28 | −14.81 | ||
| Reminder content | Clinic location information (clinic address, directions and map) | 5.62 | 1.07 | 10.17 | 23% (21%–24%) |
| Rescheduling information (phone number or email) | −4.89 | −9.71 | −0.08 | ||
| Reason for visit | −17.09 | −21.7 | −12.48 | ||
| Clinic location information (clinic address, directions and map) and rescheduling information (phone number or email) | 13.57 | 9.14 | 17.99 | ||
| Clinic location information (clinic address, directions and map) and reason for visit | 4.44 | 0.03 | 8.86 | ||
| Rescheduling information (phone number or email) and reason for visit | −5.79 | −9.68 | −1.9 | ||
| Clinic location information (clinic address, directions and map), rescheduling information (phone number or email) and reason for visit | 4.15 | 0.11 | 8.18 | ||
| Number of reminders | 1 reminder | 6.53 | 3.13 | 9.93 | 9% (8%–10%) |
| Multiple reminders–same type | −5.9 | −8.07 | −3.73 | ||
| Multiple reminders–different types | −0.63 | −3.33 | 2.06 |
Abbreviations: CI, confidence interval; EHR, electronic health record.