Enhancing the Performance of Patient Appointment Scheduling: Outcomes of an Automated Waitlist Process to Improve Patient Wait Times for Appointments

Abstract

Introduction:

Online self-scheduling of medical appointments is increasingly common. An automated waitlist can be used for patients who desire an earlier appointment time if one becomes available after they are scheduled. Our study examines outcomes of an automated waitlist and self-rescheduling process.

Methods:

We studied outcomes of an automated waitlist self-rescheduling process in which patients with existing appointments elected to be placed on an automated waitlist for an earlier appointment offer. When software found earlier dates for the same visit type, patients were then notified through an automated process and could self-reschedule. We reviewed appointments for which patients were sent new offers when earlier appointment slots were found. We compared the accepted appointment offers with the original scheduled appointments and determined the number of days that the appointment had been moved up.

Results:

Spanning the calendar year 2023 there were 1 019 698 appointment offers generated by an automated waitlist process for 229 998 appointments and sent to 164 248 patients. The waitlist process automatically found open appointments as they became available and sent the first new appointment offer within 2 days after being placed on the waitlist for 74 736 (32.5%) of the 229 998 waitlisted appointments. Patients sent back at least 1 response for 104 554 (45.4%) of the waitlisted appointments. Of the responses, 56 636 accepted one of the sent offers for an accept rate of 24.6% (56 636/229 998). For accepted, moved-up visits, appointments were self-rescheduled earlier by a mean of 22.6 days (95%CI; 22.2, 22.9, P < .0001).

Conclusion:

New appointments can be successfully self-rescheduled using an automated waitlist process that allows patients to accept or decline new appointment offers. This process can increase the efficiency of scheduling and decrease appointment wait time for patients desiring more timely access to healthcare. In addition, this process can be successfully applied across several different appointment type categories.

Introduction

Self-scheduling for medical appointments is catching up to other service providers and has expanded to meet the needs of both patients and medical practices. Zocdoc and Lybrate are examples of third-party web-based appointment platforms that allow patients to search for medical appointments from a selection of providers whose appointment calendars are linked to the platform.^1-3 Patients can search by provider type (primary care, dermatology, orthopedics, etc.), location, insurance coverage, and other factors and schedule an appointment based on their needs.

Instead of working through third-party platforms, some medical practices are providing their own patients with access to self-scheduling via their patient portal connected to their electronic health record (EHR). Large academic healthcare institutions such as Johns Hopkins, Mayo Clinic, and the University of California San Francisco (UCSF) are developing self-scheduling processes for various appointment types, including primary care visits, screening mammography, well-child visits, COVID testing, and many others.^4-14 There have also been studies of self-scheduling in more focused practices, including sexual health and dermatology.^15-17

Although self-scheduling has been successful for many different appointment types and situations, ⁹ it cannot create new appointments when access is limited. One of the most labor-intensive aspects of scheduling is finding appointments when there is a limited supply. Patients who are dissatisfied with an appointment scheduled 3 weeks later than expected might call daily to look for cancellations to obtain an earlier appointment. If they are registered to a patient portal, they may message their provider and ask if a 3-week delay in a visit is safe for their condition. Regardless of whether the delay is appropriate or not, the provider will likely need to asynchronously address the patient concern, resulting in additional work.

For appointments with high demand and earliest open appointment slots weeks or months in the future, finding newly opened appointment slots can be a challenge. It can take a significant amount of time for staff schedulers to manually go through a waitlist, search for openings and notify patients of sooner appointment openings, either proactively or in response to patient calls.

Some scheduling processes are now available that may help address these issues.^18,19 For example, practices can now use automated waitlist processes that can search appointment calendars and find openings for patients when slots open up due to cancellations or other causes. Patients associated with Ohio State University, Loma Linda University Health, Sutter Health, UCSF, and Johns Hopkins have had the opportunity to use Fast Pass, the automated waitlist technology used in the Epic EHR that is also used at Mayo Clinic.^18-23 However, there is little published research on the results of these processes. Therefore, our aim in this study was to examine the use and outcomes of the automated waitlist self-scheduling process for scheduled appointments.

Methods

Setting

This retrospective study took place at Mayo Clinic. Mayo Clinic is a multispecialty, multisite practice with locations in the United States and internationally. Mayo Clinic has millions of outpatient visits yearly. The Mayo Clinic practice has both primary and specialty care that extends across all ages, from prenatal to supercentenarian.

Mayo Clinic has a single EHR, Epic™, that also manages patient scheduling. Patient Online Services (the patient portal) of Mayo Clinic provides connectivity with patients online and by mobile app. Using the Mayo Clinic patient portal, patients have access to their medical record and can perform actions including, but not limited to, viewing upcoming appointments, requesting medication refills, performing self-triage for a variety of conditions, and self-scheduling.^10,24,25 Patients can also self-schedule multiple appointment visit types, using several different self-scheduling processes. ¹⁹ Self-schedulable visits include established patient office visits, specialty consultations, lab tests, radiographs, and immunizations. ⁹

Automated Waitlist for Scheduled Visits

Prior to the implementation of the current waitlist protocol, rescheduling appointments was primarily a manual process with staff schedulers. For appointments where an automated waitlist is not suitable, staff schedulers can still add appointments to the manual waitlist process. Although operational workflows may vary, the manual process typically involves a scheduler frequently reviewing the waitlist (eg, daily or twice daily), checking for available appointment times, temporarily holding an available slot, and attempting to contact the patient. If the patient is unavailable, the scheduler must decide whether to leave a message and continue holding the appointment or to call the next patient. This process includes multiple manual steps, such as making phone calls to confirm the appointment time with the patient. The automated waitlist system eliminates most, if not all, of these manual processing steps.

The focus for the current study is the automated waitlist process for scheduled visits that was implemented in 2020. It is an automated module in the EHR that was designed to help patients move up a future scheduled visit. The module uses software to search through future schedules and identifies open appointment slots earlier than the existing visit. Once identified, new appointment offers are sent directly to patients through their patient portal accounts.

During the study period, the automated search for open appointments occurred once daily between 4 PM and 6 PM local time. When earlier open appointment slots were found, waitlisted patients were notified by email, text, or push notifications. Up to 8 patients could be sent the same appointment offer for the earlier visit. Offers included an expiration time. Offer expiration times changed during the study period with expiration periods varying from a maximum of 14 hours (early in the study period) to 4 hours later in the study period. Patients were offered only 1 appointment slot for each of their waitlisted appointments each day. Thus, while patients can receive multiple offers per waitlisted appointment over multiple days, they receive no more than 1 offer per day. Each patient waitlist offer is unique, ensuring the patient does not see the same offer more than once.

Once the patient was notified of a new scheduling offer, the patient could log onto their patient portal account online or mobile app to view the offer. Figure 1 is a screenshot of the patient view. As shown in Figure 1, the offer is juxtaposed to the existing scheduled appointment for patient convenience in deciding whether to accept or decline the offer. Accepted offers are immediately self-rescheduled; no staff scheduler is involved, and the previously scheduled appointment slot is opened for someone else.

Figure 1.

Screenshot of an offer to move up appointment generated automatically in the automated waitlist process for scheduled appointments.

The offer is first come, first served, and has an expiration time, so it is important for patients to quickly log into the portal and decide about the offers when they arrive. Figure 1 shows how the patient is made aware of the expiration time with the following sentence included in the appointment offer: “This offer is available until today at 11:48 AM or until another patient accepts it.” It should be noted that the waitlist offers cease once the date exceeds the original booked appointment date.

There are 5 possible dispositions to the automated waitlisted appointment offers.

1. Accepted: patient accepts earlier appointment offer.

2. Declined: patient declines earlier appointment by choosing “keep existing time” Figure 2 shows what patients would see as a result. Note that patients who declines the earlier appointment are notified that “No changes have been made,” meaning the existing appointment is intact.

3. Unavailable: this designation is used if another patient had already accepted the offer before it could be acted on.

4. Expired: If the appointment offer was not acted on in any way (ie, not accepted or declined by the patient but also not accepted by another patient) then it was labeled with the outcome “expired,” as soon as the expiration time of the offer had passed.

5. Deleted: Rarely, patients wanted to be removed from the waitlist for a specific appointment. A staff scheduler could delete them from the waitlist or they could do that themselves online or mobile by opting out of the waitlist. Removal from the waitlist deleted the current offer and ceased future offers for that specific waitlisted appointment.

Figure 2.

Online portal screenshot of patient view after declining an earlier automated waitlist appointment offer.

Data Collection and Study Duration

The data for this study was collected from the Mayo Clinic Epic Systems software. The Mayo Clinic Enterprise Office of Access Management supplied the dataset for this study. We used all appointments that had the automated waitlist process initiated beginning January 1, 2023 through December 31, 2023. We included waitlisted appointments that had any appointment offer sent from January 1, 2023 through February 22, 2024.

Data analyzed included demographics of the patients having waitlisted appointments, and times and dates of process events as described in Table 1.

Table 1.

Terminology definitions, measures, and data involved in waitlist appointment and offers outcomes.

Terminology	Definition	Data involved
Appointment waitlist terminology
Entry date/time on the waitlist	Date and time the patient appointment was put on the waitlist for a sooner visit. An existing scheduled appointment can be put on waitlist for a sooner visit at the time it is initially booked or later.	Date/time of entry into the waitlist process.
Original (existing) appointment	Scheduled appointment that patient is using the waitlist process to change.	Date/time of existing (original) scheduled appointment.
New appointment offer	Appointment opening of the same visit type for an earlier slot, found by an automated software search of schedules. During the study, automated searches of schedules occurred daily in a batch around 4 PM to 6 PM local time.	Date/time of new earlier appointment.
Sent appointment offer	Appointment offer example as shown in Figure 1. Notification that the appointment offer was available for review was communicated by text, email, or push notification. Each waitlisted appointment received a maximum of 1 offer daily that was not repeated.	Date and time new appointment offer was sent. Details of each sent offer such as date and time of each new offered appointment was not available. Only accepted offers had date/time of new appointment available.
Viewed appointment offer	Appointment offer that was viewed (opened by portal online or with mobile app) by the patient. Offer must be viewed to respond. Response options of accept or decline as noted above and in Figure 1.	Date/time appointment offer was opened (viewed in the portal, online, or by mobile app). Receipt of email notification or viewing of text message notification could have been earlier but time/date of a viewed text message or emailed notification was not available.
Appointment offer disposition status date/time	Appointment offers can have one of five dispositions as explained above.	Date/time appointment offer was dispositioned as accepted, declined, etc.
Process and measure terminology
Original (existing) appointment wait time	Elapsed time from booking the original appointment until the date/time of the scheduled appointment. This is calculated by subtracting date/time of original booking from the date/time of appointment. From the patient perspective this the waiting time from booking until the appointment occurs.	Date/time original appointment booked. Date/time of original scheduled appointment.
New appointment wait time (based on original booking date/time, not based on new offer booking date/time)	Elapsed time from booking the original appointment until the date/time of the accepted offer (new appointment scheduled time minus original appointment date/time of booking).	Date/time originally booked, time/date new appointment time. Data only available when offer accepted. The date/time for new offer was not available with other offer outcomes.
Appointment move up time (wait time improvement when waitlist offer accepted)	Improvement in wait time from original to accepted new appointment time.	Time difference between date/time of new scheduled appointment and date/time of original scheduled appointment.
Elapsed time from waitlist entry to first offer sent	Time it takes for the first waitlist offer to be generated and sent after entry date/time into the automated waitlist process.	Date/time offer sent minus date/time entered in automated waitlist process.
Offer number	Number in a sequence of appointment offers. Offers continue until the offer is accepted, original appointment date is exceeded, or the appointment is deleted from waitlist. Software prevents same offer repeating day after day so no repeat of previous expired offer but can get up to 1 new offer daily. 1 = first offer, 2 = second offer, etc.	Offer number included in dataset.
Elapsed time from offer sent to offer viewed	Time from when the offer was sent until offer was viewed (opened) in patient portal. This measures how quickly the patient reviews the appointment offer.	Date/time viewed minus date/time sent. Viewed date/time is when viewed in portal (online or mobile) not text or email notification that offer is available.
Elapsed time from waitlist entry to booked automatic waitlisted appointment	Time from waitlist entry until appointment offer accepted. This measures from waitlist start to completion of a rebooking (time of an individual waitlist process success)	Date/time response sent (accept) minus date/time of entry into waitlist process.

Appointment Terminology, Definitions, and Measures

We used waitlist process diagrams to show the process flow and outcomes after an appointment was put on the automated waitlist. We also examined the change in appointment wait time that resulted from accepting a new appointment offer from the waitlist process. The definitions of required stages in the waitlist process and appointment time measures are summarized in our definitions table below.

It should be noted that appointment literature has used appointment wait time and appointment lead time somewhat interchangeably,^26,27 but there is an important distinction that we show in Figure 3. It is necessary to explain here because the distinction between appointment wait time and appointment lead time only becomes apparent when appointments are rescheduled, which is the focus of this study.

Figure 3.

Reschedule process example showing a rescheduled appointment that results in a decrease in the appointment wait time of 17 days.

Original scheduled process in boxes with solid lines, rescheduled process in boxes with dashed lines. Differences between wait time and lead time after rescheduling are shown in this example.

Our definition of wait time starts from the original date and time the appointment was booked. In this study the wait time still starts from the original date and time of booking even after a reschedule. This is different from appointment lead time, which, instead of a start time from the original booking, resets to the start time of the rebooking. The importance is that wait times can correlate with health outcomes and shorter wait times can be a valuable patient satisfier.^28,29

Although not used as an outcome measure in this study, lead time is an essential element used by the scheduling system for many appointment types. Lead time also can create a minimum wait time. For example, elective colonoscopy appointments may have a required lead time of a few days to allow patients the time to complete a bowel cleansing preparation. Thus, when rescheduling a colonoscopy, open appointments on the same day may not be listed as available because they are too soon for the lead time needed for that appointment type. Thus, lead times are a critical part of the rescheduling process and can force a minimum wait time. Since this study focuses on what the patient perceives as an elapsed wait from their original booking we will not further address lead time considerations in this study other than this clarification of terminology.

Change in Appointment Wait Time: “Move up” Versus “Move out”

A few waitlisted appointments were rescheduled further in the future than the original scheduled appointment. These were appointment “move outs” rather than “move ups.” The automated wait list process will only send offers that are improved. However, on an individual appointment basis, when there are multiple scheduling actions, an appointment “move out” may occur. For example, the original appointment may be rescheduled by staff to a new date/time that is further out than the original appointment. However, the appointment may remain waitlisted. Thus, the automated waitlist process can function correctly and still result in a new appointment offer which is further out than the original appointment.

Another published study on the waitlist process included only the appointment move ups. ¹⁸ In this study we aimed to identify and examine multiple outcomes after the waitlist process was initiated, including those that did not directly result in a move up appointment.

Statistics and Analysis

Data analysis was done using Stata 18.0 (College Station, Texas). Chi-square was used for differences in demographics. T-test was used for differences in means.

Results

Process Flow Counts for Automated Waitlist Appointment Offers

There were 164 248 unique patients who had a total of 229 998 appointments placed on the automated waitlist. Mean waitlisted appointments per waitlisted patient was 1.4 (SD 0.91; median 1, IQR 1-2). Twenty-six percent of waitlisted patients (41 994/164 248) had more than 1 waitlisted appointment during the yearlong study.

There were 1 019 698 offers sent for the 229 998 unique waitlisted appointments. There was a mean of 4.4 appointment offers per waitlisted appointment (SD 7.3). The median appointment offers per waitlisted appointment was 2 (Interquartile range, [IQR] 1, 4). The 90th percentile and 95th percentile were 10 and 17 offers per waitlist appointment, respectively.

Despite at least 1 appointment offer sent, and often multiple offers for the same waitlisted appointment, there were many waitlisted appointments where the offers were never viewed within the patient portal. After the appointment offers were viewed for the waitlisted appointment, there were also a significant number that never responded to the offer after viewing (either to accept or decline). Figure 4 shows the process flow of the individual waitlist appointments from the original 229 998 waitlisted appointments to the 56 636 waitlisted appointments where a new offer was accepted.

Figure 4.

Process flow showing how the 229 998 waitlisted appointments with at least 1 sent offer yielded 56 636 accepted new appointments. The outcomes of accepted offers are also shown. Note that this is a one-to-many process. For each unique appointment there can be many offers until a new appointment is accepted, after which the offers stop.

After determining the count of 56 636 offers that were accepted, we collected additional appointment-related dispositions which are shown in the 4 boxes in Figure 4 below the accepted offers. A significant number of rescheduled appointments, 23.7%, were canceled after an accepted waitlist offer. From the total accepted responses there were 3.25% (1842/56 636) no-shows, or, when only using accepted and not subsequently canceled appointments, no-shows were 4.26% (1842/43 228).

We also examine the process flow from the perspective of the 1 019 698 individual appointment offers. In Figure 5 we follow the process path from the sent waitlist appointment offers to each of the 5 mutually exclusive appointment offer outcomes. Unlike the appointment waitlist process which can involve multiple appointment offers for each waitlisted appointment, the appointment offer process has 1 outcome for each offer. The process pathway for reaching each of those outcomes is shown in Figure 5.

Figure 5.

Process pathway showing the flow of appointment offers from sent offer to 5 mutually exclusive offer outcome categories of: accepted, declined, expired, unavailable, and deleted. Elapsed time intervals for the processes are diagrammatically shown on left.

As shown in Figures 4 and 5, the successful culmination of the waitlist process was an accepted offer for a new appointment. The 56 636 accepted new appointments represented 24.6% (56 636/229 998) of all waitlisted appointments that were sent at least 1 offer. Figure 4 shows the appointment outcomes after accepting the waitlist offer; Figure 5 shows the offer outcomes. The intermediary process steps in Figures 4 and 5 show that success at self-rescheduling requires the patient to complete several steps in sequence. These are: (1) view the notification message that an offer is available or check the portal for an offer, (2) view the offer in the portal, (3) decide to respond, (4) respond before the expiration time and before someone else accepts an offer for the same appointment slot (to avoid an unavailable offer), and (5) decide to accept the offer or decline the new offer.

Demographics Associated With Accepted Offers

Table 2 shows the patient demographics associated with appointments that were sent new appointment offers. We aimed to compare demographics and appointment visit types associated with accepted new appointments versus waitlisted appointments that had no accepted offers. A complete explanation of visit types in Table 2 can be found in North et al. ⁹

Table 2.

Demographics associated with responses to waitlisted appointments that had at least 1 offer.

Demographic, visit measure, or category	Appointments where individuals accepted a self-reschedule offer, N = 56 636; count (%)	Appointments where self-reschedule offer was sent but not accepted, N = 173 362; count (%)	P value a
Age (years)			<.0001
0-17	3702 (6.54)	16 485 (9.51)
18-34	8032 (14.18)	26 079 (15.04)
35-49	10 474 (18.49)	31 388 (18.11)
50-64	14 258 (25.17)	41 226 (23.78)
65-74	12 451 (21.98)	32 492 (18.74)
75-84	6448 (11.38)	19 514 (11.26)
85 and up	1271 (2.24)	6178 (3.56)
Mean age [95%CI]	52.77 [52.60, 52.94]	51.12 [51.01, 51.23]	<.0001
Gender			<.0001
Female	36 662 (64.73)	109 918 (63.40)
Male	19 952 (35.23)	63 402 (36.57)
Race			.14
White	52 212 (92.19)	159 719 (92.13)
Asian	1191 (2.10)	3941 (2.27)
Black	1916 (3.38)	5803 (3.35)
Other race	603 (1.06)	1781 (1.03)
Unknown, choose not to disclose, missing, unable to provide	714 (1.26)	2118(1.22)
Ethnicity			.79
Hispanic	2510 (4.43)	7758 (4.48)
Not Hispanic or Latino	53 256 (94.03)	162 887 (93.96)
Unknown, choose not to disclose, missing, unable to provide	870 (1.54)	2717 (1.57)
Mean initial appointment wait time in days [95%CI]	52.37 [52.12, 52.63]	49.02 [48.87, 49.17]	<.0001
Appointment type category b			<.0001
Consultation	12 097 (21.36)	32 277 (18.62)
Established patient	33 334 (58.86)	111 240 (64.17)
New patient	3399 (6.00)	10 516 (6.07)
Procedure	1942 (3.43)	3994 (2.30)
Return	1270 (2.24)	3744 (2.16)
Testing	1211 (2.14)	3861 (2.23)
Therapy	3275 (5.78)	7293 (4.21)
Other	106 (0.18)	436 (0.25)

^aP value associated with null hypothesis test of accepted offer category equals the not accepted category (or equality of mean age or mean wait time).

^bAppointment type categories described in detail in North et al. ⁹

Appointments accepted from waitlist offers had some statistical differences in age and gender with appointment acceptances slightly more likely for those older and female. We did not detect a statistically significant difference in overall race or ethnic categories. Perhaps not surprisingly, accepted appointments were more likely to have had a longer original appointment wait time.

Notably, new appointments were successfully self-rescheduled across 7 different appointment type categories. The consultation appointment types were slightly more likely to be accepted than not with odds ratio (OR) of 1.19, 95%CI; 1.26 to 1.21, P < .0001. The established patient appointment types were significantly less likely to be accepted than not (OR 0.80, 95%CI; 0.78-0.81, P < .0001). Procedures, though with low volumes compared to consultations and established patient visits, were much more likely to have waitlist appointment offers accepted than not (OR 1.51, 95%CI; 1.42-1.59, P < .0001). Therapy visit appointment offers were also more likely to be accepted (OR 1.39, 95%CI 1.34-1.45, P < .0001). New patient visits and testing visits did not have odds ratios statistically different from 1 when examining whether they were more or less likely than other visit types to be accepted.

Appointments Self-rescheduled to a Sooner Date (move up)

A measure of interest was in the ability of the waitlist process to move up appointments to a sooner date, so for this measure we excluded the 4.9% (2781/56 636) that were self-rescheduled to later dates. Most self-rescheduled appointments, 95.1% (53 855/56 636), were moved up in time. For the 53 855 moved-up appointments, the mean difference in appointment wait time from original appointment to moved up appointment was 22.6 days (95%CI; 22.2, 22.9, P < .0001; Figure 6).

Figure 6.

Histogram of the days of appointment wait time improvement for moved up appointments (N = 53 855).

The median days moved up was 15 with an IQR of 7 to 31 days moved up. There were 41 627 appointments (77%) that were moved up 7 days or more, 29 616 (55%) moved up 14 days or more, and 21 647 (40%) moved up at least 21 days.

For the accepted, moved up offers (n = 53 855), the mean wait time for the original appointment was 52.7 days (95%CI; 52.4-52.9), and the accepted offer mean wait time dropped to 30.1 days (95%CI; 29.9, 30.3). Figure 7 shows the histograms of the appointment wait times showing the visible shift from longer wait times to shorter wait times after accepting the waitlist offer.

Figure 7.

Histograms of pre-waitlist and post-waitlist frequency of appointments by appointment wait time. X axis is elapsed time (in days) from original booking date/time to scheduled appointment date/time (pre-waitlist) and rescheduled (post-waitlist) appointment date/time. N = 53 835, the number of scheduled visits that were moved up after the waitlist process.

Of the 2781 appointments where the appointment was moved out, the mean appointment wait time difference, extended from originally scheduled was 26.7 days (95%CI; 25.8-27.6, P < .0001) with a median of 20 days. The mean original wait time was 46.7 days (95%CI 45.3-48.0) and the new wait time was 73.4 days (95%CI 71.6-75.2).

Using the entire 56 636 accepted offers cohort (both those with move ups and move outs) the median was a 14.2 day move up with mean move up of 20.1 days (95%CI; 19.8-20.5, P < .0001). Mean original wait time for the 56 636 offers was 52.4 days (95%CI; 52.1-52.6) with mean new wait time of 32.3 days (95%CI; 32.0-32.5).

Time From Entry on Waitlist to Having an Appointment Offer Sent

Having new appointment offers sent to patients within a few days of waitlist entry is likely important to keep them engaged in the process. We found that for many, waitlist entry to a first appointment offer was very fast. For our cohort of 229 998 waitlisted appointments that received offers, 24.5% (56 300/229 998) were sent their first offer within the first day of being on the waitlist. An additional 18 436 first offers came on day two, totaling 74 736 first offers (32.5%) within 2 days of waitlist entry. The 50th percentile for first offers was 6 days and the 75 percentile was 22 days. Figure 8 is the histogram of elapsed days from waitlist entry to first offer.

Figure 8.

Histogram of counts of offers by time (in days) from entry in the waitlist process to first sent offer.

Time From Entry on Waitlist to a Self-rescheduled Appointment

The time from waitlist entry to self-rescheduled appointment via automated waitlist process is shown in Figure 9. This figure shows an important process measure: the start to finish time for the automated waitlist self-rescheduling process. The median time from entry on the waitlist to making a self-rescheduled appointment from an offer was 10.2 days, IQR (3.1-26.5), with a mean time of 19.5 days (SD 25.0).

Figure 9.

Histogram of self-rescheduled appointments by elapsed time from entry on waitlist to date/time of accepting a new appointment offer.

The graph shows that the large number of self-rescheduled appointments made in the first 3 days of being waitlisted is 13 522 or 23.9% of the total. This combined with the previous graph of time to first offer shows that patients are following up on the rapid first offer with action that in many cases leads to a rapid acceptance of the offer.

The 13 473 moved-up appointments that were self-rescheduled within 3 days of waitlist entry had an appointment time mean move up of 20.8 days (95%CI; 20.4-21.1 days), with median of 13.8 days improved, which was similar to the entire move up sample. However, the early reschedulers (those within 3 days) had a 34.0 day (95% CI; 33.6-34.4) mean original appointment wait time that was considerably lower than the mean original wait time of 52.1 days for all those accepted.

Offers Viewed and Not Viewed

Despite over 1 million offers sent, only 62% (628 756/1 019 698) of those offers were viewed within the patient portal (Figure 5). However, due to appointments having multiple offers, of the 229 998 appointments there were 180 985 appointments (79%) that had at least 1 new appointment offer viewed (Figure 4). Figure 10 shows the counts of offers sent and offers viewed by the offer number. It should be noted that patients continued to get offers as long as the original appointment continued to be waitlisted, original appointment date had not passed, and they had not accepted an earlier waitlist offer. Figure 10 shows an expected decline in offers as more and more offers were sent and accepted. Also, the interest in viewing offers diminished with the number of offers sent. Figure 10 shows the decline in percent of offers viewed as the number of offers increased (percent viewed scale on right of Figure 10). The initial offer was viewed by 72.3% (166 252 viewed/229 998 sent). By the time of the 20th sent offer, the number viewed had dropped below 5000 with a viewed percent of 51.3% (4580 viewed/8923 sent). Of course, some of this is expected. If you are getting an offer every day to move up your appointment, and your appointment is out 20 days, by the 15th day of a daily new offer you have almost reached the date of your original scheduled appointment. As such, diminishing returns are expected as the date of the originally scheduled appointment neared.

Figure 10.

Graph of appointment offer count sent (blue solid) and viewed (red dashed) by number sequence of the offer. Black triangles represent the percent of sent offers that were viewed (percent scale on the right).

Expired, Declined, and Accepted Offers

As shown in Figure 5 and explained in Methods there were 5 mutually exclusive outcomes for each waitlist offer. Each offer could be (1) accepted, (2) declined, (3) unavailable, (4) deleted, or (5) expired. For most of the waitlisted appointments during the study, offers were sent between 4 and 6 PM local time and expired within 14 hours (about when offices reopened on weekday mornings). Figure 11 shows all but the deleted offer outcomes by offer number. The logarithmic scale used in Figure 11 (Y-axis) shows that the accepted offers decline very rapidly as patients continue to get more and more offers.

Figure 11.

Counts of the 4 major outcomes of individually viewed waitlist offers (N = 628 765) by the offer number. Deleted offers not shown.

Also shown in Figure 11, most accepted offers were accepted within the first few offers. Specifically, 58.4% of those accepting did so on the first offer, and for the second, third, and fourth offers the cumulative percent accepting increased to 77.4%, 85.7%, and 90.2% respectively. The 95th percentile of acceptances occurred at the seventh offer. Also shown in Figure 11 is that there are still many offers being declined and expiring even after the acceptances have almost completely gone. It should also be noted that overall, at each offer count, there were many more offers available than already taken (the unavailable outcome).

A further clarification of expired and unavailable may be needed here. Previously available offers become unavailable once accepted by another patient. Both unavailable and expired are no longer actionable to a patient. However, the “expired” outcome category represents available offers up to the time of expiration. It turns out that many of the offers that expired were demonstrably available but were just not acted on. Of the 356 501 viewed offers that were expired, there were 76 473 that were viewed within 4 hours of being sent (before even the shortest expiration time). They expired only later because they did not get accepted or declined when they were viewed and also no one else accepted them (which would have placed them in the unavailable outcome category). They were viewed but not acted upon. The result is that an appointment offer slipped away at expiration time because no one came back to view it again and accept it before expiration. We do not know why some actively declined the available offer, while others passively let their available offer(s) expire.

Waitlist Unavailable (Taken) Offers

Of the viewed offers there were 14.6% (91 794/628 756) that were already taken. Over half of the waitlisted appointments with offers, 58.4%, were never sent an unavailable (taken) offer. This left the remaining 41.6% (95 753/229 998) of waitlisted appointments with at least 1 unavailable offer, and 7.2% (16 629/229 998) of waitlisted appointments had 3 or more unavailable offers. Of those appointments where there were 3 or more unavailable offers, 2792 (1.2% of total) resulted only in unavailable offers. So, a “waitlist strikeout” of 3 or more offers all unavailable was fortunately low and 37% (1038/2792) of those patients with 3 or more appointment offers all unavailable did not even view any of the unavailable offers.

Waitlist Responses by Time After Offer Sent

Figure 12 shows that 65.2%, (78 121 declines and 38 677 accepts/178 985 total) of all waitlisted responses (both accepts and declines) took place within the first hour after sent. Median time to respond was 25 minutes after sent with 90th percentile of responses occurring sooner than 2.7 hours after sent offer. As shown in Figure 12, the decline responses were greater than accept responses at each hour after the offer was sent.

Figure 12.

Counts of offer response outcomes by hour after offer sent (N = 178 985). Hour 1 response counts are from 0 to 1 hour after offer sent. Hour 2 counts are responses from 1 hour after sent to 2 hours after sent, etc. At the time of the study, all offers were to expire no later than 14 hours after sent. Total responses; accepted, n = 56 636; declined, n = 122 349.

The slight uptick in responses between 12 and 13 hours later was likely because during the time of the study the offers were being sent in the evening and the patients were checking in the morning before expiration, about 12 to 13 hours later.

Once viewed, the elapsed time from view to response was extremely quick. Median time from view of offer to response was 1 minute. The 90th percentile of responses were sent within 4 minutes of viewed.

Responses to Offers on Nights and Weekends

Offer views and offer responses (both accepted and declined) frequently occurred outside of office hours. For the 628 756 viewed offers, 229 368 (36.7%) were either viewed on weekends or from 7PM up to 7AM on Monday through Friday. Offer responses had a similar percent, 39.5% (70 942/178 985), occurring during those same times.

Use of the Waitlist for Short Wait Times

The automated waitlist appeared to be used appropriately for longer appointment wait times. Only 1.5% (3343/229 998) of waitlisted appointments had original wait times of less than 7 days. There were 12.5% (28 841/229 998) of waitlisted appointments with original wait times of less than 2 weeks.

Discussion

Principal Findings

About one fourth of waitlisted appointments were successfully self-rescheduled within an automated waitlist process. Ninety-five percent of the all the rescheduled appointments had a sooner appointment with a median reduction in wait time of 15 days for the moved up appointments.

The self-rescheduling offers came quickly for many. First offers came within 2 days of waitlist entry for 32.5% of the waitlisted appointments. Response speed from patients was also quick for many; 65.2% of all responses (accept and decline) came within 1 hour of the sent offer. However, many new appointment offers were also never viewed; the percentage of appointments with at least 1 sent offer that never had a single offer viewed was 21.3%.

For the 628 756 new offers that were viewed, 57% expired because they were not accepted, declined, taken by someone else, or deleted before their offer time ran out (Figure 5). The expired offers meant that the appointments slots associated with the offers were still at least temporarily open for others, but also potentially unfilled.

Practice Implications

Access to timely appointments is extremely important to practices as well as patients. Patients calling back frequently to find an earlier appointment can be a big challenge for healthcare systems and can have significant impacts on staffing needs. Staff schedulers on behalf of patients looking for earlier appointments perform a large amount of work that is not easily quantifiable. Not only do the staff schedulers spend time on rework, looking through schedules for already scheduled visits, provider time and input is often elicited as well. Patients with an appointment for a specialty visit or procedure more than a week or two in the future often message their care team or provider directly by using portal messaging to request assistance in securing an earlier appointment. To help their patients with earlier appointments, primary care providers and their teams may call specialists directly to look for access through colleague-to-colleague networking. Thus, patients desiring earlier appointments not only involve staff schedulers but rework on these scheduled appointments can travel upstream to primary care providers and consulting specialists.

Our data shows that the automated waitlist process successfully delivered new appointment offers within 7 days for a majority of the waitlisted appointments. Figure 8 shows that 50% of waitlisted appointments were given a rescheduling offer within 6 days. We do not have the data to know if this led to fewer subsequent patient calls to schedulers or to fewer patient messages. However, a Mayo Clinic study on self-scheduling screening mammograms did show a significant decrease in proportion of visits that required multiple steps to schedule (cancellations and reschedules). In that study only 6.5% of self-scheduled appointments were associated with some rework (93.5% were one and done), but 25.5% of the staff-scheduled mammograms required scheduling rework. ¹¹ Also at Mayo Clinic, for visits in a self-triage self-scheduling process, there were fewer patient messages associated with an ear and hearing concern when patients were able to self-schedule a visit. ¹⁰ It’s also possible that patients who are concerned that they have been forgotten in the scheduling process could be reassured by the automatic alerts to new offers for earlier appointments, which could result in fewer secure messages requesting a move up.

Our median move up of 15 days is similar to the findings of a University of San Francisco (UCSF) study of the Epic “Fast Pass” process, which is the same underlying Epic technology that Mayo Clinic uses for automated waitlist rescheduling. Ganeshan et al ¹⁸ found a median move up of 14 days from their sample of 6603 accepted appointments. Ganeshan et al also hypothesized that waitlist visits accepted for appointments less than 7 days in the future would likely remain unfilled. Based on that assumption, the UCSF group estimated 2576 service hours were added by the automated waitlist process and an estimated increase of 3 million dollars in revenue from appointments that otherwise would have been unused. ¹⁸ Our study was focused on the process flow and the efficacy of this new scheduling tool and did not look at pre- and post-waitlist appointment fill rates, which also might help determine how well this process functions to decrease empty appointment slots. Of course, the automated waitlist process is not a comprehensive solution for scheduling when the root cause is limited appointment supply. However, by finding unused appointment slots, and sending offers, fill rates can improve and scheduler and provider time could be saved.

A 2020 study by Chung et al, ²² performed a retrospective analysis of “Fast Pass” and found a much lower rate (8.3%) of accepted offers then ours or that of Ganeshan et al. That may be because their offers expired more quickly (after ½ hour) though they offered more appointment rescheduling opportunities throughout the day. However, this raises interesting questions about length of time before offers expire as well as timing – for instance, they found offers more likely to be accepted in the evening than in the morning. Although we did not directly compare early versus late, we did have a lot of responses outside of business hours, which could suggest this may be a more successful time to target offers to patients. Of note, they did find a reduction in “no show” rates for rescheduled visits compared to regularly scheduled visits, ²² which would support financial benefit to practice. We did not directly compare our “no show” rates and this could be an area of further study.

The large number of expired offers resulted in a decision to change the waitlist process. During the study there was no set limit to the number of appointment offers a patient could receive for a waitlisted appointment. Our current process now is to remove appointments from the waitlist if there has been no interaction at all for 10 offers (10 expired offers). For each waitlisted appointment, staff schedulers can now view the total count of expired offers so that they can potentially reach out to the patient and learn the status of their appointment request. Future study will be needed to assess the impact of the 10-offer limit on rescheduling from the automated waitlist.

Patient Implications

Online access may reduce current patient frustrations with scheduling appointments. In a 2021 survey of over 200 individuals who booked an appointment with a service provider, 42% indicated their biggest “pain point” with the process was long telephone wait times. Another 17% in the same survey said their biggest pain point was inability to schedule outside of regular office hours. ³⁰ As noted in Results, 39.5% of the 170K responses to accept or decline offers were not during weekday hours of 7AM to 7PM. Self-scheduling online or by mobile is available 24/7 at Mayo Clinic so the 2 biggest “pain points” identified with the above survey are eliminated by self-rescheduling via the automated waitlist process. Automated waitlist self-scheduling is just one of the self-scheduling processes that patients are able to conveniently use 24/7. Patients are self-scheduling 24% to 43% of the time outside of usual weekday business hours for appointments as diverse as COVID tests, screening mammograms, well-child visits, and self-triage self-scheduling for ear and hearing concerns.^10-13

The automated waitlist process seems likely to decrease the number of calls about rescheduling to an earlier time if patients believe that the system will work for them. If the number of calls is reduced, a potential secondary benefit is decreased telephone wait times for patients calling for any reason, including those who are still using the telephone to make appointments.

Offers that are taken before the patient can act deserve special attention. How does a patient react when told a new appointment offer from the waitlist was waiting in their portal but discovered that it was already taken by someone else? Perhaps that would be discouraging for some and could hamper future engagement with the waitlist process. It’s also possible that the experience of getting one or more unavailable (taken) outcomes could contribute to an overall poor patient experience even before the appointment takes place. To reduce the unavailable outcomes, there are caps on the number of patients who can get offers for the same open appointment slot. However, choosing the right cap size for open appointment offers can be challenging. Offering an open appointment to too few on the waitlist can result in an unfilled appointment; offering it to too many can result in the disappointment of those who found the appointment unavailable when they wanted to accept.

Limitations

This study is from a single large multispecialty, multisite medical practice. Rules that Mayo Clinic has for the use of the automated waitlist may not be generalizable across all practices. It also should be noted that several confounding factors including patient demographics, urgency of appointment type, and differences across clinic locations could individually and together influence outcomes. This was a retrospective observational study, focused only on outcomes of those who were entered into the automated waitlist process. The waitlist process requires portal registration and mobile or internet access so that new appointment offers can be viewed from a mobile device or computer. Thus, this process has a significant selection bias, limiting it to those with the ability to use and access the required technology.

We do not have information on patient satisfaction with this automated waitlist approach to self-reschedule. However, as noted earlier, there was a small percentage of patients who received 3 consecutive offers, all of which were taken before the offer could be accepted. Filling every available slot is a goal for this process but we will need to further examine tradeoffs involved. Frustrating patients by sending offers that are quickly made unavailable could be a downside.

We also do not know yet which appointment types are better suited for the automated waitlist process. The waitlist is being used for multiple appointment categories as seen in Table 2. This includes lab tests and procedures. However there are rules surrounding appointment move ups that complicate matters as mentioned in Methods. Mayo Clinic has been using rules to address the issue of minimum appointment lead time for specific appointments, but this is something that must be individualized by appointment type and may vary by practice and institution.

Because this was a retrospective observational study, sample size was determined by actual counts of waitlisted appointments that occurred during the yearlong course of the study. The study duration of 1 year provided an adequate sample size of hundreds of thousands of waitlisted appointments to analyze. However, further study will be needed to look at trends over time for this new technology.

Future Research

In this study we were focused on the automated waitlist process and a successful reschedule to an earlier appointment date. Future research will be needed to see the impact of the automated waitlist on no show rates, cancellations, and additional reschedules. Patient satisfaction surveys and user experience studies could also be used to identify ways that the automated waitlist process can be improved from the user perspective.

We need to determine why large numbers of waitlisted appointments were sent offers that did not get viewed. To do this, we need to examine patients’ understanding of the process, their expectations, their preferred communication methods, and study what appointment types and circumstances are best for waitlist technology. As an example for further study, patients will soon be able to respond to offers directly from a text message without need to log into their portal account. This may change response rates and requires further study.

After data collection for this study concluded, we started running 2 daily batches of appointment searches. In addition to the one late in the late afternoon, another batch appointment search and offer was added at midday. A 4-hour expiration applied to both batches. Future study needs to be done to understand optimal timing and frequency of the automatic searches for open slots. Tied to that is the need for research on the optimal number of offers per open appointment slot and criteria for dynamic adjusting of that number.

Conclusion

The automated waitlist process to self-reschedule existing future appointments can be used to successfully move up appointments. New appointment offers were accepted in about one fourth of the waitlisted appointments and for most resulted in a significant reduction in wait time. After entry on the waitlist, offers are rapidly generated, and rapidly responded to, resulting in the self-rescheduling of large numbers of appointments. However, there are also large numbers of expired offers and offers that are never viewed.