Impact of IMPACT: Longitudinal Analysis of an Integrated Participant Scheduling System in a Clinical Research Setting

Abstract

Rapidly increasing costs have been a major threat to our clinical research enterprise. Improvement in appointment scheduling is a crucial means to boost efficiency and save cost in clinical research and has been well studied in the outpatient setting. This study reviews nearly 5 years of usage data of an integrated scheduling system implemented at Columbia University/New York Presbyterian (CUIMC/NYP) called IMPACT and provides original insights into the challenges faced by a clinical research facility. Briefly, the IMPACT data shows that high rates of room and resource changes correlate with rescheduled appointments and that rescheduled visits are more likely to be attended than non-rescheduled visits. We highlight the differing roles of schedulers, coordinators, and investigators, and propose a highly accurate predictive model of participant no-shows in a research setting. This study sheds light on ways to reduce overall cost and improve the care we offer to clinical research participants.

Introduction

U.S. health care costs in 20i8 totaled $3.6 trillion, or $11,172 per person, according to the Centers for Medicare & Medicaid Services [1]. Due to these continuously rising expenditures, extensive effort has been made to curb costs with one key focus being appointment scheduling as it serves the primary purpose of providing timely access to health services [2]. Despite its importance, previous research has highlighted a number of deficiencies in current scheduling, where missed participant appointments, waste of participant and clinician time [3], and logistical confusion have caused a serious problem for healthcare institutions across the country [4]. Research has also been prioritized to predict missed appointments, or 'no-shows,' which have been shown to increase delays for other participants, increase health care costs, and increase the likelihood of adverse health outcomes [5-7] while reducing these 'no-shows' has been shown to improve efficiency and resource utilization, improve staff satisfaction, and leads to better health outcomes [8 9]. As such, appointment scheduling has been a key area of research in the routine clinical care/research space.

Rising costs are also a significant issue in the field of clinical trials. With an average single trial cost of $19 million [10] and a projected global expenditure of $69.8 billion by 2027 [11], boosting clinical efficiency is crucial in this field where costs are rapidly becoming prohibitive. Randomized clinical trials provide high-quality medical evidence to guide medical decision-making, but their high cost severely threatens the clinical research enterprise [12], highlighting a need for novel approaches to cost-cutting. Research settings also commonly have unique logistical requirements for visit scheduling, including the need for specialized equipment or resources [13], the constraints from complex study calendars[14 15], lack of staff support [16], and the burden of in-office participant recruitment which can be a time-consuming disruption of normal clinical workflow [17]. However, little previous research has investigated the unique needs and opportunities of appointment scheduling in clinical research.

One way to boost efficiency and save cost in such an environment involves the coordination of regular clinical visits with the more restrictive research appointments, though this is no trivial task. There is a designated schedule for research visits (the study calendar), and these visits can have very strict time windows (such as "visit one must occur one month, plus or minus fifteen days, after the randomization visit" or "Chest X-ray will be done at 6 day+/-1, i2 day+/-1, i8 day+/-1, etc. during the treatment period.") and as such, clinical research coordinators (CRCs) often perform this task manually to ensure they are scheduled within the protocol-defined windows [18]. Additionally, different visits might require different time commitments, with some visits including only a blood draw while others might also include regular imaging studies or additional diagnostic tests. It frequently falls to the CRCs to find a date when all resources are available and there is sufficient time to accomplish all of the required tasks.

In order to address this pressing concern, we have previously designed and validated the Integrated Model for Patient Care and Clinical Trials (IMPACT) software system to coordinate research visits with participant clinical care visits in academic medical centers, with its technical design previously reported in [19]. By providing access to both clinical schedulers and research coordinators and by connecting electronic medical records with clinical research visits for clinical trial participants, this IMPACT system allows for far greater optimization of available resources and coordination between clinical and research needs. Since its adoption within our CTSA Program hub in 20i4, IMPACT has enabled paperless scheduling in our Clinical Research Resource (CRR) outpatient research units and improved clinical research efficiency, team awareness, and collection of fine-grained quantitative measures to facilitate dynamic and collaborative research resource management for clinical research sites. According to previous evaluations with clinical research coordinators in CRR, IMPACT demonstrated better usability than existing clinical research scheduling functionality provided by various software platforms, including WebCAMP [20], and other general-purpose scheduling software such as Microsoft Outlook.

IMPACT has been in use in adult services since 20i4 and expanded to include pediatric services starting in 20i7 with a warm reception by clinical research staff members. As of March 2020, IMPACT has served > 180 clinical trial studies and is being used by > 150 clinical research coordinators, serving > 1200 appointments each month. However, due to our institution's recent migration to Epic, IMPACT will be phased out later in 2020. Before its retirement, IMPACT still has a mission, which is to help us understand and characterize clinical research visit scheduling and identify where opportunities and challenges reside. This study collected longitudinal data over its 5 years of clinical use and provides original and unique insights into clinical scheduling in one research setting. Leveraging this expanse of collected data, we explore both participant and employee characteristics surrounding outpatient visits, participant attendance, ways to improve scheduling efficiency, and develop a statistical model to identify participants at high risk of missing visits or canceling ahead of time in a clinical research setting. We also discuss the implications and possible generalizability of our findings. Finally, this data can serve as a valuable bench mark to compare with the research scheduling efficiency using Epic in the future.

METHODS

Iterative Participatory Design of IMPACT

There are four roles within the CRR: Scheduler, Coordinator, Investigator, and Administrator.Both the scheduler and administrator are employees of the CTSA whereas the coordinator and investigator, in our model, are employees of various departments within the broad campus of CUIMC/NYC. Their job descriptions are as follows: Schedulersprimarily work at the front desk and focus on scheduling, rescheduling, and cancelations based on communication with research team members. They are also responsible for approving or rejecting appointments directly requested by Coordinators within IMPACT. Coordinatorsare primarily responsible for managing various research protocols at CUIMC/NYP and require the use of CRR resources, including IMPACT. Coordinators are allowed to request appointments, pending review and approval by Schedulers, but are not able to cancel, reschedule, or register new participants. Investigatorsconduct or collaborate on research protocols at CUIMC/NYP and have the same privileges within IMPACT as Coordinators. Administratorsare primarily responsible for normal database operation, though they may aid in scheduling events on rare occasions. Of note, all appointments are listed as "Scheduled" upon creation of the appointment and are expected to be updated accordingly following the visit so visits remaining as "Scheduled" were removed from the following analyses.

We started the development of IMPACT in 2010 following an inclusive, iterative process that used ideas and feedback from various stakeholders in the appointment scheduling workflow. Early aims of the project centered around a simple scheduling tool, but in collaboration with clinical research coordinators and investigators, increased emphasis was placed on creating a collaborative platform between research coordinators and schedulers to improve appointment scheduling. Beyond cooperation alone, customized user views were created for each role in the clinical research team based on their intended usage (e.g. schedulers view the calendar week-by-week for better appreciation of daily availability, coordinators view the calendar month-by-month for faster review of upcoming visits and protocol needs as seen in Figure 1). Specific workflow rules were also put in place based on the user's role for a more streamlined collaboration as described above. Finally, the IMPACT tool was connected to other clinical databases (e.g. WebCAMP) within CUIMC/NYP to automate necessary and repetitive tasks often performed when scheduling participant appointments. Prototyping and software evaluation was completed using Participatory Design methods in 2014, when the final product was implemented in the Clinical Research Resource (CRR) at Columbia University's Irving Institute for Clinical and Translational Research (IICTR), the recipient of a Clinical and Translational Science Award (CTSA). This product has a simplified design of [19], since some of the research-focused elements described in the original project were removed due to logistic or technical restrictions.

Figure 1. Sample screenshot of the IMPACT scheduling tool for Schedulers (left side) and Coordinators (right side).

Data Collection

Appointment scheduling information was collected from July 1st, 2014 to December 31st, 2019. Log data were collected automatically by IMPACT with two main classes of user activities: 1) logins and 2) appointment creation and modification. Logins records are created when a successful validation of user credentials from a Central Authentication Service (CAS) is received. When Schedulers create appointments, an appointment record is created with a "Scheduled" status along with the details of the appointment including the creation time, the start and end time stamps, the resources required, the research study staff overseeing the visit, and the covering physician. If research study staff (Coordinators, Investigators) create an appointment, the appointment is registered with a "Pending" and a Scheduler may approve the appointment (moving it to "Scheduled") or reject it (changing status to "Rejected"). IMPACT also allows the research study staff to modify existing appointments such that if an appointment has a time change or resource change, it is considered a rescheduled appointment. This action marks the existing appointment record in the database with a "Rescheduled" status (or "Rescheduled by Moderator" if a Scheduler changes the visit) and creates a new appointment record with the updated information while both of these records maintain a single visit id to allow for later joining of these appointments. For clarity through the remainder of this manuscript, appointment will refer to all events scheduled in IMPACT and visits will refer to all events included in and leading up to an end-stage patient encounter: Cancelled, Missed or Completed. This includes any reschedules appointments beforehand. This status flow is also depicted in Figure 2. Rooms are located on a single floor in the hospital except for rooms marked VC3 which are located on the third floor of a separate building nearby. Additionally, two clinicians have separate rooms for their own participants and their names have been changed to Docl and Doc2 to maintain anonymity.

Figure 2. Visit status flow outlining temporary status (yellow) and final status (green). Visit count in each status also included.

Data Analysis

For each appointment, a variable was created titled scheduled ahead which measured the difference between the scheduled date and appointment date. For rescheduled visits, leveraging the visit id described above, relevant appointments could be linked and to identify changes within rescheduled visits, the appropriate characteristic (e.g. covering MD, room) was collected for the first appointment and the final appointment within the single rescheduled visit and compared. A ratio was also generated for each room called the outflow ratio, which was calculated as # visits changed out of the room / # visits changed into the room, where ratios close to zero have more visits changed into the room, high ratios have more visits changed out of the room, and ratios of one have an even number of visits changed into and out of the room.

To analyze room reservation time, the total amount of time for scheduled visits and the total amount of time where a room was active (between checkin and check out) was summed for each room and each day, accounting for overlapping time periods. The rooms were then grouped according to their designated use and the scheduled time and active time were averaged, excluding weekends. To measure time to reschedule, a room was considered vacated when a scheduled appointment was rescheduled to a different time or canceled and the time from cancellation/rescheduling to the creation of a new appointment was considered the "time to reschedule." The vacated room designation was temporary and the room could be scheduled or filled by another appointment. A chi-squared test was conducted for all categorical variable comparisons with a p < 0.05.

Usage Statistics

Usage statistics were calculated for all users according to their designated role within the department: Scheduler, Coordinator, Investigator, and Administrator. The beginning and end of each session was recorded as the time of login and the logout, respectively, for billing purposes and session time is the total time between start and end of each session. Of note, closing the active browser window did not register a logout time so the end of a session was inferred from the time of last activity when no logout time was collected. Additionally, the IMPACT tool had a login-cookie expiration time of 20 minutes so any period of inactivity greater than 20 minutes forced the user to re-login to the tool. Sessions where at least one appointment was scheduled or modified were considered to be active. Only staff who had at least one login attempt to the IMPACT tool were counted for this analysis.

Predictive Modeling

Using the collected data, a predictive classification model was generated to identify participants at high risk of missing visits. For large-scale prediction and classification tasks, generalized linear models such as logistic regression are widely used due to their scalability and interpretability, but they often struggle to utilize feature combinations which rarely occur in historical data. Embedding-based models are more capable in finding new feature combinations by learning a low-dimensional dense vector (e.g. embedding) for each item or feature, although these models require a much larger number of parameters when compared to generalized linear models.

To find the most accurate predictive model and demonstrate usability for prediction tasks, three different models were tested here. Logistic regression, a simple neural-network classifier with a multi-layer perceptron (MLP), and a wide-and-deep model that has advantages of both generalized linear and embedding-based models. Figure 3 depicts the basic structure of the models. Raw sparse features from the collected data are used in both the logistic regression model and the wide component of the wide and deep model.

Figure 3. Logistic Regression (left). Wide and Deep Model (middle). MLP Classifier (right).

Dense features are processed from the raw sparse features, then fed into the neural network of MLP classifier and the deep component of the wide and deep model. Dimensions of the dense features and the number of fully connected layers in the MLP classifier and the wide and deep model are hyperparameters tuned later on.

Table 1 describes the features used to generate these models. In total, 13 features are used. Categorical variables are converted to given dimensional embeddings that are proportional to the number of categories in the variable and concatenated with numerical variables to generate a dense feature. In our model, hasservice, wasrescheduled, is_clinical_trial, and allow_peds were converted into 4-dimensional embeddings respectively, and visitrange is converted into 16-dimensional embedding. Trainable parameters in the model are weights in the wide component and the fully connected layers in the deep component and the embeddings for categorical variables. To account for an unbalanced training dataset (low number of 'Missed' visits), we under-sampled other labels and trained over 20 epochs. All hyperparameters were set in accordance with the original publication [21]. The model was implemented using Tensorflow 2.0.0 [22].

Table 1. Features used for visit no-show predictive model.

Feature	Feature Type	Description
start_dt	numeric	Time of appointment.
visit_range	categorical	“early morning”, “late afternoon”, “lunch time”, and “common hours.”
scheduled_duration	numeric	Duration of scheduled appointment.
scheduled_ ahead	numeric	Days ahead when appt is scheduled.
canceled_pct	numeric	Ratio of canceled/total appointments.
completed_pct	numeric	Ratio of completed/total appointments.
missed_pct	numeric	Ratio of missed/total appointments.
rescheduled_ pct	numeric	Ratio of rescheduled/total appointments.
has_service	categorical	If there is an addtl. service scheduled
was_ rescheduled	categorical	If the visit was previously rescheduled
num_ reschedules	numeric	Number of times visit was rescheduled.
is_clinical_trial	categorical	Is study registered as clinical trial.
allow_peds	categorical	If trial allows participants < 18yo.

RESULTS

Descriptive Statistics

Data were collected on 71,788 total appointments/scheduling events and 47,548 unique participant visits. The status of these appointments and visits is shown in Figure 2 and the distribution over time is shown in in Figure 4 (adjusted means Rescheduled or Canceled). An additional visualization of overall visit status is shown as a sunburst plot in Figure 5.

Figure 4. Visit count by status each quarter.

Figure 5. Sunburst plot of visit status (total visit count = 71,788).

Of note, all appointments are listed as "Scheduled" upon creation of the appointment and are expected to be updated accordingly following the visit. Visits remaining as "Scheduled" (n = 2,417) were considered failed effort and were removed from the following analyses. The average number of successful visits each day (excluding rescheduled, canceled, and rejected visits) was 25.9 ([1, 53], std. dev. = 8.5) and the median number of appointments was 26. Additionally, when excluding post-appointment scheduling, the mean time a visit was scheduled ahead was 13 days and 22 hours ([0 minutes, 1109 days], std. dev. = 30 days 18 hours). Of note, 9,993 (13.9% of all appointments) appointments finished scheduling after the visit was scheduled to begin (e.g. final scheduled_at time was 9:07am for a visit that was scheduled for 9-10am). Linear regression analysis showed that visits with greater scheduled_ahead time (scheduled further in advance) were less likely to be Completed and more likely to be Rescheduled, Canceled or Missed.

Rescheduled Visits

Of the 71,788 total appointments, there were 47,548 (66.2%) unique participant visits which may or may not have been rescheduled. In total, 15,616 visits (32.8%) were rescheduled at least once with an average of 1.95 reschedules ([1,11], std. dev. = 1.1 times) and 31,932 visits (67.2%) were never rescheduled. Characteristics of rescheduled appointments can be seen in Table 2 (due to the design of the IMPACT tool, no space was available for 'reschedule reason,' so these characteristics are tabulated across all rescheduled visits).

Table 2. Characteristics of rescheduled clinic visits.

Reschedule Change	Number (Percentage; n = 15,616)
Change of Room (only)	6,513 (41.7%)
Change of Start Time	5,204 (33.3%)
Change of Staff	4,551 (29.1%)
Adding Resources	2,252 (14.4%)
Change of Doctor	35 (0.2%)

The change in rooms was further explored using the outflow ratio and this value is plotted for each room and sorted from lowest to highest value in Figure 6. The highest outflow ratio was found in Barr Interview Room 1 with a value of 2.0 and the lowest ratio was found in the Dental room with a value of 0.26.

Figure 6. Flow rate for each clinical research visit room.

When reviewing the flow of the 15,616 rescheduled visits to final status, 11,844 (75.8%) visits were Completed, 2,465 (15.8%) visits were Canceled and 929 (6.0%) were Missed. Further, it was found that a visit being rescheduled had a significant effect on its final status, X₂ (2, N = 44,862) = 1215.1, p < .01. Post-hoc tests showed that rescheduled visits were less likely to be Canceled [X₂ (1, N = 44,862) = 1043.2, p < .01], less likely to be Missed [X₂ (1, N = 44,862) =57.0, p < .01], and more likely to be Completed [X₂ (1, N = 44,862) = 1167.7, p < .01] than visits that were not rescheduled.

Room/Resource Efficiency

The visit rooms were reserved for an average and median of 4.0 hours per day ([0, 13.5], std. dev. = 2 hours). Rooms were active for 83.8% of the time they were scheduled per day ([0%, 1,883%], std. dev. = 40.5%) with only two rooms averaging more active time than scheduled time over the 66-month study period.

Regarding rescheduling efficiency, rooms were scheduled 2 days and 11 hours after a previous visit had been canceled or rescheduled, on average. When looking at individual rooms, the highest average time to reschedule a room was 6 days, 11 hours (of note, this room is typically used for a single clinician's clinic) and the lowest average time to reschedule a room was just under 17 hours. Breakdowns of these statistics by room type are displayed in Table 3.

Table 3. Room reservation statistics by room type.

Room Type	Hours Sched Per Day	% Sched as Active	Time to Fill Room
Exam Room	5.01	80.60%	3 days 0 hrs
Interview Room	3.74	82.10%	4 days 3 hrs
General Purpose Room – VC3	3.73	80.90%	2 day 8 hrs
Procedure Room (Dental, Phleb)	3.37	94.80%	2 day 7 hrs
General Purpose Room – Doc1	2.54	81.80%	3 days 22 hrs
General Purpose Room – Doc2	2.39	106.30%	5 days 1 hr

Usage Statistics from Access Logs

Statistics on IMPACT usage and activity were analyzed according to the user's designated role on any given research protocol and are shown in Table 4. Schedulers had the longest average session at almost 48 minutes, the highest number of scheduling events (new visit scheduling or modifying existing appointments) per day and had the highest percentage of active time with 38.7% of all session time being used to schedule appointments. The number of sessions per day differed very little between schedulers, coordinators, and investigators, though schedulers again saw a higher percentage of total sessions being used for scheduling than coordinators or investigators. Finally, in summary, schedulers were responsible for nearly 80% of all scheduled events in the IMPACT platform.

Table 4: Usage statistics based on user role.

Role	Staff Count	Avg Session Time (min)	Pct Session Time Active	Sessions Per Day	Pct Sessions Active	Scheduling Events per Day	Contribution to Scheduled Events
Scheduler	12	47:39.2	38.7%	11.1	16.2%	16.3	79.5%
Coordinator	186	12:13.5	21.7%	10.2	12.2%	2.2	19.9%
Investigator	11	39:37.4	21.7%	11.1	12.0%	1.8	0.5%
Administrator	5	36:12.1	10.7%	1.5	2.5%	0.3	0.1%

Prediction Results

Predictive models achieved the accuracy and Area-Under-the-Curve (AUC) scores as shown in Table 5. Logistic Regression achieved the highest level of accuracy with an AUC of 0.92 and a predictive accuracy of 82.7%. The weights of all features used in the Logistic Regression predictive model are shown in Table 6 (where higher values indicated more predictive of missed visits and lower numbers indicate more predictive of not-missed visits). The feature with the highest weight in predicting missed visits was percentage of previously missed visits and the percentage of completed visits was the most negatively predictive of missed visits.

Table 5. Predictive modeling accuracy of missed visits using various methods.

Prediction Method	AUC	Accuracy
Logistic Regression	0.92	82.7%
Wide and Deep	0.81	72.4%
MLP Classifier	0.72	62.9%

Table 6. Feature weights from Logistic Regression prediction model.

Feature	Feature Level	Weight
start_dt	N/A	0.34
visit_range	Early Morning	-0.26
	Late Afternoon	0.29
	Common Times	0.12
	Lunch Times	0.07
scheduled_duration	N/A	-0.03
scheduled_ ahead	N/A	0.28
canceled_pct	N/A	-3.28
completed_pct	N/A	-3.62
missed_pct	N/A	9.69
rescheduled_ pct	N/A	-0.86
has_service	Yes	-0.03
	No	0.12
was_rescheduled	Yes	-0.86
	No	0.88
rum_reschedules	N/A	2.99
is_clinical_trial	Yes	0.01
	No	0.11
allow _peds	Yes	-0.19
	No	0.11

DISCUSSION

Over the four and a half years of data collection (2014-2019), the IMPACT initiative has collected a significant amount of information regarding various visit characteristics and scheduling activities. With the scope of the data collected, we hope that findings presented and discussed here will help inform clinical research offices ways to improve the efficiency of their outpatient scheduling and reduce their overall cost of providing care. Further, this dataset was used to generate a highly predictive tool for appointment no-shows to allow for preemptive schedule adjustments to further contribute to the improvement in efficiency and cost savings.

As this data were collected in a research setting, they may not reflect the data analysis needs in a standard outpatient clinic. Nonetheless, with an average and median daily visit count around 26 visits, a peak single-day count of 53 visits, and a relatively small standard deviation, it is clear that the outpatient research service at Columbia remained very busy and required a large number of staff to maintain. Conversely, scheduling ahead information found a wide variability in time ranges, with an average scheduled ahead time of around 14 days, median time of around 6 days, but a standard deviation of over 30 days. This is likely due to two primary reasons. The first is as this clinic serves research participants who must adhere to a strict protocol schedule, the visits must be scheduled adhering to standard time windows, including 'every 7 days,' 'every 30 days,' and 'every 90 days,' so it is possible that multiple research visits were scheduled at a single time for convenience, providing a very variable distribution.

Another possible reason for this wide distribution in scheduled ahead time is the high rate of rescheduling visits. With over 15,000 visits being rescheduled, up to a maximum of 10 times for a single visit, this is clearly a large area of staff time investment and an opportunity to improve efficiency. The most frequent feature of rescheduled visits is changing the visit room (e.g. rescheduling from Exam Room 3 to Interview Room 1) with over 40% of reschedules experiencing this. It is important to note that this reflects rescheduled visits without any associated time change, suggesting that room availability is a primary cause of the majority of rescheduled visits. Future research efforts should aim to include analysis of ideal room accommodation for visits to better assess how availability impacts participant scheduling.

Further investigation of the resource utilization phenomenon showed a high outflow rate for the clinician-specific room Interview 1 as well as multiple rooms in the neighboring Vanderbilt Clinic 3 building. Alternatively, a low outflow rate was observed for the partner clinician-specific room Interview 2, highlighting a possible preference among schedulers for that clinician, and both procedure room Phlebotomy and Dental. The low outflow rate of procedure rooms highlights the occasional need for specialized equipment, such as dental drills or lumbar puncture needles, and reschedules can occur when the required instruments are not available in the original space. One of our future works is to take this empirical knowledge to design a practically useful automated scheduling algorithm that weighs provider priority, preference for rooms, and other factors identified above.

Differences in relative use rates for the various clinic rooms also arose during the analysis. The highest amount of daily use was seen in multi-use clinic space such as exam and interview rooms both in the primary clinic space and in the nearby Vanderbilt Clinic 3 building. Lower daily use was observed in the more narrow-focused procedure rooms and the lowest usage along with highest time to reschedule a room was observed in single clinician visit offices. Notably, the lowest time to fill an empty room was found in the two procedure rooms. Finally, when comparing rescheduled to non-rescheduled visits in terms of final visit outcome, a significant difference was found for all three outcomes such that rescheduled visits showed a higher likelihood of being Completed while non-rescheduled visits showed a higher likelihood to be Canceled or Missed. This finding suggests, along with the relative importance of participant scheduling history in the logistic regression prediction model, that participants who are more active in their clinical and research care may be less likely to miss a scheduled visit. However, further research would be needed to confirm this possibility.

Significant variations in usage of the platform were also observed between different user roles as has been described previously [23]. As expected, schedulers had the longest average session times, most active sessions and scheduled nearly 80% of all appointments in the clinic. Much lower active session time was seen in coordinators and investigators, suggesting most of their usage may have been to check on calendars and review appointments instead of actively schedule participant visits, which aligns closely with predicted usage outlined during development. Additionally, coordinators had much shorter average sessions times than other users, suggesting a much more targeted and focused usage of the platform than either schedulers or investigators. Administrators also showed a very low active session time, but as the majority of their role is non-clinical, this is expected.

Towards a predictive model, the most accurate method used a logistic regression model using 13 features. Table 6 confirms that the logistic regression model outperforms wide and deep model and MLP classifier. This may indicate that a clear pattern exists between characteristics of missing visits and completed visits, allowing the logistic regression model to efficiency capture frequently occurring patterns in binary classification. We expect, however, the embedding-based models will show improved performance in a more complicated prediction setting (e.g., prediction of multiple visit labels). Based on previous work, the accuracy of this model is on par with current state-of-the-art tools [6 7]. However, this work extends these previous efforts in one important way. Where previous research has focused on clinically-oriented offices [24 25], the factors which influence scheduling in a research setting are different and have not been explored in this capacity before. As such, findings presented here provide a novel prediction tool which may be used in a wide array of clinical research settings.

Implications and Recommendations for Best Practices

Though data used for this analysis originated at a single institution, findings described here provide important insights into best practices for sites across the country. One of the most notable findings was the high rate of room or resource changes in rescheduled visits, suggesting relatively fluid needs in resource utilization and allocation. Concerns surrounding resource utilization have been discussed as far as back as the last century [26], but findings here highlight that it is not an issue unique to outpatient clinics and adds complication to clinical research efforts as well. Improved planning or appreciation of participant preference may provide some means to reduce scheduling complexity and have even been shown to reduce wait times and improve resource utilization [27]. It is important to note that not all room changes are due to insufficient planning and can be due to day-of-visit needs, but it is difficult to propose solutions to this complexity common in many fields of medicine. Another notable finding is that rescheduled visits were more likely to be completed than non-rescheduled visits. When considered in conjunction with the role of participant scheduling history in our predictive model (e.g. previous missed visits make it more likely the participant will miss again), this suggests an important aspect of participant history in appointment scheduling. Special consideration should be taken of participants with scheduling issues in the past to avoid any future no-shows and improve overall clinical workflow. Also, from user interface design improvement perspective for scheduling software, better support should be given to rescheduling considerations and to make it easy for rescheduling. Recommendations of available rooms and equipment can be made automatically to schedulers or coordinators.

Limitations

This study has several limitations. As much of this analysis focused on rescheduled visits and clinic efficiency, much of the interpretations regarding rescheduling behaviors warrant further user confirmations. As stated in the Methods section, there was no instrument available for direct capture of the rationale behind rescheduling, so this analysis relied on inferred reschedule details. Another limitation was the absence of proposed research-focused modules being deployed within the IMPACT platform. In the initial publication, there were plans for a large amount of direct connection between clinical participant data and trial-specific research data. However, as the initiative was being implemented in the real-world, some of these modules became cumbersome or overly time-consuming and as IMPACT served as the primary scheduling platform for all clinical research visits, convenience was the most important function, so some complex modules were dropped. A third limitation is the lack of clear data on the relative complexity of a given visit. For example, a visit involving blood draws and chemotherapy administration would be more complex than a simple participant interview. This has been highlighted previously as an important factor in assessing participant attendance [28] and future work may be able to better appreciate this complexity factor.

Future Directions

Though data collected throughout this initiative were expansive, one possible method to improve the quality of data is to include more granular information regarding the specific protocol visit such as visit purpose (e.g. treatment vs. screening visit) and visit schedule (e.g. Month 2, Week 3). This information would allow for a deeper analysis of staff and participant habits surrounding specific clinical research protocol activities and may provide means for research staff to prepare for difficulties before they arise. Additionally, more information about specific participants would likely improve the predictive power of our model. Much of a participant's relationship with research is experienced outside of the clinic so having a better understanding of their current medical condition may allow for research staff to better care for our participants. Finally, with the increasing use of online technologies to improve clinic flow for research staff and participants, focused assessment of the utility and impact of appointment reminders should be performed to further assist in clinical research scheduling.

Conclusions

Leveraging clinical appointment scheduling data collected with the IMPACT tool over the past 5 years, this study was able to provide several new insights into appointment scheduling, particularly in the research setting. Among other findings, it was found that one of the primary drivers for rescheduled events was room changes, visits that were rescheduled were more likely to be eventually completed instead of being canceled or missed, significant variations in how the platform was used were apparent between the different roles of research staff, and a predictive tool for participant no-shows was able to reach an accuracy of 85.5%. Though the field of appointment scheduling is a rapidly expanding area, little previous work has focused on the needs of clinical research offices and this work provides novel insight into how best to provide care and improve efficiency in those types of settings.

Acknowledgments

Funding for this research was provided by the following grants: R01LM010815 (PIs: Weng and Bigger) and CTSA grant UL1TR001873 (PI: Reilly). Acknowledgement to Janelle Nunez and the entire Clinical Research Resource (CRR) staff who were instrumental in the design and implementation of IMPACT in the clinical research setting.

Author Contributions

AB was the primary author of the manuscript. JL and YS assisted with data collection and analysis. LB, KM, HG, DF, IC, EG assisted with revision of the manuscript. CW was primary student advisor and PI for the study.