Applying operations research to optimize a novel population management system for cancer screening

Abstract

Objective

To optimize a new visit-independent, population-based cancer screening system (TopCare) by using operations research techniques to simulate changes in patient outreach staffing levels (delegates, navigators), modifications to user workflow within the information technology (IT) system, and changes in cancer screening recommendations.

Materials and methods

TopCare was modeled as a multiserver, multiphase queueing system. Simulation experiments implemented the queueing network model following a next-event time-advance mechanism, in which systematic adjustments were made to staffing levels, IT workflow settings, and cancer screening frequency in order to assess their impact on overdue screenings per patient.

Results

TopCare reduced the average number of overdue screenings per patient from 1.17 at inception to 0.86 during simulation to 0.23 at steady state. Increases in the workforce improved the effectiveness of TopCare. In particular, increasing the delegate or navigator staff level by one person improved screening completion rates by 1.3% or 12.2%, respectively. In contrast, changes in the amount of time a patient entry stays on delegate and navigator lists had little impact on overdue screenings. Finally, lengthening the screening interval increased efficiency within TopCare by decreasing overdue screenings at the patient level, resulting in a smaller number of overdue patients needing delegates for screening and a higher fraction of screenings completed by delegates.

Conclusions

Simulating the impact of changes in staffing, system parameters, and clinical inputs on the effectiveness and efficiency of care can inform the allocation of limited resources in population management.

Introduction

Changes in healthcare delivery are being driven by the need to address both rapidly rising costs and clear gaps in the quality and safety of care provided. One novel way to reorganize care is through the use of health information technology (IT) systems that take a population-based perspective and enable care delivery outside of the office visit. At our institution, we have implemented such an IT system, called technology for optimizing population care in a resource-limited environment (TopCare), for population-based cancer screening in a large primary care network.

Given current healthcare payment models, the limited resources available for novel population-based activities led us to examine how to optimize the effectiveness and efficiency of such IT systems using two methods derived from the field of operations research.¹ First, queueing theory is used to model the inherent complexities of a healthcare IT system to help decision-makers understand the core workflow interactions and bottlenecks.² Second, simulation analysis allows quantitative evaluation of multiple ‘what-if’ scenarios in order to identify key parameters associated with system inefficiencies. As few studies have employed both these techniques to evaluate a health IT system, this study applied queueing theory and simulation analysis in order to assess the potential impact of¹ changes in staffing levels for patient outreach activities,² modifications to user workflow within the IT system, and³ changes in cancer screening recommendations.

Background and significance

The need for population-based, health IT

As part of the 2001 Institute of Medicine report on quality of care, the need fundamentally to redesign care delivery was highlighted.^{2 3} A meta-analysis of 108 articles identified a combination of organizational changes, financial incentives, and patient reminders that have been shown to promote effective preventive care redesign.⁴ In our organization, in response to primary care redesign efforts embedded in patient-centered medical home activities that included support for new payment models, we had an opportunity to develop IT infrastructure to support population management. The result was TopCare: a re-engineering of our approach to preventive cancer screening by shifting the focus from solely episodic reminders during office visits to a broad, visit-independent, population-based management perspective.

TopCare enables primary care providers to view their panel of patients who appear overdue for breast, cervical, or colorectal cancer screening and select an appropriate contact mechanism or defer screening based on their knowledge of the patient. Contact mechanisms include a mailed letter, phone contact from a practice delegate responsible for scheduling screenings and appointments, or patient navigators (high-risk case managers) who have special training to provide more intensive outreach to those at high risk of non-adherence due to language barriers, missed appointments, or multiple overdue screenings.⁵ TopCare is designed to supplement visit-based reminders by screening, contacting, and tracking eligible, overdue patients without the need for face-to-face contact, thereby increasing rates of preventive care in large primary care networks.⁶

Motivation for incorporating optimization in the workflow design process

Considerable evidence supports the efficacy of reminder systems for preventive services. However, fewer studies have focused on the effectiveness and efficiency of such systems when applied in routine practice settings on an ongoing basis. While health IT systems are designed for a broad user audience, they must be optimized to local workflow needs in order to be effective.⁷ Optimization guided by analytical models can highlight system bottlenecks and process inefficiencies.^{1 8 9} Operations research, unlike more traditional techniques (eg, regression analysis), involves analytical methods capable of accounting for uncertain care environments constrained by limited resources. In particular, operations research techniques can lead to opportunities for more effective and finer-tuned control of operations, practical solutions for reducing costs, and increased worker productivity and employee morale.^{9 10}

Simulation modeling is an alternative method capable of examining increasingly complex, real-life healthcare systems with few mathematical assumptions.^{8 11 12} Simulation provides a valuable environment to examine numerous what-if scenarios without affecting real-time system operations. Although requiring more time to build, debug, and validate, simulation models have been used to improve a wide range of healthcare processes, including inpatient bed allocation, patient flow optimization, and hospital process evaluation.^{8 13–19} However, the role of queueing theory and simulation models to evaluating health IT systems is still being defined.^{12 20 21}

Our goal was to use operations research techniques to simulate changes in patient outreach staffing levels, modifications to user workflow within the IT system, and changes in cancer screening recommendations.

Methods

Queueing system modeling: patients, cancer screenings, servers, and current screening parameters

TopCare was modeled as a queueing network based on the workflow described in figure 1. In particular, we modeled patients’ cancer screening status as ‘flow units’ representing the flow of each overdue cancer screening through a multiphase, multiserver queueing system.² At any given time, cancer screenings can be in one of three locations—master list, delegate queue, and navigator queue.

Figure 1.

Activity diagram of the redesigned workflow for population-based cancer screening.

In the master list, we defined three random variables: time-to-screen for colorectal cancer; time-to-screen for cervical cancer; and time-to-screen for breast cancer (table 1). A time-to-screen for a cancer type corresponds to the interval time between two consecutive screenings for the cancer. Based on our primary institutional cancer screening prevention guidelines, we also set the ‘dues’ (maximum times allowed) in the master list to 10 years for colorectal cancer, 3 years for cervical cancer, and 2 years for breast cancer.

Table 1.

Nine possible events with their corresponding control statements in the simulation model

Event	Master list	Delegate queue	Navigator queue
Screening entrance	According to the cancer type of an entering screening, ▸Set a due to be 10 (colorectal), 3 (cervical), or 2 (breast) years ▸Assign a time-to-screen sampled from a relative frequency distribution with a mean of 103.5 (colorectal), 41.2 (cervical), or 30.9 (breast) months	Regardless of cancer types of an entering screening, ▸Set a due of 4 months ▸Assign a processing time sampled from the exponential distribution with a mean of 0.029 months	Regardless of cancer types of an entering screening, ▸Set a due of 6 months ▸Assign a processing time sampled from the exponential distribution with a mean of 0.09 months
Screening completion	▸Return to the master list after completed screening	▸Return to the master list after completed screening	▸Return to the master list after completed screening
Due	▸Transition the overdue screening to the delegate queue and send a letter reminder to the patient	▸If the patient is at low/medium risk of non-adherence, return the overdue screening to the master list ▸Or if the patient is at high risk, transition it to the navigator queue	▸Return the overdue screening to the master list

In our model, we defined two multiserver queues (figure 1): the delegate queue staffed by 49 delegates, and the navigator queue staffed by three navigators, as these were the staffing resources we had available at our institution. Both delegates and navigators were characterized by limited available work time, random interval times between screening entries onto their lists, and random processing times (ie, the time it takes a delegate or navigator to reach out to patients to schedule cancer screenings). Cancer screenings could exit or not enter a queue depending on their specific ‘dues’ or risk of non-adherence score. For example, a cancer screening would automatically exit the delegate or navigator queue after a ‘due’ (maximum time) of 4 months or 6 months, respectively. Once in a queue, cancer screenings are processed on a first-come, first-served basis. The total time spent in a queue, which equals the waiting time plus the processing time, cannot exceed a pre-assigned due. For each queue, the processing time is a random variable following an exponential distribution and is identically distributed among delegates or navigators.

During simulation, one of the three possible events—screening entrance, screening completion, or due—can take place in one of three locations. The nine (=3×3) possible events are summarized in table 1 along with the corresponding control statements.

Data collection and estimation

To understand better the delegate and navigators’ operational process, we applied grounded theory,²² a qualitative research method. We started by collecting processing times from two delegates and one navigator. We then analyzed the data and derived a working hypothesis. In this case, these interviews led us to hypothesize that the processing time follows an exponential distribution.

Time-to-screen data were collected directly from the TopCare system. We modeled the data over a 1-year period beginning from TopCare's inception (June 2011).²³ Patients were eligible if they were overdue for at least one cancer screening at inception or became overdue during the 1-year follow-up period. There were 38 890 eligible patients and three cancer types—colorectal, cervical, and breast. We collected patient gender and the numbers of cancer screenings overdue at inception and after 1 year (32 279 women screened for all three cancers, 6611 men screened for only colorectal cancer).

As flow units were cancer screenings as opposed to patients, we identified a population of cancer screenings and stratified it according to the three cancer types. We have a total of 103 448 eligible cancer screenings—38 890 colorectal, 32 279 (=38 890−6611) cervical, and 32 279 breast cancer screenings—among 38 890 patients.

For each of the three cancer types, we randomly sampled the dates of two consecutive screenings from the corresponding population to estimate mean time-to-screen in the master list. For colorectal, cervical, and breast cancers, mean times-to-screen were estimated to be 103.5, 41.2, and 30.9 months and sample sizes were 780, 2525, and 3784, respectively.

For patients with overdue cancer screenings routed to the delegate (or navigator) queue during the 1-year study period, we estimated total times in the delegate (or navigator) queue by randomly sampling dates of entry to and exit from the delegate (or navigator) queue. Mean total times in the delegate and navigator queues were estimated to be 3.26 and 4.19 months, and sample sizes were 10 145 and 119, respectively. As total time in a queue consists of waiting and processing time, we calibrated the mean processing time by repeated simulations to match the mean total time. Based on 235 working days per year, simulations yielded mean processing times in the delegate and navigator queues of 0.029 months (0.45 days) and 0.09 months (1.76 days) per cancer screening, respectively.

Simulation model implementation

The queueing network model was simulated in the C programming language following a typical next-event, time-advance mechanism.^{2 24} The key performance measure in this study was the average number of overdue screenings per patient, which is calculated by dividing the sum of overdue screenings for all patients by the number of patients.

Using the ‘baseline’ setting of current TopCare parameters, we ran a pilot simulation to examine the trajectory of the performance measure (figure 2A), which stabilized after approximately 45 years. The major contributing factors to this long period before reaching steady state are: the large queueing network: a total of 104 447 screenings to be processed by 49 delegates, and three navigators, and long times-to-screen and dues (maximum times) in years. This long period of time to reach steady state for a complex queueing network such as TopCare is not surprising, as simple ‘stand-alone’ queues can also take numerous iterations.²

Figure 2.

Trajectories of average number of overdue screenings per patient (A) and SE of CI (B).

To determine the CI of performance measure,^{2 25} we used six different streams of random numbers to simulate six ‘replications’ including the pilot run, all of which were initialized with the baseline setting. Based on these replications, the simulation time was set to be 48 years. The steady-state performance measure of the pilot run after 48 years was only 0.23 (compared to 1.17 at inception and 0.86 after 1-year study period) and the 95% CI from six replications was 0.234 to 0.228. All simulation runs were executed on a PC with 1.9 GHz and the pilot run took 1537 s of CPU time.

Our primary goal for the simulation was to identify TopCare parameters that reduced the performance measure at steady state. Because the time to reach steady state is long, we assessed whether we could use simulated values of the performance measure before reaching steady state. The trajectory showed a decreasing trend in general before steady state, but with significant variability. Generally, simulated values forecast before steady state need to be interpreted cautiously. The ‘robustness’ of our simulation model was measured by SE of CI (figure 2B), defined as the SD divided by the square root of sample size.⁶ SE were calculated from six simulated values from the six replications collected at the same points of time. Their values were small throughout the trajectory, and became larger between the 10th and 15th years. Therefore, our simulation model was robust as even the pre-steady-state variability among the six replications was small.

Model validation

We validated our approach using the formal validation process described in Barlas.²⁶ For structural confirmation, we compared our model structure directly with information obtained from TopCare during the 1-year study period.

Parameter confirmation was done on times-to-screen in the master list and processing times in the two queues. Statistical goodness-of-fit tests confirmed that no theoretical distributions adequately described the times-to-screen in the master list, so empirical distributions of relative frequencies were used to generate values of the three random variables in simulation. The mean processing times estimated by repeated simulations were also reviewed and corroborated by TopCare delegates and navigators.

Structure-oriented behavior and behavior pattern confirmations were limited because data were observed only for the 1-year study period. From the six replications, the 95% CI of performance measure after 1 year was 0.862 to 0.847 containing the observed value of 0.86 lied. Retesting with more data over longer follow-up will help ensure that TopCare performance measures match the simulated ones.

Approval for the study was obtained from the institutional review board of Massachusetts General Hospital.

Results

TopCare system characteristics

The change over time in the distribution of overdue screenings from inception to steady state is shown in table 2. TopCare reduced the performance measure from 1.17 at inception to 0.86 during simulation to 0.23 at steady state, supporting its effectiveness.

Table 2.

Distribution of overdue screenings per patient at time points before and after simulation

No of overdue screenings	Fraction at inception (from observed data)	Fraction after 1 year of study period (from observed data)	Fraction at steady state (from simulation)
0	0.000	0.281	0.790
1	0.852	0.602	0.193
2	0.121	0.096	0.016
3	0.026	0.021	0.001
Overdue screenings per patient (average)	1.174	0.858	0.229

Simulation analysis of changing staffing levels for patient outreach activities

TopCare uses practice delegates and network-level patient navigators for patient outreach activities. The impact of changing the number of available delegates (baseline of 49) in TopCare on the performance measure is shown in figure 3A. Adding more delegates to TopCare led to decreases in overdue screenings. Furthermore, the marginal benefit of increasing delegate staff by one member was much larger than adding a single navigator, with changes in the number of navigators having little overall impact on the performance measure (figure 3B).

Figure 3.

Box plots illustrating the relationship between average number of overdue screenings per patient and staff levels of TopCare delegates (A) or TopCare navigators (B) (with the upper and lower bounds of 95% CI attached).

However, increasing navigator staff levels had a large impact on those patients at high risk of non-adherence being screened at the navigator queue. Compared to the current number of TopCare navigators,³ increasing staff levels by one or two navigators improved screening completion rates by 11.0% or 19.8% on average, respectively (figure 4). Furthermore, decreases in navigators by one or two members led to decreasing cancer screening productivity by 12.6% or 28.3% on average, respectively. The ‘fraction of screenings completed’ was calculated by dividing the number of screenings completed in the navigator queue per year by the number of screenings entered in the navigator queue per year.

Figure 4.

Box plots showing the impact of changing navigator staff levels on fraction of high-risk patients’ screenings completed per year (with the upper and lower bounds of 95% CI attached).

Impact of modifications to TopCare user workflow

We investigated the impact of changing default settings in the TopCare system by varying the maximum time (or due) a patient could remain in a queue. The baseline settings for the delegate and navigator queues were 4 and 6 months, respectively. Table 3 summarizes the impact of changes in the delegate or navigator queues on the performance measure.

Table 3.

Impact of changes in the maximum time on delegate and navigator queues on average overdue screenings per patient

Maximum time on queues (months)	Decrease by 2 months	Decrease by 1 month	Baseline (no changes)	Increase by 1 month	Increase by 2 months
Delegate queue	2	3	4	5	6
Navigator queue	4	5	6	7	8
Overdue screenings per patient (average)	0.207	0.221	0.229	0.244	0.239

Changing the maximum number of months in the delegate and navigator queues led to relatively small changes in the current average overdue screenings per patient of 0.23, ranging from a minimum of 0.21 for decreases of 2 months, to a maximum of 0.24 for increases of 2 months, respectively. No monotonicity (ie, continuously increasing or decreasing values) could be identified in the average overdue values with changes in these delegate and navigator queue settings.

Predicting TopCare response to changes in cancer screening recommendations

We used simulation analysis to evaluate how TopCare would respond to changes in the frequency of breast cancer screenings. At baseline, we used a 2-year screening interval and varied it from 1 to 3 years using 6-month increments (table 4). Results found in table 4 were calculated in the following way: the ‘percent screenings completed in the master list per year’ is equal to the number of screenings completed in the master list per year divided by the number of screenings entered in the master list per year; the ‘percent screenings entered in the delegate queue per year’ is equal to the number of screenings entered in the delegate queue during 48 years of simulation time divided by 48 years of simulation time; and the ‘percent screenings completed in the delegate queue per year’ is equal to the number of screenings completed in the delegate queue per year divided by the number of screenings entered in the delegate queue per year.

Table 4.

Mammogram screening frequency effect on TopCare parameters

	Frequency of mammogram screening recommendation
TopCare outcomes	Every 1 year (decrease 1 year)	Every 1.5 years (decrease 6 months)	Every 2 years (baseline)	Every 2.5 years (increase 6 months)	Every 3 years (increase 1 year)
Screenings completed in master list per year, %	11.5	25.6	36.1	67.7	76.4
Screenings entered in delegate queue per year, average	31 485	24 509	20 692	15 827	14 191
Screenings completed in delegate queue per year, %	62.0	79.9	93.9	99.1	99.5
Overdue screenings per patient (average)	1.276	0.677	0.229	0.006	0.001

The performance measure decreased considerably from 1.276 at 1 year to 0.001 at the 3-year mammogram screening frequency. In addition, the fraction of screenings completed in the master list and delegate queue increased with longer mammogram screening frequencies. Finally, simulated increases in the screening interval showed that fewer patients with overdue screenings would arrive at the delegate queue per year.

Discussion

Using queueing theory and simulation modeling, we sought to optimize workforce, IT, and clinical factors within a population management system that performs cancer screening in a primary care network. We found that increasing the number of TopCare delegates decreased the performance measure while more TopCare navigators increased screenings among those at increased risk of non-adherence. In contrast, changes in the amount of time a patient entry stays on delegate and navigator lists had little impact on overdue screenings. Finally, lengthening the screening interval increased efficiency within TopCare by decreasing overdue screenings at the patient level, resulting in a smaller number of overdue patients needing delegates for screening and a higher fraction of screenings completed by delegates. These results may help to optimize population management systems by identifying workflow inefficiencies^{27 28} and evaluating tradeoffs when using limited resources to deliver scalable, effective preventive screening within large practice networks.

A key goal of population management systems is to target interventions based on the degree of assistance patients need in completing the activity of interest. For early responders, TopCare used reminder letters with simple contact instructions and educational materials that permitted patients to self-initiate scheduling of overdue screenings. For patients remaining overdue, more intense outreach included calls by practice delegates to patients, and for those at increased risk of non-adherence based on a provider referral or a system algorithm, referral to patient navigators.

Understanding factors that predict patient reluctance to perform recommended services is critical for the design of population management systems, and range from concerns about the purpose and nature of the screening or therapy to financial, educational, or language barriers.^29–32 Case managers aided by screening reminder technologies can address many of these factors and increase cancer screening.^{33 34} In TopCare, navigators focus on those patients who often experience disparities not only in screening completion rates but in overall morbidity and mortality.^{31 32 35} The larger navigator queue time is expected given the focus on non-adherent persuadable patients who may need additional services, such as personal escort to a colonoscopy appointment, to complete screening. Furthermore, service times in the navigator queue are likely to be larger than in the delegate queue because navigators often need repeated outreach attempts to contact and engage patients successfully. Our results highlight how modifying workforce inputs such as the combination of delegates and navigators can lead to earlier screening and potentially diagnosing cancer at an earlier stage when treatment and follow-up may be shorter, more effective, and less costly.^{32 36 37}

In our analysis, the performance measure of delegates and navigators changed very little when we modified the maximum time allowed in navigator or delegate queues. Our analysis also showed that simulations could provide an effective way to test for suboptimal settings in order to focus on parameters that could improve preventive screening behavior.^{38 39} Applying this approach to IT population management systems for other prevention or chronic conditions could improve understanding of important relationships between system parameters and clinical workflow to permit more integrated care.^{6 40–42} In the near future, we also plan to conduct a complete cost-effectiveness analysis of this TopCare system.

Finally, we simulated the effect of changing clinical screening recommendations and showed that increasing the mammogram screening interval increased completion rates at the home and delegate queues, and led to a noticeably smaller value of the performance measure. However, increased screening intervals led to fewer screenings entered in the delegate queue, suggesting a lighter system workload and less efficient use of delegates. Our findings suggest TopCare's current parameters were close to optimal, as they resulted in a small value of the performance measure and allowed a sufficient number of screenings to arrive at the queues with a solid completion rate.

With the growing volume of guidelines for different cancer screening modalities, ages, and screening frequencies, clinicians will be expected to make rapid and informed decisions about multiple (often conflicting) recommendations.^{6 40–46} Simulation analysis can accommodate and anticipate the impact of screening policy changes, thus providing clinical informatics leadership with a clear picture of the consequences of important decisions on preventive care at the institutional level given existing local resources.

Limitations

Our study had several limitations. First, our results are from a single, academic, primary care practice-based network with a well developed IT infrastructure and may not be generalizable to other settings. However, our systematic approach to applying operations research techniques to TopCare can probably be used to optimize the performance of similar population management systems. Second, the use of queueing theory required several assumptions about the distribution of multiple system parameters, which may not fully reflect the complexities of the TopCare system. We tried to address these limitations by applying simulation techniques that could better capture the core processes of TopCare's workflow. Third, the development of sophisticated simulation models is a complex and time-consuming process that requires a high level of expertise. Fourth, we did not perform formal sensitivity analyses on times-to-screen, which could limit the generalizability of our results. Finally, we have not validated the simulation results in steady state by examining the effects of implementing the results into the TopCare system. Future research should use ongoing data collection to address this question.

Conclusion

To optimize a new visit-independent, population-based cancer screening system (TopCare), we used queueing theory with simulation analysis to examine how to optimize workforce, IT system, and clinical screening policies. We found that applying operations research methodologies to a population management system may help healthcare decision-makers better understand the impact of changes in the use of limited resources on the effectiveness and efficiency of such systems.