Abstract
BACKGROUND
High value screening tests such as colonoscopy and mammography can improve early cancer detection but are often underutilized.
METHODS
We evaluated an active choice intervention using the electronic health record (EHR) to confirm patient eligibility for colonoscopy or mammography during the patient’s clinic visit and prompt the physician and his/her medical assistant to actively choose to “accept” or “cancel” an order for it. We fit multivariate logistic regression models using a difference-in-differences approach to evaluate changes in physician ordering and patient completion of colonoscopy and mammography at the intervention practice compared to two control practices, adjusting for time trends, patient and clinic visit characteristics.
RESULTS
The sample comprised 7,560 patients due for colonoscopy and 8,337 patients due for mammography. Pre-intervention trends between practices did not differ. In the adjusted models, compared to the control group over time, the intervention practice had a significant increase in ordering of colonoscopy (11.8 percentage points, 95% CI: 8.0 – 15.6, P < 0.001) and mammography (12.4 percentage points, 95% CI: 8.7 – 16.2, P < 0.001). There was a significant increase in patient completion of colonoscopy (3.5 percentage points, 95% CI: 1.1 – 5.9, P < 0.01), but no change in mammography (2.2 percentage points, 95% CI: −1.0 – 5.5, P = 0.18).
CONCLUSIONS
Active choice through the EHR was associated with an increase in physician ordering of colonoscopy and mammography. The intervention was also associated with an increase in patient completion of colonoscopy but no change in patient completion of mammography.
Keywords: active choice, choice architecture, physician behavior, behavioral economics, electronic health record, cancer screening, colonoscopy, mammography
1. INTRODUCTION
Cancer is a leading cause of mortality in the United States, accounting for about one in four deaths each year.1 High value screening tests can improve early cancer detection.1–2 However, these tests are often underutilized,1–6 and there is a significant need for new approaches to address this issue.
Well-designed clinical decision support within the EHR has been demonstrated to improve clinician performance across many process measures, but there has been less evidence evaluating their impact on patient outcomes.7–10 Recently, there has been growing interest in using insights from the behavioral sciences to design choices within the EHR to impact patient care.11–13 For example, our prior work demonstrated that changing prescription order entry defaults could be used to increase generic medication utilization.14
Active choice is a method that has been demonstrated to change behavior by providing an opportunity for a decision between options to be made before one can proceed to the next step in the process.15 In these contexts, the decision-maker is prompted at the appropriate time (e.g. when the patient is there for a clinic visit) using an ‘interrupted alert’, information can be provided to highlight the desirable features of the option preferred by the choice architect (e.g. your patient is due for this high value screening test), and the choice can be made mandatory (‘forced choice’) so the respondent has to make a decision before proceeding with the visit or to the next stage of the decision-making process. Active choice has been shown to increase patients’ renewal of prescription medications and intent to obtain an influenza vaccination.15 However, the application of active choice within healthcare is limited and the impact on both physician and patient behavior has not been well evaluated. The objective of this study was to evaluate the association of an active choice intervention with changes in physician ordering and patient completion of colonoscopy and mammography, two high-value cancer screening tests.
2. METHODS
This study was approved by the University of Pennsylvania institutional review board. Informed consent was waived because it was not possible given the retrospective study design and the study posed minimal risk to patients.
2.1. Study Sample
The sample comprised patients with a clinic visit at one of three internal medicine practices at the University of Pennsylvania Health System between February 15, 2011 and February 14, 2013 (one year before and after the intervention start date). All three sites were academic teaching practices with faculty and residents located within proximity (0.3 miles apart) in Philadelphia, Pennsylvania.
The sample of patients eligible for colon cancer screening were age 50 to 74 years. To ensure we evaluated a sample of patients that were due for a screening colonoscopy, we excluded patients with any of the following: 1) colonoscopy procedure completed within 10 years of the clinic visit based on health system insurance claims; 2) electronic medical record noted the patient was up to date on colon cancer screening (using health maintenance data from EPIC, the outpatient electronic medical record); 3) fecal occult blood test (FOBT) or fecal immunochemical test (FIT) completed within one year of clinic date; 4) history of colon or rectal cancer, inflammatory bowel disease, any type of colitis, or gastrointestinal bleeding (Supplement).
The sample of patients eligible for breast cancer screening by mammography were females age 50 to 69 years. To ensure we evaluated a sample of patients that were due for a screening mammography, we excluded patents with any of the following: 1) mammography completed within one year of the clinic visit based on health system insurance claims (at the time of the intervention national guidelines recommended annual mammography screening); 2) electronic medical record noted the patient was up to date on breast cancer screening (using health maintenance data from EPIC); 3) history of breast cancer, breast mass, or breast surgery (Supplement).
2.2. Intervention
Prior to the intervention, providers at all three clinics had to manually check if a patient was due for a colonoscopy or mammography and then place an order for the test. On February 15, 2012 one of the clinics implemented a change to the electronic health record settings by using a best practice alert in EPIC. This intervention confirmed patient eligibility for the test during the clinic visit and upon signing into the electronic health record for that patient prompted the provider to actively choose to “accept” or “cancel” an order for a colonoscopy, mammography, or both. This alert was delivered to physicians (who could place and sign orders) and their medical assistants (who could place orders for the physician to sign).
2.3. Main Outcome Measures
The primary outcome measures were the percentage of patients eligible for colon cancer screening that had a colonoscopy ordered by the physician and the percentage of patients eligible for breast cancer screening that had a mammography ordered by the physician. The secondary outcome measures were the percentage of patients eligible that completed colonoscopy and mammography. To identify a reasonable period of time that a completed test could be attributed to the visit, we used a prior sample of patients who completed each test and estimated the time within one year one after the visit for which about 80% of tests were completed. Based on this data, we classified colonoscopy completion as within six months of the visit and mammography completion as within three months of the visit.
2.4. Data
Clarity, an EPIC reporting database, was used to obtain data on patient demographics and comorbidities, clinic visits including type of visits and status of provider as primary care physician or not, and test orders for colonoscopy and mammography. Health insurance claims were obtained from the billing system at University of Pennsylvania Health System. Data on Medicare or Medicaid insurance were missing for some patients during the pre-intervention year because the method by which the health system captured this data changed. These patients were coded as having other insurance.
2.5. Statistical Analysis
Unadjusted analyses were performed to evaluate test order and completion rates over time. Given the monthly fluctuation due to sample size, these data are presented at the quarterly level.
We used multiple time series research design,16–17 also known as difference-in-differences, to compare within-practice pre- and post-intervention outcomes between the intervention practice and the two control practices. While some opportunity for residual confounding remains, this approach reduces potential biases from unmeasured variables from three possible sources.17–19 First, a difference between groups that is stable over time cannot be mistaken for an effect of the intervention because practice site fixed effects are used to compare each practice with itself before and after the intervention. Second, changes affecting both groups similarly over time, such as technological improvements or pay-for-performance initiatives, cannot be mistaken for an effect because the regression models use monthly time fixed effects. Third, if the patient mix is changing differently among practices, and if these changes are accurately reflected in the measured risk factors, this cannot be mistaken for an effect of the intervention because the regression models adjust for these measured risk factors.
Similar to prior work,14,20–21 a multivariate logistic regression model was fit to the binary outcome measures (test ordered or test completed) using the patient as the unit of analysis and adjusting for demographics (age, gender, race/ethnicity), comorbidities (using the Charlson Comorbidity Index which predicts 10-year mortality),22 insurance type, whether the visit was with the primary care provider or not, and visit type (new, return, reassign provider, other). The model compared the post-intervention year (February 15, 2012 to February 14 2013) to the pre-intervention year (Feb 15, 2011 to Feb 14, 2012), adjusting for calendar month (one term for each month of the year) and practice site fixed effects. Standard errors in the models were adjusted to account for clustering by patient.23–24 To assess the mean effect of the intervention in the post-intervention period, we exponentiated the mean of the monthly interaction term log odds ratios for the outcome measure.14,21,25–26 To obtain the adjusted difference in percentage of patients with a test ordered or completed along with 95% confidence intervals, we used the bootstrap procedure, resampling patients.27–28 For all measures, a test of controls was conducted to test the null hypothesis of parallel trends between the intervention and control practices using monthly data during the pre-intervention period. Two-sided hypothesis tests used a significance level of 0.05; analyses were conducted using SAS 9.4.
3. RESULTS
The sample comprised 7,560 patients eligible for colonoscopy with 14,546 clinic visits and 8,337 patients eligible for mammography with 14,410 clinic visits. Tables 1 and 2 show patient characteristics by practice for patients eligible for colonoscopy and mammography, respectively.
TABLE 1.
Sample Characteristics for Colonoscopy Screening
| Characteristic | Intervention Practice | Control Practice #1 | Control Practice #2 | |||
|---|---|---|---|---|---|---|
| Pre Year | Post Year | Pre Year | Post Year | Pre Year | Post Year | |
| Patients, n | 1257 | 967 | 1442 | 1311 | 1260 | 1323 |
| Clinic visits, n | 2539 | 1716 | 3016 | 2661 | 2300 | 2314 |
| Age, mean years (SD) | 58.8 (6.9) | 58.8 (7.0) | 59.2 (7.0) | 59.9 (6.9) | 60.0 (7.0) | 60.2 (7.0) |
| Female gender, (%) | 63.1% | 63.4% | 63.3% | 62.5% | 55.2% | 53.1% |
| Race/ethnicity, % | ||||||
| Non-hispanic white | 21.6% | 24.3% | 48.8% | 48.3% | 49.3% | 48.8% |
| Non-hispanic black | 71.8% | 69.3% | 42.2% | 42.1% | 40.6% | 41.3% |
| Hispanic | 1.0% | 1.2% | 1.2% | 1.3% | 1.1% | 1.6% |
| Other | 5.6% | 5.2% | 7.8% | 8.3% | 9.0% | 8.2% |
| Insurance type, % | ||||||
| Private | 50.1% | 47.2% | 53.7% | 58.0% | 48.9% | 45.5% |
| Medicare | 5.6% | 28.6% | 5.4% | 26..9% | 8.9% | 33.0% |
| Medicaid | 13.4% | 21.5% | 8.6% | 12.9% | 9.0% | 19.0% |
| Other/self-insured | 30.8% | 3.1% | 32.3% | 2.1% | 33.2% | 2.5% |
| Charlson Score, median (IQR) | 0 (0,1) | 0 (0,1) | 0 (0,1) | 0 (0,1) | 0 (0,1) | 0 (0,1) |
| Clinic visit type, % | ||||||
| New patient | 16.6% | 21.1% | 16.1% | 22.3% | 42.8% | 54.3% |
| Return patient | 62.8% | 58.1% | 59.2% | 49.8% | 54.5% | 39.3% |
| Reassign patient provider | 8.0% | 6.2% | 3.5% | 4.7% | 1.4% | 3.5% |
| Other visit type | 12.6% | 14.6% | 21.2% | 23.1% | 1.3% | 4.9% |
| Clinic visit PCP status, % | ||||||
| Yes with PCP | 66.6% | 74.9% | 69.3% | 73.8% | 41.4% | 41.4% |
| No | 27.6% | 24.4% | 26.6% | 25.7% | 33.2% | 57.1% |
| PCP not assigned | 5.8% | 0.7% | 4.2% | 0.5% | 25.4% | 1.5% |
Data on Medicare or Medicaid insurance were missing for some patients during the pre-intervention year because the method by which the health system captured this data changed. These patients were coded as having other insurance
TABLE 2.
Sample Characteristics for Mammography Screening
| Characteristic | Intervention Practice | Control Practice #1 | Control Practice #2 | |||
|---|---|---|---|---|---|---|
| Pre Year | Post Year | Pre Year | Post Year | Pre Year | Post Year | |
| Patients, n | 1422 | 1507 | 1467 | 1507 | 1164 | 1270 |
| Clinic visits, n | 2542 | 2425 | 2566 | 2695 | 2010 | 2172 |
| Age, mean years (SD) | 58.5 (5.5) | 58.6 (5.6) | 58.3 (5.5) | 59.0 (5.6) | 58.9 (5.5) | 59.1 (5.6) |
| Female gender, (%) | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Race/ethnicity, % | ||||||
| Non-hispanic white | 16.9% | 17.3% | 46.0% | 40.3% | 41.2% | 39.8% |
| Non-hispanic black | 77.7% | 77.3% | 47.2% | 52.2% | 50.7% | 52.3% |
| Hispanic | 0.8% | 0.9% | 1.2% | 0.7% | 1.2% | 1.3% |
| Other | 4.6% | 4.4% | 5.7% | 6.8% | 7.0% | 6.6% |
| Insurance type, % | ||||||
| Private | 52.1% | 48.1% | 53.4% | 61.2% | 55.5% | 54.2% |
| Medicare | 7.9% | 28.3% | 5.8% | 23.5% | 7.8% | 26.5% |
| Medicaid | 13.1% | 20.8% | 8.3% | 12.9% | 8.4% | 16.2% |
| Other/self-insured | 26.9% | 2.8% | 32.4% | 2.4% | 28.3% | 3.1% |
| Charlson Score, median (IQR) | 0 (0,1) | 0 (0,1) | 0 (0,1) | 0 (0,1) | 0 (0,1) | 0 (0,1) |
| Clinic visit type, % | ||||||
| New patient | 10.3% | 7.2% | 10.2% | 12.3% | 27.0% | 31.9% |
| Return patient | 69.0% | 70.7% | 63.3% | 58.7% | 69.4% | 59.3% |
| Reassign patient provider | 7.7% | 5.9% | 3.4% | 4.1% | 1.8% | 2.4% |
| Other visit type | 12.9% | 16.5% | 23.2% | 24.8% | 1.8% | 6.4% |
| Clinic visit PCP status, % | ||||||
| Yes with PCP | 68.4% | 72.6% | 66.5% | 71.7% | 51.4% | 53.9% |
| No | 27.8% | 27.1% | 30.9% | 27.7% | 32.3% | 45.1% |
| PCP not assigned | 3.8% | 0.3% | 2.5% | 0.6% | 16.3% | 1.0% |
Data on Medicare or Medicaid insurance were missing for some patients during the pre-intervention year because the method by which the health system captured this data changed. These patients were coded as having other insurance
For the sample of patients eligible for colonoscopy, unadjusted quarterly order rates by practice are displayed in Table 3. While the intervention practice had higher rates at baseline, a test of controls for the pre-intervention period could not reject the null hypothesis of parallel trends between the intervention and control groups over time for colonoscopy order rates (Odds ratio [OR]: 1.01, 95% confidence interval [CI]: 0.63 – 1.64, P value [P] = 0.96) and completion rates (OR: 0.94, 95% CI: 0.50 – 1.77, P = 0.85). Order rates for colonoscopy at the intervention practice increased from quarter 4 (pre-intervention) to quarter 5 (post-intervention) (Table 3). In quarters 7 and 8 these rates declined as the sample of patients still due for colonoscopy fell from 435 to 292. Order rates at the control practices remained mostly steady throughout the pre- and post-intervention periods.
TABLE 3.
Unadjusted Quarterly Order Rates by Practice
| Pre-Intervention Year | Post-Intervention Year | ||||||||
| Quarter | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
| Colonscopy Order Rates, %, (ordered/eligible) | Intervention Practice | 29.0 (166/573) | 28.4 (146/514) | 25.3 (118/466) | 24.6 (103/419) | 49.9 (217/435) | 38.4 (131/341) | 16.4 (48/292) | 24.2 (78/322) |
| Control Practice #1 | 15.8 (95/600) | 16.2 (91/562) | 16.8 (99/589) | 17.8 (100/562) | 17.3 (100/579 ) | 14.1 (74/524) | 14.7 (71/482) | 12.7 (64/505) | |
| Control Practice #2 | 14.0 (66/470) | 12.0 (51/425) | 13.8 (63/457) | 14.4 (66/459) | 12.2 (58/475) | 11.3 (55/485) | 10.1 (43/425) | 10.3 (49/476) | |
| Pre-Intervention Year | Post-Intervention Year | ||||||||
| Quarter | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | |
| Mammography Order Rates, %, (ordered/eligible) | Intervention Practice | 42.9 (225/524) | 36.2 (174/481) | 31.0 (152/491) | 32.9 (167/507) | 53.3 (333/625) | 45.0 (234/520) | 44.9 (200/445) | 46.9 (198/422) |
| Control Practice #1 | 29.3 (159/542) | 29.3 (149/508) | 30.8 (161/523) | 29.4 (145/493) | 29.0 (172/593) | 27.6 (155/562) | 29.1 (144/494) | 29.6 (143/483) | |
| Control Practice #2 | 27.5 (122/444) | 25.9 (103/397) | 18.8 (75/399) | 19.8 (78/394) | 24.7 (118/478) | 23.7 (112/472) | 22.8 (90/394) | 22.6 (92/407) | |
Patients may present several times within one year, therefore quarterly rates may be different than annual rates in other tables
In the adjusted difference-in-difference model, compared to the control group over time, the intervention practice had a significant increase in patients that had a colonoscopy ordered (11.8 percentage points, 95% CI: 8.0 – 15.6, P < 0.001) and completed (3.5 percentage points, 95% CI: 1.1 – 5.9, P = 0.004) (Table 4).
TABLE 4.
Change in Colonoscopy Screening Rates at the Intervention and Control Practices
| Site | Order Rate (%) | Completion Rate (%) | ||||
|---|---|---|---|---|---|---|
| Pre Year | Post Year | Difference | Pre Year | Post Year | Difference | |
| Intervention Practice | 42.4 | 49 | 6.6 | 16.5 | 17.1 | 0.6 |
| Control Practices | 23.4 | 19.5 | −3.8 | 10.7 | 8 | −2.7 |
| Unadjusted Difference-in-Difference | 10.4 | 3.3 | ||||
| Adjusted Model Estimates | ||||||
| Difference-in-Difference | 11.8 | 3.5 | ||||
| 95% Confidence Interval | 8.0 – 15.6 | 1.1 – 5.9 | ||||
| P Value | <0.001 | 0.004 | ||||
For the sample of patients eligible for mammography, unadjusted quarterly order rates by practice are displayed in Table 3. While the intervention practice had higher rates at baseline, a test of controls for the pre-intervention period could not reject the null hypothesis of parallel trends between the intervention and control groups for mammography order rates (Odds ratio [OR]: 0.68, 95% confidence interval [CI]: 0.46 – 1.01, P value [P] = 0.06) and completion rates (OR: 0.75, 95% CI: 0.49 – 1.16, P = 0.20). Order rates for mammography increased at the intervention practices from quarter 4 (pre-intervention) to quarter 5 (post-intervention) and remained above 45% for subsequent quarters (Table 3). Order rates at the control practices remained mostly steady throughout the pre- and post-intervention periods.
In the adjusted difference-in-difference model, compared to the control group over time, the intervention practice had a significant increase in patients that had mammography ordered (12.4 percentage points, 95% CI: 8.7 – 16.2, P < 0.001), but no significant change in the percentage of patients with mammography completed (2.2 percentage points, 95% CI: −1.0 – 5.5, P = 0.18) (Table 5).
TABLE 5.
Change in Mammography Screening Rates at the Intervention and Control Practices
| Site | Order Rate (%) | Completion Rate (%) | ||||
|---|---|---|---|---|---|---|
| Pre Year | Post Year | Difference | Pre Year | Post Year | Difference | |
| Intervention Practice | 50.5 | 64 | 13.5 | 38.5 | 41.2 | 2.7 |
| Control Practices | 37.7 | 36.9 | −0.8 | 26.6 | 29.5 | 2.9 |
| Unadjusted Difference-in-Difference | 14.3 | −0.2 | ||||
| Adjusted Model Estimates | ||||||
| Difference-in-Difference | 12.4 | 2.2 | ||||
| 95% Confidence Interval | 8.7 – 16.2 | −1.0 – 5.5 | ||||
| P Value | <0.001 | 0.18 | ||||
4. DISCUSSION
Our findings demonstrated that the active choice intervention in this study was associated with a significant increase in physician ordering of colonoscopy and mammography when compared to a control group over time. The intervention was also associated with a significant increase in patient completion of colonoscopy, but no change in rates of mammography completion. These findings indicate that choice architecture may play an important role in influencing medical decision-making towards increased ordering of high value care services. However, it may still be necessary to also focus on ways to better engage patient to complete screening tests after they are ordered.
Our findings expand the understanding of how these interventions may impact physician and patient behavior in several ways. First, colonoscopy and mammography order rates increased on a relative basis by 25–28% compared the control group over time. While active choice has not been well evaluated within healthcare,15 this intervention used a combination of an interrupted alert and forced choice, elements of clinical decision support shown to be effective in prior work.29–30 This was a low-cost intervention that could be easily scalable to other settings.
Second, prior evidence has shown that increased alerts to physicians can result in alert fatigue causing the intervention effectiveness to decrease over time.31–34 We did find that colonoscopy order rates declined beginning in the 2nd quarter after the intervention. While it is possible alert fatigue contributed to these trends, we believe that another important contributing factor may have been that increased patient completion of colonoscopy left a remaining sample patients who were harder to nudge. Trends in mammography order rates may support this mechanism as patient completion of mammography did not change significantly and order rates remained steady in the post-intervention period. In other contexts, such as with drug-drug interaction alerts, expert panels have convened to review the evidence and determine which process and outcomes may be better suited for an interrupted vs. non-interruptive alert.34 Our findings help to demonstrate the effectiveness for increasing ordering of cancer screening tests using an interruptive alert.
Third, prior reviews of studies evaluating clinical decision support mechanisms have consistently found improved provider performance but less evidence of their impact on patient outcomes.7–10 Similarly, in our study, the impact on patient completion was inconsistent. Several studies suggest that EHR interventions may have a greater impact on patient outcomes if they are combined with other methods to target patients. A study by Green and colleagues randomized patients at 21 primary care centers to usual care or automated reminders for colonoscopy screening using either an EHR-linked mailing, telephone call, or both plus a nurse navigator.35 They found that all interventions increased colonoscopy screening with the EHR-linked mailings doubling the completion rate by two years. A study by Sequist and colleagues randomized 11 ambulatory health centers to usual care or reminders to the patient and physician.36 They also found that mailings to patients significantly increased colonoscopy screening rates. These studies suggest that an active choice intervention targeted to physicians, such as the one in our study, could be more effective at increasing screening completion if patients were also sent reminders.
This study is subject to several limitations. First, any observational study is susceptible to unmeasured confounders. However, by comparing outcomes over time in the intervention practice compared to the control practice, potential bias from unmeasured confounding is reduced. Second, our findings are from patients at a small number of practices in the same location which may limit generalizability to other settings. Third, since the intervention was delivered to both physicians and their medical assistants we were not able to assess relative differences in effects between the different providers. Fourth, we present data for the first year after the intervention and longer-term studies are needed to evaluate sustainability of the intervention over time. In addition, we used six and three month cutoffs for colonoscopy and mammography completion, respectively. While these method helps to identify completion that is more likely related to the visit, longer follow-up periods may have captured more test completion. Fifth, since the intervention was effective at increasing colonoscopy completion rates, the remaining sample of unscreened patients may have been harder to impact over time and this conservatively biases our findings for this sample toward the null. Sixth, we did not evaluate alternate methods for colon cancer screening such as FOBT or FIT and it is possible that some patients may have received these screening tests instead of a colonoscopy. Seventh, for many patients we did not have claims data to rule out colonoscopy within the last 10 years and instead relied upon EHR documentation by the provider.
5. CONCLUSION
Active choice through the EHR was associated with an increase in physician ordering of colonoscopy and mammography. The intervention was also associated with an increase in patient completion of colonoscopy but no change in patient completion of mammography. These findings indicate that changes to choice architecture could help improve approaches to increase the ordering of high value cancer screening tests, but also suggest that other patient focused interventions may need to be combined with EHR-based interventions to increase screening rates.
Supplementary Material
ACKNOWLEDGEMENTS
This study was funded by a grant from the Leonard Davis Institute of Health Economics at the University of Pennsylvania and a grant from the National Institute on Aging, P30AG034546 through the LDI Center for Health Incentives and Behavioral Economics. Dr. Patel had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. Dr. Volpp is a principal at the behavioral economics consulting firm, VAL Health. Dr. Volpp also has received consulting income from CVS Caremark and research funding from Humana, CVS Caremark, Discovery (South Africa), Hawaii Medical Services Association, and Merck, none of which are related to the work described in this manuscript.
REFERENCES
- 1.Siegel RL, Miller KD, Jemal A. Cancer statistics, 2016. CA Cancer J Clin. 2016;66(1):7–30. [DOI] [PubMed] [Google Scholar]
- 2.Wilt TJ, Harris RP, Qaseem A, High Value Care Task Force of the American College of Physicians. Screening for cancer: advice for high-value care from the American College of Physicians. Ann Intern Med. 2015;162(10):718–25. [DOI] [PubMed] [Google Scholar]
- 3.Cokkinides VE, Chao A, Smith RA, Vernon SW, Thun MJ. Correlates of underutilization of colorectal cancer screening among U.S. adults, age 50 years and older. Prev Med. 2003;36(1):85–91. [DOI] [PubMed] [Google Scholar]
- 4.Meester RG, Doubeni CA, Lansdorp-Vogelaar I, Goede SL, Levin TR, Quinn VP, Ballegooijen Mv, Corley DA, Zauber AG. Colorectal cancer deaths attributable to nonuse of screening in the United States. Ann Epidemiol. 2015;25(3):208–213.e1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Joseph DA, King JB, Miller JW, Richardson LC. Prevalence of colorectal cancer screening among adults—Behavioral Risk Factor Surveillance System, United States, 2010. MMWR Morb Mortal Wkly Rep. 2012; 61(Suppl):51–6. [PubMed] [Google Scholar]
- 6.Mammography and breast cancer. Centers for Disease Control and Prevention FastStats. Available online at: http://www.cdc.gov/nchs/fastats/mammography.htm. Accessed April 25, 2016.
- 7.Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280(15):1339–46. [DOI] [PubMed] [Google Scholar]
- 8.Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, Sam J, Haynes RB. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. JAMA. 2005;293(10):1223–38. [DOI] [PubMed] [Google Scholar]
- 9.Souza NM, Sebaldt RJ, Mackay JA, Prorok JC, Weise-Kelly L, Navarro T, Wilczynski NL, Haynes RB, CCDSS Systematic Review Team. Computerized clinical decision support systems for primary preventive care: a decision-maker-researcher partnership systematic review of effects on process of care and patient outcomes. Implement Sci. 2011;687. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Bright TJ, Wong A, Dhurjati R, Bristow E, Bastian L, Coeytaux RR, Samsa G, Hasselblad V, Williams JW, Musty MD, Wing L, Kendrick AS, Sanders GD, Lobach D. Effect of clinical decision-support systems: a systematic review. Ann Intern Med. 2012;157(1):29–43. [DOI] [PubMed] [Google Scholar]
- 11.Patel MS, Volpp KG. Leveraging insights from behavioral economics to increase the value of health-care service provision. J Gen Intern Med. 2012;27:1544–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Halpern SD, Ubel PA, Asch DA. Harnessing the power of default options to improve health care. N Engl J Med. 2007;357(13):1340–4. [DOI] [PubMed] [Google Scholar]
- 13.Meeker D, Linder JA, Fox CR, Friedberg MW, Persell SD, Goldstein NJ, Knight TK, Hay JW, Doctor JN. Effect of Behavioral Interventions on Inappropriate Antibiotic Prescribing Among Primary Care Practices: A Randomized Clinical Trial. JAMA. 2016;315(6):562–70. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Patel MS, Day S, Small DS, Howell JT, Lautenbach GL, Nierman EH, Volpp KG. Using Default Options Within the Electronic Health Record to Increase the Prescribing of Generic-Equivalent Medications: A Quasi-experimental Study. Ann Intern Med. 2014;161:S44–S52. [DOI] [PubMed] [Google Scholar]
- 15.Keller PA, Harlam B, Loewenstein G, Volpp KG. Enhanced active choice: A new method to motivate behavior change. J Consum Psychol. 2011;21:376–383. [Google Scholar]
- 16.Campbell DT, Stanley JC. Experimental and Quasi-Experimental Designs for Research. Dallas, TX: Houghton Mifflin Co; 1963. [Google Scholar]
- 17.Dimick JB, Ryan AM. Methods for Evaluating Changes in Health Care Policy: The Difference-in-Differences Approach. JAMA. 2014;312(22):2401–2402. [DOI] [PubMed] [Google Scholar]
- 18.Rosenbaum PR. Stability in the absence of treatment. J Am Stat Assoc. 2001;96:210–219. [Google Scholar]
- 19.Shadish WR, Cook TD, Campbell DT. Experimental and Quasi-Experimental Designs for Generalized Causal Inference. Boston, MA: Houghton-Mifflin; 2002. [Google Scholar]
- 20.Patel MS, Volpp KG, Small DS, Hill AS, Even-Shoshan O, Rosenbaum L, Ross RN, Bellini L, Zhu J, Silber JH. Association of the 2011 ACGME Resident Duty Hour Reforms With Mortality and Readmissions Among Hospitalized Medicare Patients. JAMA. 2014;312(22):2364–2373. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Patel MS, Patel N, Small DS, Rosin R, Rohrbach JI, Stromberg N, Hanson CW, Asch DA. Change in length of stay and readmissions among hospitalized medical patients after inpatient medicine service adoption of mobile secure text messaging. JGIM. 2016; Published Online March 25, 2016. DOI: 10.1007/s11606-016-3673-7. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373–383. [DOI] [PubMed] [Google Scholar]
- 23.Rogers WH. Regression standard errors in clustered samples. Stata Technical Bulletin Reports. 1993;3:88–94. [Google Scholar]
- 24.Williams RL. A note on robust variance estimation for cluster-correlated data. Biometrics. 2000;56:645–6. [DOI] [PubMed] [Google Scholar]
- 25.Newman SC. Biostatistical methods in epidemiology. New York: Wiley; 2001:136. [Google Scholar]
- 26.Agresti A An introduction to categorical data analysis. New York: Wiley;2007:108–9. [Google Scholar]
- 27.Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York, NY: Chapman & Hall; 1993. [Google Scholar]
- 28.Davison AC, Hinkley DV. Bootstrap Methods and their Application. Cambridge, England: Cambridge University Press; 1997. [Google Scholar]
- 29.Myers JS, Gojraty S, Yang W, Linsky A, Airan-Javia S, Polomano RC. A randomized-controlled trial of computerized alerts to reduce unapproved medication abbreviation use. J Am Med Inform Assoc. 2010;18(1):17–23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Roshanov PS, Fernandes N, Wilczynski JM, Hemens BJ, You JJ, Handler SM, Nieuwlaat R, Souza NM, Beyene J, Van Spall HG, Garg AX, Haynes RB. Features of effective computerised clinical decision support systems: meta-regression of 162 randomised trials. BMJ. 2013;346f657. [DOI] [PubMed] [Google Scholar]
- 31.Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med. 2011;8:e1000387. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32.van der Sijs H, Aarts J, Vulto A, Berg M. Overriding of drug safety alerts in computerized physician order entry. J Am Med Inform Assoc. 2006;13:138–47. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Avery AJ, Savelyich BS, Sheikh A, Cantrill J, Morris CJ, Fernando B, et al. Identifying and establishing consensus on the most important safety features of GP computer systems: e-Delphi study. Inform Prim Care. 2005;13:3–12. [DOI] [PubMed] [Google Scholar]
- 34.Phansalkar S, van der Sijs H, Tucker AD, Desai AA, Bell DS, Teich JM, Middleton B, Bates DW. Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records. J Am Med Inform Assoc. 2013;20(3):489–93. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Green BB, Wang CY, Anderson ML, Chubak J, Meenan RT, Vernon SW, Fuller S. An automated intervention with stepped increases in support to increase uptake of colorectal cancer screening: a randomized trial. Ann Intern Med. 2013;158(5 Pt 1):301–11. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 36.Sequist TD, Zaslavsky AM, Marshall R, Fletcher RH, Ayanian JZ. Patient and physician reminders to promote colorectal cancer screening: a randomized controlled trial. JAMA Intern Med. 2009;169(4):364–71. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.