Unintended Consequences of Health Information Technology: Evidence From Veterans Affairs Colorectal Cancer Oncology Watch Intervention

Abstract

Purpose

We evaluated the Colorectal Cancer (CRC) Oncology Watch intervention, a clinical reminder implemented in Veterans Integrated Service Network 7 (including eight hospitals) to improve CRC screening rates in 2008.

Patients and Methods

Veterans Affairs (VA) administrative data were used to construct four cross-sectional groups of veterans at average risk, age 50 to 64 years; one group was created for each of the following years: 2006, 2007, 2009, and 2010. We applied hospital fixed effects for estimation, using a difference-in-differences model in which the eight hospitals served as the intervention sites, and the other 121 hospitals served as controls, with 2006 to 2007 as the preintervention period and 2009 to 2010 as the postintervention period.

Results

The sample included 4,352,082 veteran-years in the 4 years. The adherence rates were 37.6%, 31.6%, 34.4%, and 33.2% in the intervention sites in 2006, 2007, 2009, and 2010, respectively, and the corresponding rates in the controls were 31.0%, 30.3%, 32.3%, and 30.9%. Regression analysis showed that among those eligible for screening, the intervention was associated with a 2.2–percentage point decrease in likelihood of adherence (P < .001). Additional analyses showed that the intervention was associated with a 5.6–percentage point decrease in likelihood of screening colonoscopy among the adherent, but with increased total colonoscopies (all indicators) of 3.6 per 100 veterans age 50 to 64 years.

Conclusion

The intervention had little impact on CRC screening rates for the studied population. This absence of favorable impact may have been caused by an unintentional shift of limited VA colonoscopy capacity from average-risk screening to higher-risk screening and to CRC surveillance, or by physician fatigue resulting from the large number of clinical reminders implemented in the VA.

INTRODUCTION

A large body of literature suggests that health information technology (HIT), such as automatic clinical reminders using electronic health records, helps achieve greater adherence to recommended clinical guidelines and improve preventive care.^1–4 Positive effects of HIT on quality of care are most manifest when HIT is applied in integrated health care systems.⁵ The Veterans Affairs (VA) health care system, the largest integrated health care provider in the United States, has shown remarkable improvements in performance in the past 15 years.^6–8 Adoption of HIT has been hypothesized as one reason for these improvements, although well-conducted empirical studies evaluating this hypothesis are limited.

Colorectal cancer (CRC) screening adherence has been widely used as a performance indicator for quality of care, because strong evidence suggests that CRC screening is an effective and cost-effective strategy for reducing deaths resulting from CRC.^9,10 A 2007 VA directive mandated CRC screening coverage for average- and higher-risk veterans.¹¹ As a result, Veterans Integrated Service Network 7 in the southeast region of the United States launched the CRC Oncology Watch (OncWatch) intervention during fiscal year 2008.

Unlike many other computerized clinical reminders implemented across the VA, OncWatch, a point-of-care reminder, was specifically designed and implemented in Network 7 only, which includes eight hospitals. (Network 7 is one of the 21 networks across the United States, which may make regional decisions on implementing interventions such as OncWatch.) In addition, OncWatch uses the vast electronic health records of the VA to determine veterans' up-to-date CRC screening adherence status during primary care visits and to track quality of CRC care. The intervention included detailed protocols (eg, patient flow process), prelaunch trainings, and postlaunch meetings for quality assurance. The ongoing OncWatch has two objectives. The first is to increase CRC screening rates for all veterans at average or higher risk for CRC by determining their screening adherence status and making appropriate screening recommendations, according to CRC screening guidelines.^11–13 The second is to provide timely diagnostic and surveillance services using colonoscopy to improve quality of CRC care.

The main objective of this study was to evaluate the effect of OncWatch on the likelihood of CRC screening adherence among average-risk veterans age 50 to 64 years eligible for screening. We focused on those at average risk (ie, age 50 to 75 years without personal or family history of adenoma or CRC) because 75% of CRC cases occur among individuals not regarded at higher risk for CRC, and on those age 50 to 64 years because the OncWatch clinical reminder system may not track complete health services utilization of elderly veterans, a large proportion of whom also use Medicare services, which may not be accurately recorded in VA electronic health records.^14,15 In addition, we examined the associations of OncWatch with the likelihood of screening adherence using colonoscopy among veterans who are adherent and with overall colonoscopy utilization for all indicators.

PATIENTS AND METHODS

Data Sources

As a result of the 2007 VA directive mandating CRC screening coverage for veterans, OncWatch was implemented by Network 7 approximately during fiscal year 2008 (from October 1, 2007, to September 30, 2008).¹¹ Table 1 summarizes the implementation timing. All eight hospitals in Network 7 participated in OncWatch. Although implementation in the last three hospitals in Table 1 occurred slightly before fiscal year 2008, this study considered OncWatch implementation as having occurred within fiscal year 2008 for modeling purposes.

Table 1.

Summary of Implementation Time of OncWatch in the Eight Veterans Affairs Hospitals Within Veterans Integrated Service Network 7

Implementation	Alabama			Georgia			South Carolina
	Birmingham	Tuscaloosa	Montgomery/Tuskegee	Atlanta	Augusta	Dublin	Columbia	Charleston
Date	March 2008	January 2008	July 2007	March 2008	September 2007	January 2008	July 2007	March 2008

Abbreviation: OncWatch, Colorectal Cancer Oncology Watch.

The main data source was 1997 to 2010 VA administrative data, containing information on VA-provided and VA-contracted health services (information on contracted services is available from Fee Basis data on services such as colonoscopy, for which VA has limited capacity). These data are only available after 1997. Primary and secondary International Classification of Diseases (Ninth Revision) Clinical Modification diagnosis procedure codes and Current Procedural Terminology (Fourth Edition) codes were used, as described in the previous work.¹⁵

Study Sample

Using 1997 to 2010 administrative data, we created a study sample of four cross-sectional groups of average-risk veterans, age 50 to 64 years, one group for each of the following years: 2006, 2007, 2009, and 2010. The inclusion and exclusion criteria used for creating each group were based on a previously published algorithm.¹⁵ The following is a brief summary of this algorithm, used to create the 2006 group. Veterans were included if they were age 50 to 64 years and had at least two primary care visits during 2006. Using primary and secondary diagnoses codes available from 2001 to 2006, we further excluded those who: one, had CRC diagnostic and/or surveillance diagnoses (representing possibly higher risk for CRC); two, were diagnosed with liver, pancreatic, esophageal, lung, or colorectal cancer; or three, received hospice care. After constructing the four groups of veterans, we appended them together to create the final study sample.

Key Dependent Variables

The key dependent variable of interest was veteran-level, annual CRC screening adherence status. The methods for constructing this variable were based entirely on the published algorithm using a 10-year retrospective window for searching screening colonoscopies.¹⁵ The following briefly illustrates the algorithm for measuring this variable for the 2006 group.

Using the data from 1997 to 2006, veterans in 2006 were considered adherent if they received fecal occult blood tests in 2006, flexible sigmoidoscopy or barium enema from 2001 to 2006, or colonoscopy from 1997 to 2006. A secondary dependent variable, used in an additional analysis, was the status of screening adherence by colonoscopy. Conditional on being adherent, veterans were assigned to adherence by colonoscopy if they underwent colonoscopy from 1997 to 2006, regardless of the presence of other modality codes during the same period, or to other modalities. (Switching the assignment order had no significant effect on the results.)

Explanatory Variables

The key explanatory variable, labeled as OncWatch, capturing the effects of intervention was the interaction of the intervention sites and the postintervention period. Specifically, this variable was coded as 1 if veterans sought care in any hospital within the intervention sites during either 2009 or 2010, and as 0 otherwise.

Veteran-level covariates included age, sex, use of outpatient clinics, and the number of outpatient clinic visits per year. Race was not analyzed because of a large proportion of missing data.¹⁵ This analysis included three time-varying hospital-level covariates, used as proxies for demand and supply for CRC screening. The first covariate was the number outpatient clinics, each of which is affiliated with one hospital, normalized by the number of all-age veterans (per 1,000 veterans) visiting its hospital and affiliated outpatient clinics; the second measured the volume of all-age outpatient clinic visits per hospital and affiliated outpatient clinics (in 100,000 visits); the third measured the number of unique all-age veterans who sought any heath care per hospital and affiliated outpatients (in 10,000 veterans).

Statistical Analysis

Because OncWatch is a nonexperimental intervention, one key concern in our statistical analysis is biases caused by confounders, particularly any important differences between the intervention and control sites. Because the intervention was implemented at the hospital level, potential confounders in this study may be primary at the hospital-level variables. Hospital-level differences in baseline adherence rates and in structural characteristics (eg, local organizational adaptiveness to HIT innovations) could be important confounders threatening the internal validity of our estimates.^15,16 Additional confounders may also come from variables at the national level (or secular trends), such as VA-wide policy changes that could influence clinical practice across the VA health system. For example, the 2007 VA directive mandating CRC screening coverage for average- and higher-risk veterans may have improved overall CRC screening adherence across the VA.¹¹

This study used a difference-in-differences (DD) model to compare intervention sites with nonequivalent control sites (ie, we allowed, and adjusted for, systematic differences between the intervention and control sites before and after the intervention being evaluated). This DD model assumes that the pre/postintervention difference in the control sites is an appropriate estimate of what the pre/post difference in the intervention sites would have been if OncWatch had not been implemented. Thus, our statistical model compares the difference in the likelihood of adherence to CRC screening in the intervention sites before and after implementation of OncWatch with the difference over the same time period among the control sites. (A detailed description of the DD model can be found in previous work.¹⁷)

A linear probability model (LPM) with hospital and year fixed-effects was used for estimation. (We preferred the LPM to the logit model, because OncWatch effects were captured by an interaction term that may have caused complications in calculating average effects in a logit model.¹⁸) We regressed the adherence status among veterans age 50 to 64 years at average risk for CRC on the variable OncWatch, veteran-level covariates, and time-varying hospital-level characteristics. Hospital fixed effects would capture all assumed cross-hospital, time-invariant differences as potential confounders between the intervention and control sites, and year fixed effects would capture secular time trends in CRC screening adherence. SEs were adjusted via the Huber clustered SEs correction for within-hospital clustering at the hospital level.¹⁹

We performed two additional analyses of colonoscopy utilization. Although OncWatch does not explicitly prioritize use of colonoscopy for higher-risk screening, diagnostic, or surveillance purposes, some physicians within the OncWatch facilities may have shifted colonoscopy services away from screening average-risk veterans because of limited VA colonoscopy capacity. (Note that VA started screening colonoscopy coverage in average- and higher-risk veterans in 2005.) This practice may raise a concern that OncWatch could potentially reduce use of screening colonoscopy among average-risk veterans. To address this concern, we first examined the impact of OncWatch on the likelihood of screening adherence by colonoscopy among average-risk veterans age 50 to 64 years who were adherent by any of the prescribed routes. We then examined the overall impact of OncWatch on colonoscopy utilized for all indicators (including screening for the average risk) among veterans age 50 to 64 years who used any VA services, using an unadjusted DD model. (Ideally, we would have examined the direct impact of OncWatch on colonoscopies used for screening higher-risk veterans, diagnostics, and surveillance. However, accurately determining use of colonoscopy for these purposes was beyond the scope of this study and is methodologically difficult in administrative data analysis.²⁰)

RESULTS

Descriptive Statistics

Table 2 summarizes the mean statistics among the nonelderly veterans at average risk for CRC by pre- and postintervention periods and by the intervention and control hospitals. There were a total of 4,352,082 veterans (some of whom were observed more than once) in the 4 years. The number of veterans increased from 1,056,075 (63,683 in the intervention sites) in 2006 to 1,127,930 (74,795 in the intervention sites) in 2010. There are eight hospitals in the intervention sites and 121 in the control sites. The proportions of veterans adherent to CRC screening were 37.6%, 31.6%, 34.4%, and 33.2% in the intervention sites each year from 2006 to 2010 (2008 was omitted), respectively, and the corresponding proportions in the control sites were 31.0%, 30.3%, 32.3%, and 30.9% (Fig 1). There was a notable dip in the proportion of adherence in the intervention sites from 2006 to 2007 in contrast to the trends in the control sites. Similar to the variation in CRC screening adherence rates, there was variation in veteran- and hospital-level characteristics between the intervention and control sites and over time. Compared with veterans in the control sites, veterans in the intervention sites were relatively younger, slightly more likely to use outpatient clinics for primary care, and more likely to seek care in higher-volume hospitals. The proportions of adherent veterans at the intervention sites who achieved adherence through screening colonoscopy were 13.1%, 16.4%, 24.4%, and 30.8% for years 2006, 2007, 2009, and 2010, respectively, and the proportions for the control sites were 20.9%, 26.2%, 38.6%, and 47.1%, respectively.

Table 2.

Mean Statistics of Average-Risk Veterans Age 50 to 64 Years Eligible for CRC Screening

Variable	Before OncWatch				After OncWatch
	2006		2007		2009		2010
	Intervention	Control	Intervention	Control	Intervention	Control	Intervention	Control
Sample size	63,683	992,392	65,876	998,572	71,349	1,032,270	74,795	1,053,145
Veteran level
CRC screening adherence	0.376	0.310	0.316	0.303	0.344	0.323	0.332	0.309
Male sex	0.928	0.927	0.928	0.925	0.919	0.916	0.921	0.920
Age, years
50-54	0.325	0.284	0.315	0.274	0.317	0.269	0.316	0.264
55-59	0.414	0.426	0.393	0.399	0.299	0.301	0.265	0.262
60-64	0.261	0.290	0.293	0.327	0.384	0.430	0.419	0.474
Use of outpatient clinics	0.564	0.561	0.589	0.562	0.698	0.594	0.628	0.608
No. of outpatient clinic visits per veteran	11.446	11.327	11.789	11.640	12.262	12.317	12.535	12.723
VA hospital level
No. of outpatient clinics per hospital per 1,000 veterans	0.092	0.130	0.089	0.130	0.100	0.133	0.103	0.137
No. of outpatient clinic visits per hospital (in 100,000)	4.440	5.996	4.830	6.046	5.807	6.646	6.439	7.073
No. of unique veterans per hospital (in 10,000)	4.460	6.004	4.649	5.956	5.190	6.183	5.527	6.346
No. of VA hospitals	8	121	8	121	8	121	8	121

Abbreviations: CRC, colorectal cancer; OncWatch, CRC Oncology Watch; VA, Veterans Affairs.

Fig 1.

Trends of proportion of colorectal cancer (CRC) screening adherence before and after the CRC Oncology Watch implementation among average-risk veterans age 50 to 64 years eligible for CRC screening.

Regression Results

Table 3 summarizes the estimates from the linear probability models with hospital and year fixed effects. Because of the large sample size used for estimation, almost all SEs were extremely small. (All P values were statistically significant at the < .001 level.) Thus, CIs were not reported in this study.

Table 3.

Estimates From Linear Probability Models With VA Hospital Fixed Effects

Variable	Eligible for CRC Screening*	Adherence to CRC Screening†
Sample size	4,352,082	1,363,454
Veteran level
OncWatch (interaction of intervention and postintervention period)	−0.022	−0.056
Male sex	0.101	0.014
Age, years
50-54 (reference)	NA	NA
55-59	0.044	0.021
60-64	0.046	0.020
Use of outpatient clinics	0.017	−0.054
No. of outpatient clinic visits per veteran	0.003	0.001
VA hospital level
No. of outpatient clinics per hospital per 1,000 veterans	0.070	0.150
No. of outpatient clinic visits per hospital (in 100,000)	0.014	−0.036
No. of unique veterans per hospital (in 10,000)	−0.006	0.011
2006 (reference)	NA	NA
2007	0.013	−0.057
2009	0.014	0.148
2010	−0.008	0.246
No. of VA hospitals	129	129

NOTE. SEs corrected by Huber (cluster) SE correction. Because of large sample size, SEs were extremely small and thus not reported here. All significant at < .001 level except data in final row of table.

Abbreviations: CRC, colorectal cancer; OncWatch, CRC Oncology Watch; NA, not applicable; VA, Veterans Affairs.

^*Dependent variables: 1, adherence; 0, nonadherence.

^†Dependent variables: 1, adherence to screening colonoscopy; 0, adherence to other screening modalities.

Among nonelderly veterans at average risk for CRC, OncWatch was associated with a 2.2–percentage point decrease in the likelihood of CRC screening adherence (Table 3). This estimated effect represents a 6.44% reduction in the average adherence rate of 0.341 in the intervention sites over the 4 years. Male veterans were 10 percentage points more likely than female veterans to be adherent to CRC screening; veterans age 50 to 54 years were less likely to be adherent than those age 55 to 64 years. Additionally, the analysis suggested that individual veterans who used outpatient clinics for any care were more likely than those who did not use outpatient clinics to be adherent to CRC screening.

Our additional analysis showed that among veterans at average risk for CRC who were adherent to screening, OncWatch was associated with a 5.6–percentage point decrease in the likelihood of screening adherence using colonoscopy (Table 3). This estimated effect represents a 26.11% reduction in the average screening colonoscopy rate of 0.214 among the adherent in the intervention sites over the 4 years. The assessment of the overall impact of OncWatch on colonoscopies used for all indicators showed that the rate of colonoscopies per 100 veterans age 50 to 64 years was 24.97 during the preintervention period and 27.93 during the postintervention period in the intervention sites, and the corresponding rates were 21.41 and 21.08 in the control sites. An unadjusted DD model suggested that OncWatch was associated with an increase of approximately 3.63 colonoscopies for all indicators per 100 veterans age 50 to 64 years.

DISCUSSION

Contrary to expectations, our study found that OncWatch had little impact on CRC screening adherence rates among nonelderly veterans at average risk for CRC. The published literature overwhelmingly suggests positive effects of innovative HIT on quality of care.^1,2 For example, one recent systematic review showed that computerized clinical systems achieved on average 4.2% quality improvement.¹ Some other studies, however, suggest that HIT, particularly clinical reminders, has a negative impact on quality.² One reported that the adoption of computerized physician order entry did not result in decreased medical error rates at the Salt Lake VA Medical Center.²¹

Our study showed the unintended consequences of OncWatch, a sophisticated clinical reminder system designed to increase CRC screening adherence rates. There are two possible explanations. First, clinical reminders like OncWatch do not work. There have been literally hundreds of clinical reminders implemented at the VA over time. These reminders may cause interruptions of a physician's normal workflow, which may adversely affect quality of care. However, this explanation alone may not adequately address the lack of impact of OncWatch on CRC adherence rates. A second explanation is that some physicians in the intervention sites may have unintentionally allocated limited colonoscopy resources away from screening of the average- to higher-risk veterans or those with symptoms. As evident in a recent study, fecal occult blood tests, flexible sigmoidoscopy, and barium enema may be more efficient screening options than colonoscopy if there are budget constraints for using costly colonoscopy for CRC screening.²² This second explanation is partially supported by our additional analyses, demonstrating that OncWatch was associated with decreased screening colonoscopy rates among the adherent but with increased colonoscopy utilization for all indicators. These unintended consequences of OncWatch on CRC screening adherence rates may stem from the challenges of meeting the two competing objectives of the intervention and from limited VA colonoscopy capacity. Indeed, it is possible that OncWatch may have improved the overall allocation efficiency of colonoscopy use within the VA by implicitly creating a balance in its application to average-risk and higher-risk veterans.

Our study has four main limitations. First, although the hospital fixed effects may control for all time-invariant hospital-level confounders, some uncontrolled time-varying hospital-level confounders may still have biased the estimated effects of OncWatch. One specific example is time-varying hospital-level CRC screening capacity (eg, gastroenterologist staffing) during the study period. If the implementation of the OncWatch intervention was in part a solution to anticipated changes in CRC screening capacity at the intervention sites, our estimates may be subject to biases. However, there is no evidence suggesting that OncWatch was implemented with this aim in mind.

Second, this study specified a year fixed-effects approach, assuming that secular trends had similar effects on both intervention and control sites. However, this assumption might not be valid. For example, there was a sizable 0.05–percentage point decline in adherence rates in the intervention sites from 2006 to 2007, although the exact reasons for this decline are unclear. Thus, our year fixed-effects specification might not have adequately controlled for confounders contributing to this abrupt change in secular trends. To test the sensitivity of our results, we varied the number of years included in the analysis (eg, changing from the current 4 years to 6 years, including 2004 and 2005). This sensitivity analysis suggested that the effect of OncWatch on overall CRC adherence rates was modest, although statistically significant, but that the effects of OncWatch on screening colonoscopy remained strong and consistent with our main finding.

Third, spillover effects of OncWatch from the intervention to the control sites may have diluted the estimated impact of OncWatch. During the study period, three nonintervention VA medical centers (ie, North Texas, Detroit, MI, and Long Beach, CA) might have been using OncWatch to improve CRC screening adherence and treatment, although there is little information available at present on the effectiveness of implementation at the three sites.¹¹ To test the sensitivity of our results, we excluded veterans in the three hospitals and found no change in coefficients of OncWatch.

Fourth, this study may have limited generalizability. Although the data included all veterans at average risk for CRC, age 50 to 64 years, who relied on the VA for primary care during the 4 recent years, one must be cautious in extrapolating our findings to veterans age 65 years or older, to non-VA settings, to time periods outside those examined in the study, or to other HIT applications.

Our study fills an important gap in the growing literature on the impact of HIT on the quality of care. Specifically, we demonstrate how the influence of an HIT innovation could be significantly affected by the practice environment in which it is introduced. As the United States moves toward building stronger HIT infrastructures, a better understanding of the impact of HIT on quality and outcomes is needed.

AUTHORS' DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

Although all authors completed the disclosure declaration, the following author(s) indicated a financial or other interest that is relevant to the subject matter under consideration in this article. Certain relationships marked with a “U” are those for which no compensation was received; those relationships marked with a “C” were compensated. For a detailed description of the disclosure categories, or for more information about ASCO's conflict of interest policy, please refer to the Author Disclosure Declaration and the Disclosures of Potential Conflicts of Interest section in Information for Contributors.

Employment or Leadership Position: None Consultant or Advisory Role: Deborah A. Fisher, Epic Genomics (C) Stock Ownership: None Honoraria: None Research Funding: None Expert Testimony: None Other Remuneration: None

AUTHOR CONTRIBUTIONS

Conception and design: John Bian, Charles L. Bennett, Maria Ribeiro, Joseph Lipscomb

Financial support: John Bian

Collection and assembly of data: John Bian

Data analysis and interpretation: John Bian, Charles L. Bennett, Deborah A. Fisher, Joseph Lipscomb

Manuscript writing: All authors

Final approval of manuscript: All authors