Optimizing health IT to improve health system performance: A work in progress

Abstract

Background:

Despite significant investments in health information technology (IT), the technology has not yielded the intended performance effects or transformational change. We describe activities that health systems are pursuing to better leverage health IT to improve performance.

Methods:

We conducted semi-structured telephone interviews with C-suite executives from 24 U.S. health systems in four states during 2017–2019 and analyzed the data using a qualitative thematic approach.

Results:

Health systems reported two broad categories of activities: laying the foundation to improve performance with IT and using IT to improve performance. Within these categories, health systems were engaged in similar activities but varied greatly in their progress. The most substantial effort was devoted to the first category, which enabled rather than directly improved performance, and included consolidating to a single electronic health record (EHR) platform and common data across the health system, standardizing data elements, and standardizing care processes before using the EHR to implement them. Only after accomplishing such foundational activities were health systems able to focus on using the technology to improve performance through activities such as using data and analytics to monitor and provide feedback, improving uptake of evidence-based medicine, addressing variation and overuse, improving system-wide prevention and population health management, and making care more convenient.

Conclusions and implications:

Leveraging IT to improve performance requires significant and sustained effort by health systems, in addition to significant investments in hardware and software. To accelerate change, better mechanisms for creating and disseminating best practices and providing advanced technical assistance are needed.

1. Introduction

The Health Information Technology for Economic and Clinical Health (HITECH) Act of 2009 has been lauded as the key piece of federal legislation that prompted the majority of physicians and hospitals to adopt electronic health records (EHRs). Originally, HITECH outlays were described not as investments in technology per se, but as efforts “to improve the health of Americans and the performance of their health care system” (emphasis added).¹ In the decade since its passage, adoption of EHRs has grown substantially, and some functionalities—such as electronic prescribing and computerized provider order entry (CPOE)—have arguably become part of the standard of care.^2,3

Yet the performance of health systems across the U.S. continues to lag. A recent review shows that there is no consistent definition of health system performance, but cost and quality are used most commonly.⁴ In the past decade, health expenditures have increased from 17.3% to 17.8% of GDP, and measures of quality and outcomes continue to show major deficits.^5–8 Some measures are getting worse. The information technology (IT) revolution, which has catalyzed transformations in many other industries (e.g., transportation, finance, hospitality, commerce) has yet to yield the quality and cost effects in health care that policymakers intended. After more than a decade of investment and increases in adoption, it is worth considering why change is not occurring as rapidly as desired, investigating what kinds of health IT-related changes are occurring today that will help to improve performance, and identifying what can be done to accelerate progress.

We address these questions by investigating how health systems in the U.S. are using IT (in particular, EHRs) to improve performance, broadly defined. By health systems we mean hospitals and physician organizations (POs) (i.e., inpatient and ambulatory) that are aligned into single ownership or management structures to deliver coordinated care to patients.⁹ We focus on health systems because strong trends in the U.S. toward both vertical integration (e.g., when hospitals merge with medical groups) and horizontal consolidation (e.g., when hospitals acquire other hospitals)—along with the push toward value-based payment—suggests that efforts to leverage IT to improve performance are likely to be taking place within this particular organizational context. While some prior research on health systems has focused on adoption of specific types of IT,^10,11 less is known about how IT is being used by health systems to improve performance.

We describe health system attempts to leverage IT by analyzing responses from Chief Information Officers (CIOs) and other C-suite executives across 24 health systems who are managing these efforts in a range of market contexts. Although these health systems and their executives are not necessarily representative of all health systems nationally, our findings illustrate the range of activities health systems are engaged in to leverage health IT to improve performance, broadly defined. These findings help move the policy debate beyond federal incentives and requirements for using health IT and payment reform, and suggest a reorientation toward supporting efforts that create and disseminate best practices for how to optimally leverage the technology.

2. Methods

We collected data as part of the RAND Center of Excellence on Health System Performance under the Agency for Healthcare Research and Quality’s (AHRQ) Comparative Health System Performance (CHSP) Initiative.¹²

2.1. Sample

We identified, recruited and collected data on a total of 24 health systems across four states (California, Washington, Minnesota, and Wisconsin) that are each hosting a health care improvement collaborative and agreed to provide performance data. We used maximum variation sampling to achieve variability on selected attributes (e.g., size and performance using a composite of publicly reported quality measures). For further detail on sampling, see Appendix A.

2.2. Data collection

A team of Ph.D.-level investigators experienced in elite interviewing conducted 90-min semi-structured interviews with the Chief Information Officer and other c-suite-level executives in each of the 24 health systems (for a total of 162 executives). The interviews were conducted by telephone between August 2017 and March 2019. Interviews with CIO’s and/or Chief Medical Information Officer included both clinical and infrastructure-related questions and were conducted by researchers with a health IT background (See Appendix A for content of the health IT-related interviews.). Interviews were recorded and professionally transcribed. Our Institutional Review Board approved this study.

2.3. Data analysis

We used Dedoose, a web-based tool that facilitates the coding and analysis of qualitative data,^13,14 to identify all material from the CIOs, and any relevant IT-related content from other health system executives. (Material unrelated to health IT was analyzed and is being reported separately.) From this corpus, two investigators (RSR, SHF) developed an initial codebook based on the interview questions. We refined the codebook inductively by reviewing relevant transcript excerpts. Both reviewers then coded transcripts from three health systems; reviewed each other’s coding; discussed any inconsistencies until consensus was reached; finalized the codebook; and coded the remaining transcripts. Within each code, we used the constant comparative method¹⁵ to identify themes within two categories of IT-related activities that emerged from the data: (1) laying the foundation to improve performance with IT; and (2) using IT to improve performance.

We classified IT-related activities as more or less advanced based on executives’ descriptions of (1) some activities as being more basic or elemental and to be superseded (sooner or later) by other more complex or sophisticated structures, actions, or processes, (2) consistency in responses among executives regarding the direction of their health system as well as their impressions of broader trends, (3) dependencies between activities necessitating that the less advanced occur prior to the more advanced, and (4) degree of systematization compared with more ad hoc approaches.

3. Study findings

Table 1 summarizes key characteristics of the 24 health systems. There are clear differences across the health systems in terms of their size, organizational structure, governance, mission, culture (including communication), financing, and EHR, which reflect, in part, the origins and history of each system and the markets in which they operate.^16–18 Based on the interviews, we identified a series of IT-related activities that could lead to higher performance, which we sorted into two broad categories: laying the foundation for performance improvement and actually using IT to improve performance. The types of activities were similar across health systems but, within a given type of activity, some health systems were more notably advanced than others .

Table 1.

Characteristics of health systems and their IT infrastructure.

No. Hospitals owned (mean, range)a		6 (1–37)
No. Physician Organizations (mean, range)		[0–10+]
Mean Percent Primary Care Physicians employed (mean, range)		62%
% with Clinically Integrated Network		46% (11 of 24)
% with Accountable Care Organization contract		83% (20 of 24)
Primary EHR vendor
Epic		15
Cerner		6
Meditech		2
Allscripts		1
Current EHR structure	In transitionb	Not in-transitionb
Single instance (one vendor)	1c	16
More than one instance (one vendor)	2
Inpatient and ambulatory use different EHR vendors	3	1
Multiple EHR vendors within a single setting (inpatient or ambulatory)	1

IT=Information Technology; EHR = Electronic Health Records.

^aSome health systems also had affiliated hospitals.

^bIn-transition means health system is in the process of reducing number of vendors and/or instances or changing EHR product. Not in-transition means health system has no plans to change vendors or instances.

^cHealth system is making major upgrade to EHR product.

3.1. Laying the foundation to improve performance with IT

This category includes five types of activities that may not improve performance directly but are foundational to leveraging IT to improve performance (see Table 2 for examples.) The most substantial effort by the health systems was devoted to work in this category.

Table 2.

Laying the foundation to improve performance with IT^a.

Type of Activity	Characteristics of Less Advanced Systems	Characteristics of More Advanced Systems
Making structural changes to IT platforms	• Multiple vendors and/or instances	• One vendor and one instance (or minimal number of vendors/instances)
Standardizing data and analytic functions	• Data are fragmented across multiple EHR instances or vary even within one instance because of local control; multiple analytics functions produce inconsistent results	• Clinical data from EHR largely standardized and in one platform; health system-wide analytics capabilities; multiple streams of clinical and financial data standardized and integrated
Standardizing care processes in EHR	• Few care processes (e.g., order sets and pathways/protocols) standardized; cultural resistance, or technical constraints impeding progress; multiple/independent scheduling systems and inability to easily schedule across POs	• Most or all order sets and common pathways/protocols completely standardized and hard-wired into EHR; system-wide scheduling system with ability to schedule across system
Configuring and enhancing the EHR	• Eliminating previously made EHR customizations that prevent or complicate upgrades; unable to exchange records with external organizations; cannot monitor EHR usage	• Optimizing EHR; upgrade process is standardized; using third-party tools to overcome EHR’s limitations; participating in multiple data exchange efforts; able to routinely monitor to target training and identify problems • Example: “We monitor the hell out of it. When we see a physician that has lower [scores] than we would expect, we go and we talk with that physician to understand the barriers. Is it because we have a poor order set? Is it because we don’t have an order set for them?”
Maintaining a secure and reliable infrastructure	• Not confident of ability to detect and respond to a cyberattack, recent experience losing money or exposing patient data	• Confident of ability to detect and respond to a cyberattack

IT=Information Technology; EHR = Electronic Health Records.

^aNote: Authors’ analysis and quotes from interviews with executives in 24 health systems across 4 states. Characteristics of less and more advanced health systems for each types of activities was determined based on health system executives of their progress and future plans (see Methods).

3.1.1. Making structural changes to IT platforms

Two thirds were already using the same EHR vendor on a single instance and most others were in the process of converting many or most of their hospitals or clinics to a single instance, which executives generally agreed in the near term resulted in lower software costs and greater ease in managing and upgrading to new functionalities, while enabling higher performance in the future in multiple ways, such as better information sharing, easier ability to standardize data for quality and safety measurement and reporting which is important for participating in ACOs, ability to standardize and optimize care processes and better integration of care. Only one health system used more than one EHR instance (one for inpatient and one for outpatient) without plans to transition – they believed that such a transition would require an immense amount of work and they had already spent 10 years optimizing their current system which they thought was working well enough. Most of the health systems were helping newly acquired POs and hospitals adopt the system’s EHR, an effort that diverted substantial resources away from other IT projects. Although these efforts were organizational priorities, they require many years to accomplish. For example, one system started the transition to a health system-wide EHR in 2012 and hoped to complete it by 2020.

3.1.2. Standardizing data and analytic functions

Executives said that standardizing data and analytics across the health system was necessary to use the data for future activities to achieve high performance, such as monitoring areas for improvement. Health systems that were less advanced in this area were just beginning to establish a data and analytics department and planned for it to be operational within a few years. Health systems with multiple EHR vendors had more challenges assembling their clinical data in one place, complicating or preventing use of those data in analytics. (This shortfall was a major driver toward establishing a single EHR instance across the health system.) Even some health systems with a single EHR instance had multiple hospitals or POs that operated largely autonomously and allowed data elements to be defined locally throughout the system. In one extreme case, roughly 2500 unique patient visit types (coded values indicating the type of service or setting) were used when only approximately 15 were needed, complicating analytics as well as efforts to standardize scheduling and improve access to care across the system.

3.1.3. Standardizing care processes in EHR

Executives viewed process standardization efforts as essential to functioning as a single system, meeting patients’ expectations for consistency in experience and quality across the health system, and spreading initiatives aimed at improving performance across the whole system. The EHR was critical for “hard-wiring” the process. Targets for process standardization included: clinical workflows; order sets; clinical pathways and protocols; clinical guidelines; scheduling workflows; and decision support rules. Processes related to blood use, sepsis, c-sections, and oncology were the most frequently mentioned clinical domains that were targets of process standardization. More advanced health systems in this area had been working to standardize processes for many years (almost two decades in one case) so standardization was part of the culture, but most were less advanced and struggled because they had previously allowed their hospitals or POs to develop their own local processes.

3.1.4. Configure and enhance the EHR

Less advanced health systems were working on basic EHR activities, spending most of their efforts undoing previously made customizations, and installing and learning how to use EHRs tools, such as those used to monitor physician EHR use to individually target additional training. At the other extreme, some health systems had so optimized their EHRs that they were encountering limits to the software capabilities and looking for third-party solutions.

3.1.5. Maintain a secure and reliable infrastructure

Cybersecurity has recently emerged as a priority for multiple reasons: an increase in threats (not specific to healthcare) such as ransomware; fines from state and federal government for breaches of patient data; and in one case, a cyberattack that stole money from the health system. Some executives were more concerned about cybersecurity than others, but we did not have sufficient data to identify detailed characteristics or more and less advanced health systems in this regard.

3.2. Using IT to improve performance

Building on their foundational work, health systems leveraged their assets and processes (e.g., IT hardware, software, data and standardization of processes) on activities that were directly aimed at improving performance. Five types of such activities emerged (see Table 3 for examples.)

Table 3.

Using IT to improve performance^a.

Type of Activity	Characteristics of Less Advanced Systems	Characteristics of More Advanced Systems
Using data/analytics to monitor and provide feedback	• Data require manual downloads to use in analytics programs; analytics distributed across health system without coordination; measures include only what is required by regulations	• Real-time, clinician-friendly dashboards available in EHR, updated automatically, and integrated in workflows; continually developing new dashboards with multi-level views of clinical and financial data beyond reporting requirements; planning or piloting AI/ML tools • Example: “We build tools, dashboards and reports and other things based on business needs … The providers don’t even know it’s not actually in [EHR] … The real power is in combining the clinical and financial data to give a really more full picture of our performance.”
Improving uptake of evidence-based medicine	• Does not use EHR to implement evidence, or relies only on basic CDS alerts (e.g., drug-drug) to implement evidence	• Systematic process for implementing/updating new evidence (e.g., clinical review committees, third-party subscription) and hard-wiring in EHR (e.g., care pathways)
Identifying and reducing variation in care and overuse	• Ad hoc analysis of data, resistance to change practice • Example: “… the imaging operational component in our system has been absolutely unwilling to allow us to do much because they’re worried about losing their overall profitability.”	• Systematically analyzing data (e.g., orders, referrals, use of data exchange) to identify variation and outliers; process redesign and working with clinicians; planning/piloting AI/ML solutions • Example: “Sometimes we’ll have a spike of a med that some doc has never ordered before, and then all of a sudden we go why would you order this med. Then we’ll call the office and be like oh, so really, the drug rep just showed up yesterday, and then that day, later that day, Doc X ordered five of this med they’ve never ordered before and should never be ordered. We track all of that …” • Example: “We have some great examples where we have physicians that were of equal quality but leaving patients in the hospital for an extra day or two days, and sitting down with them and saying, ‘Why is that? You have the same outcomes, but you’re spending more. What are we missing about that?’ Then coming out of those meetings where the physician says, ‘I can easily drop off a day.’”
Improving prevention and population health management	• Minimal or just starting to address	• Uses a population health platform to identify high risk patients; beginning to experiment with AI/ML for risk stratification
Making care more convenient and interactive for consumers	• Multiple patient portals (e.g., inpatient and ambulatory) with minimal use; portal implemented solely to address meaningful use requirements; minimal use of apps or virtual visits	• Patient portal used actively by >50% of patients; ongoing experiments with adding new portal features, apps, virtual visits

IT=Information Technology; EHR = Electronic Health Records; AI = Artificial Intelligence; ML = Machine Learning.

^aAuthors’ analysis and quotes from interviews with executives in 24 health systems across 4 states. Characteristics of less and more advanced health systems for each types of activities was determined based on health system executives of their progress and future plans (see Methods).

3.2.1. Using data and analytics to monitor and provide feedback

This occurred mostly in the form of “dashboards” that allow leadership to identify issues and clinicians to benchmark their performance and identify patients needing additional services. The potential benefit of these tools was widely recognized: “In a world where transparent reporting of outcomes is going to be key, no one wants to be the one that didn’t participate in the improvement process and ended up with worse metrics.” Health systems with more basic data and analytics functions used manual processes to produce dashboards, had few physicians or managers using them, and encountered data and technical challenges when creating new ones. More advanced data and analytics activities involved real-time reporting for individual physicians in their workflows and EHRs, comparisons with peers, and new dashboards frequently developed in response to clinician requests.

3.2.2. Improve uptake of evidence-based medicine

Executives emphasized that implementing new clinical pathways or clinical decision support (CDS) alerts in the EHRs was not sufficient to improve performance because they might not be used. More advanced efforts conducted process improvement systematically—alerting clinicians about variations in care, engaging clinicians in developing solutions (e.g., clinical protocols or order sets), rolling out solutions through education and coaching, “hard-wiring” solutions into the EHR, monitoring compliance, and updating the evidence regularly. Less advanced processes relied on individual physicians or groups to incorporate evidence and lacked a method for regularly updating it.

3.2.3. Identifying and reducing variation in care and overuse

For the most part, health systems in our sample clearly recognized that they had substantial variation in adherence to care process, quality, and outcomes across the system and several were actively using or investigating analytic tools to target variation. Executives gave examples in which their health system had demonstrably reduced variation, such as: implementing the Choosing Wisely recommendations in their EHRs; using CDS alerts, including hard-stops, for imaging and enforcing the use of evidence-based clinical pathways; coaching individual physicians whose metrics identified them as outliers in per-patient total cost of care; and monitoring and enforcing the use of order sets. Several health systems were beginning to explore artificial intelligence (AI) and machine learning (ML) tools to help identify variation. However, even executives who had been engaged in these activities for many years recognized that substantial opportunity remained for improvement. Many health systems were unable to identify variation because data were not standardized, could generate robust reports in only some hospitals or POs but do not integrate them for the whole system, or encountered resistance (e.g., from imaging centers) that were generating profits under their existing mostly fee-for-service payment method.

3.2.4. Improving prevention and population health management

These programs included risk prediction and targeted upstream interventions, such as: prompting primary care physicians and nurses to reach out to specific patients; remote monitoring of high-risk patients; and notifications for emergency visits. However, executives from health systems with even the most advanced efforts in this area said they had just begun this journey.

3.2.5. Making care more convenient and interactive for consumers

These efforts were driven in the short term by local competition. Longer term motivations included fear of new more “patient-friendly” competition in healthcare delivery, the growth of high-deductible health plans (in which patients will have more “skin in the game”) and the general belief that patient expectations for convenience will continue to increase over time. However, executives struggled with determining what to do in this area. All executives who discussed consumerism said that they were just beginning this effort, and some questioned the value proposition. One admitted that the main reason for doing it was to “get ahead of the curve.”

4. Discussion

The 24 health systems we examined engaged in similar types of activities to leverage health IT to advance health system performance but varied considerably in their level of advancement. Although we could not quantify it, health systems seemed to be spending the most effort on foundational activities which may not directly improve performance but which lay the groundwork for doing so. For many activities, health systems were only beginning to leverage their health IT capabilities to improve performance and even health systems that were the most advanced in some areas were only beginning the journey.

Our findings may help explain why the hoped-for IT-enabled transformation has not yet arrived in health care: substantial foundational work is required and many health systems are only beginning these efforts. This finding is consistent with historical studies of technology in other industries that suggest that major changes to technology, organizational processes, and organizational culture take years if not decades.¹⁹ For example, it took early 20th-century manufacturers almost two decades to realize the benefits of electricity. In health systems, this foundational work may be even more challenging than in other industries because it involves fundamental changes in the structure and processes of complex organizations, and requires dedicated resources, commitment by leadership, support by staff, and operational knowledge. The use of application programming interfaces^20,21 promises to ease adoption of IT tools; however, implementation of many IT tools will likely require addressing socio-technical factors and complexities.^22–24

In particular, executives emphasized that the work of standardizing data and processes required considerable sustained effort and many years of work before health systems could perform such value-added functions as effectively moving evidence into practice, systematically reducing variation, and rolling out population health initiatives. This finding is consistent with theorical frameworks of organizational digital maturity and enterprise architecture which describe a stage of standardization as necessary for organizations to undergo in their digital journey.^25,26

Our findings reflect the perspective of health system leaders drawn from in-depth discussions, providing information that cannot be attained from self-reported use of health IT functionalities.^27–29 Studies focused solely on adoption of discrete IT functionalities may produce misleading or inexplicable results. For example, a health system may have adopted a number of health IT functionalities but not be able to use the data to identify opportunities for reducing variation because of disparate data definitions.

Our study has some limitations. Our sample is limited to 24 health systems that have devoted considerable resources toward health IT and the perspective of CIO’s and other c-suite executives. While we did look across a broad range of executives (roughly 7 per health system) for areas of agreement/disagreement in technical and clinical domains, we did not interview frontline staff. As our data is self-reported, results may present health systems as higher performing than they are (i.e., social desirability bias) and executives actively involved in health IT efforts.

The common focus of activities across health systems and variable degrees of progress suggests that there is an opportunity to better disseminate best practices for optimally leveraging health IT to improve performance. The more advanced health systems clearly had discovered or adopted best practices that others had not. It is likely that many health systems are spending considerable effort rediscovering the same lessons that others have already mastered. Murphy et al.’s recent investigation in EHR inbox-related functionality found similar results for one functionality: desired configurations are not adopted across sites, even among those with same EHR vendor.³⁰ Health systems are likely engaging in redundant efforts to create such standards across their systems.^31–33 The substantial effort at foundational work suggests that in many cases best practices do not yet exist. Publicly funded technical assistance has arguably been effective in promoting EHR adoption and as part of some quality improvement programs, but has had limited application in many of the activities we identified.^34–38 These findings suggest that there is a need for greater effort in identifying and disseminating best practices to leverage health IT to more quickly realize its benefits.

In conclusion, although U.S. health systems continue to make substantial investments in IT to improve performance, much of the work is foundational and progress is highly variable. Given the time and effort needed by health systems to make these improvements, current federal initiatives to advance IT and to create value-based payment programs to incentivize performance may be insufficient to stimulate rapid progress.³⁹ Health systems may need direct help to accelerate change, such as through the development and widespread dissemination of proven best practices and more targeted technical and implementation assistance. Absent more such direct assistance, the pace of change in performance from use of health IT may continue to disappoint.

Appendix A. Methods supplement

Sampling design

For sampling purposes, we defined a health system as being comprised of at least one hospital and at least one physician organization (such as a medical group or IPA) affiliated through shared ownership or a contracting relationship for payment and service delivery.

The systems were chosen from four states that represent two geographically-situated pairs (two in the west and two in the mid-west) that are similar in geography and population demographics. The state were California, Washington, Minnesota and Wisconsin.

This convenience sample of states was selected because: (1) these states have been at the forefront of collecting and publicly reporting standardized performance measures; (2) each is hosting a health care improvement collaborative and promoting consumer engagement¹; and (3) the collaborative in each of these states agreed to partner with RAND to provide performance data, to help us understand the local health care market context, to assist us in recruiting a health system sample for analysis, and to collaborate in disseminating findings.

To develop the sampling frame, we obtained from each of our four partners a list of all physician organizations (POs) publicly reporting performance data (e.g., HEDIS, CAHPS, total cost of care measures) in their state. Using secondary data sources, we identified whether each of those POs was affiliated with a health system and, if so, with which health system.^2,3 From that universe of health systems we selected a purposive (non-random) sample of 24 health systems – we targeted 10 in the largest state, and 5 each in the smaller states.

To develop the sample, within each of the four states, we classified health systems on size (large/medium/small) based on the number of physicians and performance (high/medium/low) based on a rough composite of publicly-reported quality measures in that state.⁴ We sampled on size because it was one of the only attributes that the literature suggests is related to high performance.⁵ We sampled on performance because without variation in that dimension, we would be unable to infer whether particular attributes are related to performance. In one of our states, because we were sampling from a larger universe, were selecting a larger number of health systems for study, and had the information available, we were able to consider two additional selection factors: small markets⁶ (yes/no); and type of physician organization (medical group or IPA).⁷

Health systems were not randomly sampled within a region – the recruitment sample was selected to provide some variability on the measured attributes and to skew the overall sample slightly towards larger health systems.⁸ All selections were made by senior statisticians with extensive input from members of the study’s executive committee.

In addition to selecting health systems, we selected a single physician organization from each sampled health system for intensive study. Some health systems were single organizations that employ physicians, but many were multi-entity organizations (such as a corporation with hospital and PO subsidiaries). In the case of health systems with multiple POs, we limited the study sample to a single PO per health system due to resource limitations and the burden data collection represents for these organizations.

In order to ensure a good balance of PO types across the study sample, we included type (medical group v. IPA) as a selection criteria when this information was available in advance of recruitment. When this information was not available in advance, during the initial recruitment call with the CEO of the health system we asked the CEOs to identify the type of POs within their health system. Where there were multiple POs, we varied our selection of POs to achieve a rough balance of PO type within each region.

We have completed data collection in 24 health system/PO dyads. Of the original sample, we had 20 health systems decline to participate. Most refusals had to do with press of business (e.g., leadership were too busy with activities around mergers/acquisition or accreditation) or study burden, but a few declined without stating a reason. As we experienced refusals, we refreshed each state-level sample from the remaining health systems in order to maintain variability on the measured attributes to the extent possible. Our final set of 24 health systems is fairly well balanced on the selected attributes. Across the 24 health systems, a total of 8 executives declined to participate in an interview (even though their CEO had already agreed to the study) and one did not respond despite repeated attempts at contact.

We sent an initial e-mail to the Chief Executive Officer (CEO) of each of the health systems explaining the study and conducted an initial recruitment interview with a key contact in each of the health systems (the CEO or designee). Once we had their agreement to participate, we provided a list of the study domains and obtained from the key contact a list of interview participants. We then invited those executives including the CEO, Chief Financial Officer, Chief Medical Officer, Chief Quality Officer and Chief Information Officer (CIO) to participate in a “virtual” site visit.

Table A.1.

Final Sample by Health System Sampling Attributes

Attributes	All markets	Largest state market
Large	44%	20%
Medium	35%	40%
Small	22%	40%
High performance	35%	40%
Medium performance	35%	30%
Low performance	30%	30%
Operate in small markets		30%
Entirely IPA		10%

Content of health IT interviews

Interview protocols were tailored to each respondent type to focus on their areas of responsibility. Collectively, the protocols included questions on market context; origin of the health system; organization, governance and management; payment and contracting; leadership compensation; the influence of the health system on hospital and PO operations; culture and leadership; physician compensation and performance measurement; health IT; care redesign and population management; quality improvement; moving evidence to practice; the characteristics of high performing health systems; and the “value added” (if any) of belonging to a health system.

In a prior analysis we reported on the variation across health systems in their organizational structure and governance, and on the influence of health systems on their affiliated hospitals and physician organizations.⁴⁰ In this analysis we report findings and lessons learned on their development of IT infrastructure and leveraging that IT to improve performance.

The following tables contain the global codes we used for this analysis.

Table A.2.

Health IT global code

HIT

Health Information Technology

Short Definition: Health information technology (HIT) and electronic health record (EHR). Operational Definition: Statements or discussions about the health system’s or physician organization’s health information technology (HIT) and electronic health record (EHR). Include references to patient registries, patient portals, data warehouses, telemedicine, physician portals, health information exchange, and mobile applications. Include statements about whether HIT is a centralized function, or if individual hospitals or physician organizations own and operate their own HIT applications. Include descriptions about challenges to deploying HIT across the health system or physician organization. Capture descriptions about the history of the development of HIT at the health system or physician organization, including whether HIT was ever centralized and challenges encountered, and whether the health system or physician organization was an early or late adopter of EHRs and other technologies. Include statements about the first go-live date for the health system or physician organization’s EHR and the year in which the health system or physician organization first attested to meaningful use. Capture descriptions about how HIT is currently managed in the health system or physician organization, including decision-making, departments handling HIT, staffing, how HIT is funded, the HIT governance process, and interactions with vendors, including challenges encountered. Include descriptions of the health system or physician organization’s highest priority HIT related strategies or activities and why these were selected. Capture statements about whether external forces are affecting the health system or physician organization’s HIT priorities and activities, and what those forces are. Include references to state and federal regulations or health care reform, including Stark, anti-kickback laws, the Affordable Care Act and the Medicare Access and CHIP Reauthorization Act (MACRA). Capture discussions about the health system or physician organization’s EHR, including whether it is enterprise-wide, what vendor(s) is used, and whether health system affiliates are on the same or multiple EHR instances, and any challenges associated with deploying and implementing the EHR. Capture descriptions about the use of the EHR by clinicians, including whether physicians are required to use the EHR and strategies the health system or physician organization uses to increase physician utilization of EHR functionalities such as using physician champions, training, and financial incentives. Capture statements about whether the health system or physician organization monitors physician EHR usage and whether there are any health system or physician organization interventions in place to change the usage habits of physicians. Include statements about the exchange of digital information across the health system or physician organization, and whether the health system or physician organization participates in a regional health information exchange. Include discussions about how the health system or physician organization uses any HIT applications to address health system or physician organization performance, including the use of HIT to stay up to date on scientific evidence, to put new clinical evidence into practice, to generate quality measures either internally or publicly focused, to identify variations in care, to produce any reports or conduct analyses using patient or provider-level data, or to analyze data for physician performance profiling.

HIT SUBCODES

Table A.3.

Health IT sub code: Strategy

HIT-STRAT

HIT – Enterprise Strategy

Short Definition: The enterprise HIT strategy and management. Operational Definition: Statements or discussions about the health system’s or physician organization’s overall strategy and management of HIT. Include discussions about whether HIT is a centralized function, or if individual hospitals or physician organizations own and operate their own HIT applications. Include descriptions about challenges to deploying HIT across the health system or physician organization. Capture descriptions about the history of the development of HIT at the health system or physician organization, including whether HIT was ever centralized, and whether the health system or physician organization was an early or late adopter of technologies. Capture descriptions about how HIT is currently managed in the health system or physician organization, including decision-making, departments handling HIT, staffing, how HIT is funded, the HIT governance process, and interactions with vendors, including challenges encountered. Include descriptions of the health system or physician organization’s highest priority HIT related strategies or activities and why these were selected. Capture statements about whether external forces are affecting the health system or physician organization’s HIT priorities and activities, and what those forces are. If related to HIT, include references to state and federal regulations or health care reform, including Stark, anti-kickback laws, the Affordable Care Act and the Medicare Access and CHIP Reauthorization Act (MACRA).

Table A.4.

Health IT sub code: EHR Core functionalities

HIT-EHR CORE

HIT – EHR Core Function-alities

Short Definition: Core functionalities of Electronic health record (EHR) or electronic medical record (EMR). Operational Definition: Statements or discussions about the health system’s or physician organization’s electronic health record (EHR) including core functionalities. Capture discussions about whether the EHR is enterprise-wide, what vendor(s) is used, and whether health system affiliates are on the same or multiple EHR instances, and any challenges associated with deploying and implementing the EHR. Include whether the health system or physician organization was an early or late adopter of EHRs. Include statements about the first go-live date for the health system or physician organization’s EHR and the year in which the health system or physician organization first attested to meaningful use. Capture descriptions about the use of the EHR by clinicians, including whether physicians are required to use the EHR and strategies the health system or physician organization uses to increase physician utilization of EHR functionalities such as using physician champions, training, and financial incentives. Capture statements about whether the health system or physician organization monitors physician EHR usage and whether there are any health system or physician organization interventions in place to change the usage habits of physicians.

Table A.5.

Health IT sub code: data exchange

HIT-DATA EXCH

HIT – Data Exchange

Short Definition: Data exchange within and external to the organization. Operational Definition: Statements or discussions about the health system’s or physician organization’s data exchange practices with other providers. Include statements about the exchange of digital information among and between health system and physician organization affiliates as well as external to the health system or physician organization, and whether the health system or physician organization participates in a regional health information exchange. Capture discussions about any challenges related to data exchange.

Table A.6.

Health IT sub code: other EHR functionalities

HIT-OTHER EHR

HIT – Other-EHR Function-alities

Short Definition: Other EHR functionalities. Operational Definition: Statements or discussions about the health system’s or physician organization’s health information technology (HIT) functionalities. Include references to patient registries, patient portals, data warehouses, telemedicine, physician portals, and mobile applications.

Table A.7.

Health IT sub code: performance impact

HIT-IMPACT PERF

HIT – Impact of HIT on Performance

Short Definition: Impact of HIT on performance. Operational Definition: Statements or discussions about how the health system or physician organization uses any HIT applications to impact performance, including the use of HIT to stay up to date on scientific evidence, to put new clinical evidence into practice, to generate quality and safety measures either internally or publicly focused, to identify variations in care, to produce any reports or conduct analyses using patient or provider-level data, or to analyze data for physician performance profiling.

Logic Model

We based our health IT interview questions on topics derived from recent systematics reviews^41,42 and the goals of the focus on health systems and performance. To help structure the interview topics, we used the following logic model, based on Donabedian’s structure-process-outcome framework.⁴³

Table A.8.

Logic model and interview topics

Structure	Process	Outcomes (performance)
Vendor and instances of EHR	Decision making process and influences	Quality measures
Management structure of IT department	Data exchange internally and externally	Health outcome measures
EHR capabilities • Basic functionalities (computer-provide order entry, e-prescribing)	EHR implementation and training methods	Cost
• Clinical decision support • Data Exchange	Monitoring EHR usage	Access to care
• Generating reports	Incorporating new evidence into practice	Security
	Identifying variation in care	Adherence to regulatory requirements