Benchmarking health IT among OECD countries: better data for better policy

Abstract

Objective

To develop benchmark measures of health information and communication technology (ICT) use to facilitate cross-country comparisons and learning.

Materials and methods

The effort is led by the Organisation for Economic Co-operation and Development (OECD). Approaches to definition and measurement within four ICT domains were compared across seven OECD countries in order to identify functionalities in each domain. These informed a set of functionality-based benchmark measures, which were refined in collaboration with representatives from more than 20 OECD and non-OECD countries. We report on progress to date and remaining work to enable countries to begin to collect benchmark data.

Results

The four benchmarking domains include provider-centric electronic record, patient-centric electronic record, health information exchange, and tele-health. There was broad agreement on functionalities in the provider-centric electronic record domain (eg, entry of core patient data, decision support), and less agreement in the other three domains in which country representatives worked to select benchmark functionalities.

Discussion

Many countries are working to implement ICTs to improve healthcare system performance. Although many countries are looking to others as potential models, the lack of consistent terminology and approach has made cross-national comparisons and learning difficult.

Conclusions

As countries develop and implement strategies to increase the use of ICTs to promote health goals, there is a historic opportunity to enable cross-country learning. To facilitate this learning and reduce the chances that individual countries flounder, a common understanding of health ICT adoption and use is needed. The OECD-led benchmarking process is a crucial step towards achieving this.

Objective

Despite markedly different approaches to the organization and financing of healthcare, countries across the globe are moving towards increased use of information and communication technologies (ICTs) to improve healthcare delivery. There is shared enthusiasm about the potential for these technologies to address the common challenges of inconsistent quality and inefficient delivery of care. There is also growing sentiment that health ICTs are essential to reforming existing care delivery models and rapidly changing the way care is provided in the future.^1–3 Investment in health ICTs is viewed by many governments as key to modernizing their healthcare systems and a ‘cost of doing business in 21st century healthcare.’⁴

Even with the broad enthusiasm for health ICTs, there have been substantial challenges in both implementing these systems and extracting gains in quality and efficiency. Many of these challenges, such as physician resistance,⁵ share important similarities across countries and therefore create an unprecedented opportunity for countries to learn from each other. Such learning could substantially improve the likelihood that countries are successful in promoting the adoption and effective use of health ICTs.

To date, cross-country learning has been hampered by a lack of common terminology for health ICTs. Even the widely-used term ‘electronic health record’ (EHR) has substantially different meanings across countries, making it very difficult to determine which policies and approaches could serve as models of successful adoption and use. Until definitional issues as well as variations in approach to measurement are resolved, we cannot know for certain which countries are successful in a given health ICT domain and which countries are struggling.

In response, an international effort, led by the Organisation for Economic Co-operation and Development (OECD) is underway to develop benchmark measures of health ICTs that can be applied  by countries to track their progress. Such measures will help clarify, using comparable metrics, where countries fall within key e-health domains. As no country is at the forefront in all domains, having comparable metrics will enable countries to more readily identify models from which they could learn.

In this paper we seek to raise awareness of this international effort and inform policymakers, researchers, and the broader health informatics community about the anticipated availability of benchmark data. We begin by describing the motivation for and history of the effort. We then discuss the approach to developing benchmark measures, a significant challenge given the different contexts and approaches to ICT adoption across countries. We report the current status of the measures, which are organized in four domains, and illustrate country-specific examples for each. We conclude with reflections on what we have learned thus far in the project, the next steps and the policy implications of this work.

Background and significance

Motivation for ICT adoption

There is substantial interest across countries in fostering the availability and effective use of ICTs to improve the functioning of their healthcare systems as well as the health of their populations. In 2010, an OECD survey of countries identified four core objectives for ICT implementation: (1) to increase the quality and efficiency of care; (2) to reduce the operating costs of clinical services; (3) to reduce the administrative costs of running the healthcare system; and (4) to enable entirely new models of healthcare delivery.

A wide range of ICT systems play a role in addressing these objectives. Moving from paper records to EHRs can, for example, improve the quality of care and make it more efficient by enabling timely access to and better transmission of patient medical information across the healthcare continuum. The effective use of electronic records can also facilitate clinical research, effective public health planning, and the evaluation of healthcare interventions and their quality at the practice level. There is a large body of literature on the experiences of specific organizations and providers in implementing EHRs and other related applications such as e-prescribing and computerized physician order entry (CPOE) systems.^5–8 Taken together, these studies demonstrate that, under the right conditions, health ICTs can drive improvements in quality and efficiency. ICTs can also enable entirely new ways of delivering care. For example, advancements in telecommunications have led to the emergence of tele-ICUs in which specialists can remotely monitor and direct care for the sickest patients who might otherwise lack access to such advanced care.⁹

Governments have recognized the large-scale changes that are made possible by health ICTs and in response they are developing approaches to leverage these technologies to pursue a range of health system reforms, such as primary care renewal and pay for performance. The first step towards realizing the array of potential benefits from health ICTs is for governments to develop an e-health strategy. E-health strategies define and prioritize the approach to ICT adoption by describing underlying policies’ intended impact on ICT adoption and the resulting goals from adoption. In 2008 a review of 27 European Union (EU) countries found that the majority of governments had formulated specific strategies about their intentions and priorities for e-health. The most commonly stated policy targets were efficiency, improving or reforming the healthcare system, improving quality of care, and promoting patient-centered services.¹⁰

Need for benchmarking

As the USA and other countries develop and implement their e-health strategies, they will need to monitor progress to ensure their efforts are effective. In 2007, the OECD undertook a study on how member countries were monitoring their own heath ICT progress under their e-health strategy. The goals of the study were to identify: (1) the policy objectives that underlie the e-health strategy and the information needed to determine whether the objectives are being met; (2) best practices which might be further developed and implemented across various clinical settings; (3) a framework for the selection of internationally comparable indicators; and (4) areas for international action and future research efforts. The study concluded that available national and international data on health ICTs are rarely comparable, due to inconsistent definitions (eg, what constitutes an EHR differs across countries) as well as statistical reasons (eg, different sampling techniques). As a result, it is difficult to draw global conclusions on ICT adoption, use, or impact on care. It is similarly challenging for countries to identify candidate models from which they could learn.

Methods

Early work: defining the vision and high-level approach

Prompted by the 2007 study, the OECD laid the foundation for the development of benchmark measures in health ICT. Two critical decisions emerged from their activities. First, it would be useful to organize benchmark measures along a continuum, starting from ICT availability, moving next towards effective use, and ending with measuring outcomes and impact on population health. A continuum-based approach has the advantage of accommodating countries that are at different levels of maturity and progress towards achieving their e-health goals. In particular, advanced countries are unlikely to devote substantial resources to collecting data on the availability of ICTs if their policy needs are focused on effective use and better outcomes. Having a continuum approach allows these countries to participate in the broader process.

The second critical decision was to use the OECD ‘model survey’ framework, which takes a staged approach in moving international measurement work forward. To be broadly useful, the OECD model survey is composed of separate, self-contained modules that ensure flexibility and adaptability to a rapidly changing environment. Core modules can be added on to existing national surveys or administered as a stand-alone survey, while supplemental modules can be used as needed by countries. The approach allows broad measurement of core concepts on an internationally comparable basis while allowing countries to tailor some of the content they collect to address country-specific needs.

Operationalizing the vision

One of the key challenges to a model survey is to ensure that the terminology has comparable meaning across different countries, and that when changes are made by individual countries, they are done in ways that maintain this comparability. For example, while many OECD countries use the terms electronic medical record (EMR) and electronic health record (EHR) interchangeably, in Canada, EMRs refer to systems used by a healthcare professional to manage patient health information in a specific medical setting. The EHR is a distinct concept that involves pooling data from multiple different clinical settings, allowing access to a more comprehensive patient record. If a core module question asked physicians about EHR use, the answers from Canada and the USA would, for example, mean very different things. These differences in interpretation and approach across countries would impede meaningful benchmarking.

To avoid such issues, we focused on developing indicators using a functionality-based approach. The underlying notion was simple: while it was unlikely that we could achieve consensus across all OECD countries on what constitutes an ‘electronic health record,’ we could likely achieve consensus on core types of clinical activities that are supported by electronic systems. This would allow us to create sample survey questions within modules that would enable policymakers to generate comparable benchmark measures.

Case studies

To decide which functionalities to include in the initial set of benchmark measures, a group of seven OECD countries were selected to serve as case studies: Australia, Canada, Denmark, England, Finland, the Netherlands, and the USA. We purposefully selected countries with more mature e-health strategies in order to ensure sufficient health ICT measurement activity to review. However, in order to ensure that our initial focus on these advanced high-income nations did not result in measures that would not generalize, we are currently engaging a broader set of countries in the refinement and finalization of the measures (described below in the ‘Moving forward’ subsection of ‘Results’).

For each case study, key sources of data on both the ‘availability’ of electronic systems and the ‘use’ of these systems which were available in English, were identified. To develop an initial set of functionality-based measures for availability and use, we created four categories of broadly defined domains in which ICTs support care delivery:

1. Provider-centric electronic records

Often referred to as electronic medical records (EMRs), electronic health records (EHRs), or electronic patient records (EPRs), provider-centric electronic records include systems that are used by healthcare professionals to store and manage patient health information and data, and include functionalities that directly support the care delivery process.

• Patient-centric electronic records

Often referred to as personal health records (PHRs), patient portals, and other patient-centric electronic records, these systems are typically used by patients and their families to access and manage their health information and organize their healthcare.

• Health information exchange

Health information exchange (HIE) refers to the process of electronically transferring, or aggregating and enabling access to, patient health information and data across provider organizations. Exchange may take place between different types of entities, for example, e-transfer of patient data between ambulatory care providers or e-transfer of data at the regional level.

• Tele-health

Tele-health encompasses a broad set of technologies that support care between patients and providers, or among providers, who are not co-located. Tele-medicine is often defined as synchronous video-mediated consultations between physicians and patients. However, it may also include applications such as remote home monitoring of patients, tele-ICUs, and tele-radiology.

We compared approaches to definition and measurement within these four domains across countries, focusing on identifying explicit or de facto functionalities included in each domain. For each country, one or more ICT experts and official country representatives were asked to validate our findings (see online supplementary appendix).

Results

Definitions

Table 1 summarizes how case study countries define each of the four domains. The functionalities that achieved a relatively broad level of agreement for benchmarking in the provider-centric electronic record domain, in ambulatory settings, included: (1) storage of patient data, (2) results management, (3) clinical documentation,(4) computerized order entry, and (5) clinical decision support. The functionalities of a provider-centric electronic record in the hospital setting were less often defined. This likely reflects greater heterogeneity in the systems and functionalities that are currently available in the hospital setting.

Table 1.

Overview of country definitions by ICT domain

Country	Provider-centric electronic record	Patient-centric electronic record	Health information exchange	Tele-health
Australia	No single officially sanctioned definition De facto definition includes any type of electronic record with patient data maintained by a provider	National definition An eHealth record is an electronic record that contains a summary of patient health information	National definition as it relates to the creation of the eHealth recordNational exchange standards are defined to support the eHealth record	National definition Tele-health consultations consist of video consults with specialists, consultant physicians, or consultant psychiatrists
Canada	National definition An EMR is a partial health record under the custodianship of a healthcare provider	No single officially sanctioned definition	National definition HIE is used to create a health record that includes relevant health information about a person over their lifetime (called an EHR) Includes six core functionalities National exchange standards are defined to support EHR creation	National definition Delivery of services by healthcare organizations using ICT solutions when the clinician and patient are not in the same location
Denmark	No single officially sanctioned definition De facto definition focused on a set of functionalities (eg, clinical notes, medication administration)	No single officially sanctioned definition	National definition Known as the ‘one-letter solution,’ the HIE approach uses a single electronic form for secure messages between physicians and hospitals Separate public health focused HIE network	De facto definition is digitally supported healthcare service over distance
England	National definition A Summary Care Record (SCR) is a health record that includes three types of health information about a person (eg, current medications and allergies) Locally held record systems are a partial health record under the custodianship of a healthcare provider	No single officially sanctioned definition Nationally provided services enable patient booking of appointments and prescription refills	National definition as it relates to the creation of the SCR National exchange standards are defined to support SCR and other transactions (eg, referrals and discharge summaries)	No single officially sanctioned definition
Finland	National definition in progress De facto definition includes any type of electronic record with patient data maintained by a provider	National definition in progress	National definition in progress	No single officially sanctioned definition
The Netherlands	No single officially sanctioned definition De facto definition includes all electronic information functionalities that assist health professionals in patient care	No single officially sanctioned definition De facto definition includes all electronic information support, mainly by the internet, for the self-management and health professional interaction	National definition National EHCR or EPR that links providers’ local electronic records	National definition Care delivered in which there is a physical distance bridged using ICTs between two parties, one of whom is a healthcare professional
USA	No single officially sanctioned definition Functionality-based de facto national definition Different functionalities for inpatient and ambulatory settings	No single officially sanctioned definition	De facto national definition All electronic information support for cooperation between healthcare organizations and professionals in patient care	National definition All electronic information support for the delivery of care in which a physical distance is bridged between the health professional and the patient

EHCR, electronic health care record; EHR, electronic health record; EMR, electronic medical record; EPR, electronic patient record; HIE, health information exchange; ICT, information and communication technology; SCR, Summary Care Record.

We found that few countries had defined patient-centric electronic records using a functionality-based approach. England was the only country among those we examined that defined and measured patient-centric records. The three key functionalities they included were that: (1) they are internet-based, (2) they support patient entry of personal health information, and (3) they support ambulatory appointment scheduling. In this domain, many other countries examined electronic communication between patients and providers.

HIE was not defined consistently across case study countries. Some countries defined and examined HIE from the perspective of who was connected; others examined it from the perspective of what type of information is exchanged; and others examined HIE in the context of regional repositories or shared electronic records. These differences reflect the country-specific approaches to HIE.

In the tele-health domain we also found little consistency in functional applications across our case study countries. Some countries used telecommunication technology for a wide range of health applications, from physician education to tele-ICUs, while others focused on tele-visits. Even within a given application area, there were important differences in the approach that countries used. For example, in some countries a tele-visit involves a patient connecting to a clinician; in other countries, a tele-visit requires a clinician connecting  to a clinician.

Measurement

Table 2 compares measurement approaches across case study countries in one domain, provider-centric electronic records, which had the most measurement activity.

Table 2.

Country approaches to measuring the adoption of provider-centric electronic records

Country	Measurement approach: provider-centric electronic record availability and use
Australia	Ad hoc academic and foundation surveys and reports
Canada	Regular data collection from provincial/territorial governments and programs based on standard metrics Ad hoc academic and foundation surveys and reports
Denmark	National survey conducted by the Danish Centre for Health Informatics; yearly Ad hoc academic and foundation surveys and reports
England	Department of Health and Connecting for Health Evaluation Programme maintains a central database to monitor the use of Summary Care Records Ad hoc academic and foundation surveys and reports
Finland	National survey conducted by the National Institute for Health and Welfare; every 2 years Ad hoc academic and foundation surveys and reports
The Netherlands	Ad hoc academic and foundation surveys and reports
USA	Annual surveys conducted by the American Hospital Association (AHA) and the National Center for Health Statistics (NCHS) Other annual and ad hoc surveys by industry groups, academics, and foundations

Most case study countries approached the measurement of the ‘availability’ of health ICTs similarly, using a survey to ask health professionals about the presence of specific functionalities. There were, not surprisingly, marked differences in how countries measured the ‘use’ of health ICTs. Some countries with more centralized ICT approaches, like the UK, have the ability to measure use directly by gathering data from the electronic systems themselves. Other countries, like the USA, rely on surveys and ask respondents directly about their use of key electronic functionalities (or whether specific tasks are performed on paper or electronically).¹¹ A subset of these countries is more specific and asks how often each functionality is used (eg, some of the time, most of the time, all of the time). In the USA, there has been an explicit effort by the federal government to define and measure whether provider-centric records are used to perform a specific set of clinical functions (known as meaningful use) and tie such activities to financial incentives.¹²

Challenges

The case studies enabled us to identify key challenges to developing a standardized, module-based approach to functionality-driven ICT benchmarking. To our surprise, we found relatively few conflicts in how surveys described functionalities. The larger problem was that many of the surveys did not use functionalities at all, or when they did, they did not consistently include the same set. This also suggested a lack of clear distinctions between the four domains.

The more concerning issues emerged from approaches to measuring use. As described earlier, some countries were able to collect data on use directly from their electronic systems. For those that relied on surveys, we observed substantial variations in who responded. This issue is particularly relevant in the hospital setting, where a survey that targets the chief information officer (CIO) and asks about use for the entire organization is likely to obtain different answers than one that targets front-line health professionals.

Moving forward

Given the limited set of functionalities that is currently measured across countries and the inconsistencies in measurement approach, the OECD decided that further multi-country input was needed to develop an initial set of benchmark measures. Four taskforces, one for each domain and with representation from at least seven countries, are currently working to develop and refine a set of functionality-based indicators and associated sample questions that will comprise the model survey modules (table 3). Despite the lack of clear distinctions between the four domains, the domains were preserved in order to structure and report on measure development. The taskforces are coordinating their efforts and we anticipate that the final modules will be organized in a way that does not require rigid classifications (eg, questions can be relevant to multiple domains).

Table 3.

Proposed benchmark measure functionalities by domain

Provider-centric electronic record	Patient-centric electronic record	Health information exchange	Tele-health
1. Entry of core patient data (eg, medication allergies, clinical problem list) 2. Decision support (eg, drug–drug alerts) 3. Closed-loop medication administration 4. Clinical documentation	1. Viewing of clinical data (eg, test results) 2. Supplementation of clinical data (eg, entering or modifying current medications) 3. Appointment scheduling 4. Medication renewal	1. Secure messaging between professionals 2. Ordering and reporting of medications and laboratory tests with result receipt 3. Patient referrals	1. Tele-home care/tele-monitoring 2. Remote consultation 3. Asynchronous communication

Countries that are represented on the taskforces include Argentina, Australia, Belgium, Brazil, Canada, the Czech Republic, Denmark, Finland, France, Israel, Italy, Japan, the Netherlands, Norway, Poland, South Korea, Spain, Sweden, Switzerland, the UK, and the USA. The initial modules are expected to be available in mid-2013 for countries to pilot along with a methodological guide to promote the validity and comparability of resulting benchmark measures. The guide describes the types of meta-data, such as response rates and sampling approaches, which countries will be asked to report in order to help assess the validity of their data. To promote comparability, the guide leverages international classifications of healthcare delivery settings (eg, hospitals) as well as healthcare professionals (eg, physicians) to ensure consistent interpretations of the target populations and respondents.

Several countries will test the modules and methodological guide to provide feedback to the broader OECD community about how they work in the field in order to enable their refinement. We anticipate reporting on their experience, which will inform  the viability of our approach to develop benchmark measures of health ICT adoption, in early 2014. We also expect to report on an expanded set of benchmarking domains that countries have suggested. In particular, many countries are interested in measures of mobile health (m-health), which strongly appeal to both developed and developing nations.

Discussion

Lessons learned

We have learned several key lessons to date. First, there is no monolithic approach to ICT adoption or measurement. Countries have distinct approaches that are shaped by their healthcare system structure, priorities, and resources. For example, England and many of the smaller European countries have pursued a more centralized approach with more shared infrastructure. In contrast, the USA is using financial incentives to promote the adoption of provider-centric records and HIE, with hundreds of potential systems that could be implemented with almost no central infrastructure. This is one of the reasons benchmarking is so challenging.

Second, the need for benchmarking, and the international enthusiasm for doing it, is substantial. No country is advanced in all four domains and each is looking for models of adoption and effective use. While e-health strategies are typically expansive and comprehensive, countries have made different decisions about which types of ICT to prioritize. The USA appears to be advanced in the adoption of provider-centric electronic records in the hospital setting and has recently introduced incentives to further increase hospital adoption as well as spur ambulatory adoption. Australia had early success in the ambulatory setting following a similar incentive-based model. Canada is successfully promoting common standards at the federal level to support a longitudinal, patient-centric record. It is these similarities and differences that create the potential benefit from benchmarking. By establishing a shared understanding of each country's current status, countries that are wrestling with how to most successfully implement the next phase in their e-health strategy can identify examples from which to learn.

Finally, we have repeatedly encountered the challenges inherent to a voluntary, multi-country effort to collect comparable measures of health ICT adoption. There is no overarching body in a position to control countries’ decisions about how to collect data. It will be up to each country to strike a balance between the recommended approach to promote comparability and their specific needs and constraints. In addition, there will be little ability to influence the motivation of individuals and organizations across countries to provide the data. While the result is limited ability to guarantee high-quality benchmark data, we are encouraged by the strong voluntary participation of countries over the past several years, the multiple countries that are already preparing to pilot the modules, and the universal perceived need for measures that enable countries to learn from each other.

Conclusion

As countries move to develop and implement strategies to increase the use of ICTs to promote health goals, there is a historic opportunity to enable cross-country learning. This learning occurs when countries share a common understanding of what others have done and how they got there. Doing so requires detailed attention to creating benchmark measures in ways that allow for a granular picture of each country's status and progress.

Given the substantial investment that countries are making in health ICTs, the financial and health costs of failure are high. This is a clear area in which greater cooperation, greater learning, and better models of effective adoption and use can be immensely helpful. In order to facilitate this learning and reduce the chances that individual countries flounder in their efforts, a common understanding of health ICT adoption and use is needed. The OECD-led benchmarking process is a crucial step to getting us there.

Appendix 1.

International experts

Country	Expert name	Organization	Title
Australia	Kathryn Bramwell	Australian Public Service	Director, International, eHealth Strategy Branch
Canada	Jennifer Zelmer, PhD, MA	Canada Health Infoway	Senior Vice President, Clinical Adoption and Innovation
Canada	Mark Nenadovic, MHSc	Canada Health Infoway	Group Director, Emerging Technologies Group
Denmark	Christian Nohr, PhD	Aalborg University, Department of Development and Planning	Professor
Denmark	Kenneth Bøgelund Ahrensberg	Division of the National Board of e-Health, Denmark	Liaison Officer
England	Jeremy Thorp	NHS Information Reporting Services DH Informatics Directorate	Programme Delivery Director
Finland	Paivi Hamalainen, Psy-PHD	STAKES, National R&D Centre for Welfare and Health	Head of Centre of Excellence for Information and Communication Technology in Health and Social Care
The Netherlands	Michiel Sprenger, PhD	National IT Institute for Healthcare	Senior adviser IT and innovation
The USA	Michael F Furukawa, PhD	US Department of Health and Human Services Office of the National Coordinator for Health Information Technology	Director, Office of Economic Analysis, Evaluation, and Modeling