Steps in Moving Evidence-Based Health Informatics from Theory to Practice

Abstract

Objectives

To demonstrate and promote the importance of applying a scientific process to health IT design and implementation, and of basing this on research principles and techniques.

Methods

A review by international experts linked to the IMIA Working Group on Technology Assessment and Quality Development.

Results

Four approaches are presented, linking to the creation of national professional expectations, adherence to research-based standards, quality assurance approaches to ensure safety, and scientific measurement of impact.

Conclusions

Solely marketing- and aspiration-based approaches to health informatics applications are no longer ethical or acceptable when scientifically grounded evidence-based approaches are available and in use.

I. Introduction

Over the last 15 years the expectation that evidence-based principles should be applied in health informatics practice, bringing the discipline into line with all other health sciences and technologies, has gained ground. This concept is called evidence-based health informatics (EBHI). It is defined as the conscientious, explicit, and judicious use of the current best evidence when making de cisions about the introduction and operation of IT in a given healthcare setting [1]. This concept has to be an ethical imperative since no practice change in healthcare should be implemented unless it is proven to be safe, beneficial, and introduced in a way which optimises net benefits.

Much of the work on EBHI has been undertaken by the International Medical Informatics Association’s Technology Assessment and Quality Development Working Group (IMIA WG), or by individual members. A powerful trigger event was the exploratory workshop run by Elske Ammenwerth in Innsbruck in 2003, which set the vision and created an implementation plan [2]. A major milestone was the review of progress over the following decade in the IMIA Yearbook 2013 which took EBHI as its theme [1]. The latest significant step was the publication of a comprehensive globally sourced book on EBHI [3].

Research and a research mindset underpin the EBHI approach in two essential ways. First, research is needed to create the tools and methodologies to evaluate health informatics systems in piloting and in practice, as these are the essential (and arguably the only) means of generating evidence on which to base informed practice. Secondly, research is important to assess the effectiveness of different policy-setting approaches. While evaluation itself is arguably not research as (similarly to audit) it looks at whether intended and expected outcomes are achieved, a strict research mindset and approach are needed to produce objective, trustworthy, and credible evidence.

However, a sound idea is of little value unless it is put into practice. Translation of the principles of EBHI into sound national policies, and implementation of those policies into operational health systems, are essential next steps. This paper reports four current practical initiatives. These examples were selected for their currency and to demonstrate the need to span from awareness of EBHI to addressing the specific valid concerns of various stakeholders in national policy formation and health informatics application.

II. Bringing an Evidence-Based Approach into National Awareness – the BCS and UK eHealth Week

The United Kingdom is a long-standing location for innovation in health information technology. The British Computer Society (BCS, the Chartered Institute for IT) has a specialist group, BCS Health [4], which has played a key part in promoting health informatics in the UK. BCS Health is the UK national member body for IMIA and for the European Federation of Medical Informatics (EFMI), and it publishes an academic journal, the Journal of Innovation in Health Informatics [5]. BCS Health has extensive collaborative participation, including representation in international health informatics standards development work, and in national, clinically-led information standards development through the UK Professional Record Standards Body [6].

BCS Health has taken a leading role in the development of health and care informatics as a recognized and regulated discipline, now taking shape as the Federation for Informatics Professionals, in partnership with the Institute for Health Records and Information Management (IHRIM) and the UK Council for Health Informatics Professions (UKCHIP). BCS Health also participates in the national Chief Information Officer (CIO) network and has an advisory role in the formation of a Faculty of Clinical Informatics.

As an independent learned society, BCS Health takes a ‘critical friend’ stance towards political strategies in health information technology. This has been demonstrated in making constructive proposals for successive governmental programmes (such as [7]) and offering frank and reasoned critiques of policy strengths and weaknesses, such as [8].

For over three decades, BCS Health has staged a major annual event which provides a forum for national policy and programme discussions, showcases commercial products and services, and presents a scientific programme. Originally known as Healthcare Computing (HC), the national event is now run collaboratively by BCS Health, HIMSS (Healthcare Information and Management Systems Society), and NHS England, and it has been re-branded as UK eHealth Week since 2015. In 2016, the BCS Health programme strongly featured EBHI as a conference theme. To support that, top-level international expert speakers (including Professor Charles Friedman of University of Michigan, USA and Professor Enrico Coiera of Macquarie University, Sydney, Australia) were invited to participate in an academic forum and to give presentations in several conference sessions on the theme of creating and applying rigorous evidence. A forthcoming issue of the Journal of Innovation in Health Informatics will feature selected papers from the event.

There remains a tension between a generic policy demand for ‘going digital’ on the assumption of massive cost savings and beneficial service transformation, the reality of what the industry is offering, and the pragmatic need for genuine evaluation of what works and what does not so as to give solid and credible evidence. BCS Health continues to work nationally and internationally to promote best practice in health and care informatics through its role as a constructive yet critical advocate at national-level health informatics policy and practice as well as an advocate of building forward based on research-standard evidence.

III. Ensuring Safety of Health Informatics – Detecting and Mitigating Health IT Safety Issues

Translation of evidence into policy and practice is critical to prevent patient harm. Alongside its many benefits, health IT can disrupt care delivery and pose risks to patient safety. The capacity to reap the optimal net benefits of health IT is contingent upon detecting and mitigating risks throughout the IT lifecycle, including design, implementation, and use. A variety of national strategies, systems, and standards can aim to improve the safety of health IT, the most important of which are presented here.

1. Safety Management Systems

These have evolved in other high-risk industries, such as aviation. Safety management systems encompass the overall set of processes used to identify and mitigate hazards throughout the life cycle of a system. England’s safety management program for health IT which was initiated in 2005 is one of the best known programs [9]. For national-scale systems, software manufacturers are required to create a safety case which sets out the evidence of how hazards have been identified and managed. The safety case will be continuously updated with new hazards identified during implementation or when changes are made to the system.

2. Standards

There are few international standards that directly address the safety of health IT [10]. England’s safety management program has implemented two standards for managing clinical risks in the design, implementation, and use of health IT [10]. These standards were formally adopted as NHS standards in 2009 and are applicable to IT systems that are procured locally by NHS Trusts (which are public sector organisations that provide health services).

3. Guidelines

In many nations, practices for safe design and implementation are being promoted via guidelines. One of the most comprehensive guidelines for health IT is the Safety Assurance Factors for EHR Resilience (SAFER) guides sponsored by the US Office of the National Coordinator for Health IT [11]. The nine SAFER guides are intended to be a set of proactive, self-administered risk assessment tools for manufacturers and healthcare organizations. Another guideline which promotes an evidence-based approach is the US National Institute of Standards and Technology (NIST) guide to evaluate Electronic Health Record (EHR) usability [12].

4. Certification

Certification provides independent assurance that software is fit for purpose and that it meets specific requirements for functionality, interoperability and security. Safety is addressed alongside interoperability in the Australian certification program, but it is not explicitly addressed in the United States and Canadian programs, though conformance with functionality, usability, interoperability, security, and privacy requirements may lead to safer systems [10].

5. Regulation

Although standalone software has largely been outside the strict regulatory regimen applied to medical devices, current initiatives indicate a gradual move towards regulation. In Europe, the safety of medical devices is regulated through a directive that focuses on manufacturing and pre-market testing leading to a declaration of conformity [10].

6. Surveillance of Emerging Safety Issues

The monitoring of incidents is central to detecting emerging safety problems [13]. While it is mandatory to report incidents associated with regulated software, the reporting of general patient safety incidents (including those involving most health IT) is voluntary. One large-scale program directed at monitoring and responding to IT incidents is part of England’s safety management program, which has been in place since 2005 [9,10]. In other nations, IT incidents are being reported amongst general patient safety incidents and alongside reports of medical device failure and hazards. One source of such reports is the US FDA’s MAUDE (Manufacturer and User Facility Device Experience). Although the FDA does not enforce its regulatory requirements with respect to IT, some manufacturers have voluntarily listed their systems and reported incidents [10].

IV. Linking with Health Technology Assessment

The credibility and reliability of health IT systems as contributors towards safer and cost-effective care have been questioned for over two decades due to the scarcity of methodologically strong evidence. The adaptation of health technology assessment (HTA) methodology has been suggested repeatedly as a remedy to the gaps in health IT evidence production. HTA has been defined as “The systematic evaluation of the properties and effects of a health technology, addressing the direct and intended effects of this technology, as well as its indirect and unintended consequences, and aimed mainly at informing decision making regarding health technologies” [14,15].

Linking research principles with the local application of proven and validated methods in the assessment of treatment and technologies, HTA should have strong synergy with evaluation and production of evidence on health informatics applications. A mapping of shared target areas is now in hand elucidating the synergy potential, which should enable countries to take advantage of local processes [16].

In the field of health informatics, HTA methodology has been tested in the domain of telemedicine services through use of the MAST model [17], an evaluation framework for telemedicine applications, building on the principles of the HTA core model [18]. Recent work on the model has concerned its extension and adaptation to cover aspects of social and healthcare integration as well as its application in accordance with HTA methodological recommendations for observational studies.

In times of financial austerity and reduced resources, both HTA and health IT are striving to achieve the two goals of speed and high quality, experimenting with methods for rapid evidence generation, while preserving high quality standards. Experiences thus far have pinpointed the transferability and generalizability of findings as challenges shared by both domains.

Comparative efficacy and effectiveness assessment, in their traditional and rapid forms, also constitute a meeting point for HTA and health IT—the common ground being the collection of relevant data. At the core of the process are EHR systems, the mainstay of health IT. As the value engrained in extensive collections of curated longitudinal patient data is increasingly recognized, another key source of observational data has emerged in the form of patient registries. Paradoxically, a major limitation of registries as a research resource is the currently low uptake and utilization of IT in their standard operations. In the European Union, the target of high quality and interoperable electronic registry data has been addressed though a collaborative effort of the European Commission and several member states (the PARENT Joint Action) [19], including the engagement of the EU HTA community.

An important pathway in the future development of both disciplines is the incorporation and utilization of patients’ experiences and perceptions as part of regular data collection and analysis processes. The trend concerns the provision of healthcare services as well as statistics and research, particularly with regard to safety aspects of medical technologies.

As technology keeps evolving, new challenges lie ahead for health IT and HTA researchers. Examples reflecting the move of science and practice towards personalized, preventive, and integrative solutions are direct to consumer digital health technologies, and modelling tools [15].

The academic and policy worlds often remain apart from each other. Research organisations and communities should take up the task of establishing communication and collaboration channels to decision makers. Commitment to the objective of generating robust quality and policy-relevant evidence for health IT is a step in the right direction.

V. National Indicators of eHealth Progress

In many fields of health care provision, there is a strong interest in developing indicators to measure progress and enable comparison against benchmarks and against comparable countries. Improving health, quality, and efficiency of health care system and equity of access are often included as key goals [20]. Through initiatives for health sector and health information system reform, many countries are actively building upon their national foundations for eHealth towards these goals. However, leveraging eHealth as a national strategic asset demands a coordinated approach to planning, implementation, monitoring, and evaluation. [21]

To ensure that an evidence-based approach is used when indicators are being developed, a current strong twin-pack research-based development programme is being run under the Nordic Council of Ministers, in collaboration with the Organisation for Economic and Social Development (OECD), “to develop, test and assess a common set of indicators for monitoring eHealth in the Nordic region for use by policy makers and scientific communities to support development of Nordic welfare” [22].

The Nordic collaboration is based on similar progress in adoption of health IT. The countries also share many similarities culturally and politically as well as in healthcare and welfare systems. To this end, a mandate was given to a network of research organisations by the social and healthcare ministries in the Nordic countries to create a common platform for collaboration to work towards measurable policy goals and provision of evidence on progress. It also paves the way to extending evidence-based policy making into eHealth.

For the purposes of monitoring eHealth interventions, the challenge is to define mechanisms via which various eHealth applications or interventions impact goals for healthcare performance and to translate them into quantifiable measures from the viewpoint of various stakeholders [23]. The Nordic collaboration has followed a 4-phase methodology to achieve this: (1) defining the context for measurement (key stakeholders and relevant interventions); (2) defining the goals by combining top-down and bottom-up-approaches; (3) defining methods for indicator selection and grouping; and (4) defining, collecting, and reporting the data for feedback on its utility.

For phases 1 and 2, the eHealth policies in each country were analysed to detect common goals, stakeholders, and interventions, while for phase 3, existing eHealth monitoring measures in each country were collected, grouped according to the goals found in phases 1 and 2, and compared for consistency. At this point, collaboration also started with the development of the OECD model survey on the availability and use of Information and Communication Technologies in the Health Sector [24]. For phase 4, existing national eHealth monitoring surveys were updated (or in some cases, initiated). Data were collected in the Nordic countries, where it was available, for 49 eHealth related healthcare structural, process, output, and outcome indicators. Results were reported as a Nordic Council of Ministers publication [25].

However, as a formal publication is not necessarily the best interface to support decision making, a dynamic reporting system is under development in Finland, combining data from several sources. It allows users to get an overview of progress per strategy goal or trends over time, compare regional situations, and so on. With increasing amounts of automated log and register data collection, the role of research will shift from data collection and basic reporting towards improving the indicators and advanced statistical analysis of their associations to changes in healthcare system performance.

VI. Conclusion

We have presented four practical initiatives to translate the principles of EBHI into sound national policies, including large national conferences as well as systems and regulations to detect and mitigate health IT safety issues, linking with health technology assessment, and national eHealth indicators.

Implementation of health informatics without underpinning scientifically valid evidence has to be considered unethical because actual benefits, risks, and effects are not known, and patients can be disadvantaged or even harmed (and health professional staff and organisations detrimentally affected as well). However, changing the culture away from aspiration and marketing as the prime drivers and towards a rational, evidence-based approach requires sustained effort addressed to suppliers, politicians, healthcare organisations, clinical users, patient representatives, and society, using research-based approaches. This requires action at national and service delivery levels, and a variety of tools and techniques must be used. This paper has presented steps now being taken, and innovations in a number of countries, which can be utilised much more widely, all being based on research and informed application of its findings.