Consensus Statement on Electronic Health Predictive Analytics: A Guiding Framework to Address Challenges

Abstract

Context:

The recent explosion in available electronic health record (EHR) data is motivating a rapid expansion of electronic health care predictive analytic (e-HPA) applications, defined as the use of electronic algorithms that forecast clinical events in real time with the intent to improve patient outcomes and reduce costs. There is an urgent need for a systematic framework to guide the development and application of e-HPA to ensure that the field develops in a scientifically sound, ethical, and efficient manner.

Objectives:

Building upon earlier frameworks of model development and utilization, we identify the emerging opportunities and challenges of e-HPA, propose a framework that enables us to realize these opportunities, address these challenges, and motivate e-HPA stakeholders to both adopt and continuously refine the framework as the applications of e-HPA emerge.

Methods:

To achieve these objectives, 17 experts with diverse expertise including methodology, ethics, legal, regulation, and health care delivery systems were assembled to identify emerging opportunities and challenges of e-HPA and to propose a framework to guide the development and application of e-HPA.

Findings:

The framework proposed by the panel includes three key domains where e-HPA differs qualitatively from earlier generations of models and algorithms (Data Barriers, Transparency, and Ethics) and areas where current frameworks are insufficient to address the emerging opportunities and challenges of e-HPA (Regulation and Certification; and Education and Training). The following list of recommendations summarizes the key points of the framework:

1. Data Barriers: Establish mechanisms within the scientific community to support data sharing for predictive model development and testing.

2. Transparency: Set standards around e-HPA validation based on principles of scientific transparency and reproducibility.

3. Ethics: Develop both individual-centered and society-centered risk-benefit approaches to evaluate e-HPA.

4. Regulation and Certification: Construct a self-regulation and certification framework within e-HPA.

5. Education and Training: Make significant changes to medical, nursing, and paraprofessional curricula by including training for understanding, evaluating, and utilizing predictive models.

Background

Algorithms and models have long been used in health care to assist decision-making. There are well-established frameworks that provide guidance to the development and utilization of these algorithms and models. A well-known example of an algorithm is the Acute Physiology and Chronic Health Examination (APACHE) II score for intensive care unit mortality.1 This model uses a few variables that can be calculated by hand without a computer. A well-known framework that provides guidance for the development and utilization of such algorithms and models is the one proposed jointly by the International Society for Pharmacoeconomics and Outcomes Research (ISPOR) and the Society for Medical Decision Making (SMDM) through their ISPOR-SMDM modeling good research practices task force.2–6

e-HPA shares common features with earlier generation algorithms, models, and frameworks providing guidance to the initial development and implementation of e-HPA. However, the current quantity of input data and the growing sophistication of the algorithms and models in e-HPA are on a different scale than the earlier algorithms and models. The implementation of e-HPA on a wide scale to aid in real-time, point-of-care decision-making brings a new set of challenges and opportunities that are not covered by earlier frameworks. As a result of rapid development, many ethical, legal, regulatory, methodological, and technical challenges are emerging; consequently, the existing frameworks in these areas are not well equipped to provide sufficient guidance for addressing these new challenges.7,8 Thus, there is an urgent need to identify these opportunities and challenges in order to establish a framework to ensure robust development of the field.

To achieve these goals, the Gordon and Betty Moore Foundation (GBMF) provided a grant to Parkland Center for Clinical Innovation (PCCI), a nonprofit research and development organization in Dallas, TX, to assemble a panel of experts with a broad range of expertise to provide a framework for identifying and addressing the opportunities and challenges of e-HPA.

This paper reports the emerging opportunities and challenges identified and the framework (Figure 1) proposed by the panel. The proposed framework builds upon previous frameworks of model development and utilization with a focus on the new opportunities and challenges that are emerging in the era of big data. This framework can serve multiple purposes including, but not limited to the following:

1. Identifying knowledge gaps in the clinical application of predictive analytics and encouraging the research community to focus on filling these gaps;

2. Informing funding agencies to provide targeted funding opportunities for specific areas of e-HPA research that align with their missions; and

3. Initiating a discussion on how to establish the appropriate legal, regulatory, and ethical frameworks for health care organizations, governmental agencies, and other oversight agencies to make use of e-HPA.

Figure 1.

Guiding Framework

Transparency: Why should I trust my care to a computer model?

Regulation & Certification: What’s the right balance between regulation and certification on the one hand, and innovation on the other?

Data Barriers: Can modeling techniques keep pace with the size and complexity of data?

Ethics & Privacy: What should be the right framework to think through some of the ethics issues arising from e-HPA?

Education & Training: Can doctors keep pace with e-HPA?

Methodology

The panel members were selected by GBMF and PCCI staff to ensure broad representation of the field. The final panel includes 17 nationally or internationally recognized experts: from academia (7), private foundations (2), health care delivery systems (3), model developers (3), and governmental agencies (2). Their collective expertise included data information technology, methodology, legal, regulatory, ethics, and health care delivery. Prior to the symposium, GBMF and PCCI staff carried out semistructured, in-depth interviews based on a topic guide with each individual panel member. This allowed for a detailed exploration to identify the key emerging opportunities and challenges, and to finalize the in-person symposium discussion topics. Interviews were audio recorded and transcribed verbatim with participant permission. A variation of content analysis was then used to develop a coding scheme for performing a qualitative description of the themes discussed by interviewees. The final codebook included both inductive and deductive codes that were finalized after reaching a consensus among the three-member research team at PCCI.

As a result, the codebook was used to create a draft framework with three interrelated topics where e HPA differs qualitatively from earlier generations of models and algorithms (Data Barriers; Transparency; and Ethics). Additionally, the framework includes two areas where current supporting infrastructure frameworks are insufficient to address the emerging opportunities and challenges of e-HPA (Regulation and Legal; and Education and Training).

These five focus areas provide a structure for the framework. The PCCI research team’s proposal to build the framework around these five areas was accepted unanimously by the panel. After these individual interviews, a PCCI writing team summarized an initial list of key points for each focus area and circulated it to all panelists prior to the symposium. The symposium began with a session that clarified the definition of e-HPA for all participants, reviewed the purposes and goals of the symposium, and reviewed the initial list of key points for each focus area. Panelists were then divided into five subgroups, each covering a focus area. These subgroups provided recommendations to realize the opportunities and address the challenges of each focus area. After the subgroup sessions, full panel collaboration synthesized the subgroup findings. The research team then reviewed all meeting transcripts and drafted a manuscript. All panelists provided feedback on earlier versions of the manuscript, and both reviewed and approved the final version of the manuscript.

Findings and Recommendations

Data Barriers and Model Development

Recommendation 1: Owners and potential users of high quality, de-identified, actionable, and diverse data sources should develop and implement a mutually agreed upon mechanism so that these data sources can be made available to the scientific community for development and testing of predictive models. Such mechanisms need proper governance as well as formal processes for ensuring collaboration in the development, testing, and validation of predictive models. Appropriate protocols to address data breaches or re-identification of de-identified data are also needed as even a robust de-identification process does not make re-identification impossible.24

High quality and diverse data sets fuel predictive models.9,10 Historically, data used in models have been limited to the data collected during the patient encounter with primary access to this data from hospital EHRs, and laboratory, pharmacy, and administrative data sets.11–13

Comprehensive electronic record and other data sources, such as laboratory, pharmacy, and radiological reports that capture detailed patient data have now attained mainstream use in the United States.14–16 For example, over 9 in 10 (93 percent) hospitals possessed a certified EHR technology in 2013, increasing by 29 percent since 2011.17 In addition to making clinical data routinely available, electronic data sources also have greater vertical and horizontal connectivity (i.e., within and across health care institutions). One example of horizontal connectivity is data exchanges among health care providers via health information exchanges (HIEs). Having an e-HPA framework will provide guidance on the development, implementation, operation, and utilization of such HIEs. New interface capabilities, which allow patients to submit patient generated data via cell phones, personal computers, and personal monitoring devices, provide additional data sources that are collected externally to the health care environment.18–21

Creating new interface capabilities represents a major challenge—and opportunity—in medical informatics and health care management more broadly. For example, patient generated data, such as frequent blood sugar measurements recorded by user devices and transmitted to the health care providers, can provide much richer information than simpler data of such measurements taken during clinic visits, enabling a more timely and accurate diagnosis and treatment.

On the other hand, development of diverse and comprehensive data sets presents significant challenges and would require a concerted effort by stakeholders. For example, collection of patient-reported data requires engagement and commitment from patients. Similarly, obtaining data from social media, health forum websites, and even at-home monitoring technologies, involves coordination with IT developers as well as other stakeholders.

Patient level linkage of a wider array of disparate data sets allows for greater opportunities for data to be combined in a way that could threaten patient privacy.22 Data collection methods from various sources would need to be standardized to improve protection of patient information. As a result, it is crucial to ensure the quality of linked data; usually, data from different sources need to be harmonized, cleaned, and reliably linked before e-HPA models can produce reliable results. For example, patient weight and height need to be recorded in the same units. When data variables are harmonized, such as is the case in many HIEs, providers in these exchanges can exchange information.

To realize such opportunities and address these challenges, the panel believes that expanding the scientific community’s access and ability to share diverse data types with appropriate protections and use agreements could propel an era of impactful predictive modeling.

In addition to making patient-level clinical data accessible to the scientific community, there is a need to explore data capture opportunities in other settings. We need to look beyond administrative, clinical, and physiological variables, and capture and incorporate social, behavioral, patient-reported, patient-monitored, genetic, public health, and environmental data into predictive models.14, 22–24 This could improve the accuracy and scope of predictive models.

Transparency and Model Evaluation

Recommendation 2: Developers of health care predictive analytics should adopt the principle of transparency and establish a consensus process involving key stakeholders to develop standards around validation and transparency of predictive modeling that include the following: (1) defining best modeling practices; (2) ensuring clinical coherence; (3) incorporating diverse data sources; (4) specifying guidelines for validation and unbiased evaluation; and (5) explaining model performance in a language clearly understandable by patients and clinicians.

Besides increased size and complexity of data, modeling techniques that can use such increasingly big and complex data to provide accurate predictions also present potential opportunities and challenges.

With increasing availability of data and computational power, models that make specific predictions around the effectiveness of alternative therapeutic plans for a specific patient may soon be routine. For example, a current heart failure program based on a crude prediction profile reserves the most intensive treatments for patients at highest risk and most likely to benefit,26 but in a new paradigm, a predictive model could recommend specific clinical and care management strategies, such as specific drugs and follow-up periods, and even whether case management would be effective. Moreover, with the growing (and potential) use of genetics information for diagnostics and therapeutics, there are expanding opportunities for personalized medicine that will rely on e-HPA methods.27

On the other hand, these increasingly complex models may make transparency, defined here as “the practice of making key aspects of the models and algorithms available to enable review and unbiased replication of development and testing methods,” more difficult. Moreover, independent evaluations of these models also become more challenging as new modeling technologies are becoming less understandable to nonexperts. For instance, a random forest- or neural-network method using hundreds of variables may be more difficult to interpret than a logistic regression method using 12 variables.28 Publishing an unvalidated model or a model used inappropriately by users may lead to suboptimal results, resulting in skepticism or even backlash against all models—potentially hindering the development of e-HPA.

As the volume and level of detail of patient and other data increase, consideration must be given to the development of additional methodologies that—at first sight—are not directly related to predictive modeling. The major factor behind this need is the problem of bias. Suppose that a health system makes full-scale genomic testing available for a minimal co-pay and that X percent of its members sign up for it. A predictive model for condition Z developed from health system Y data could reach very wrong conclusions (e.g., reverse causation) due to the fact that people who volunteer for such testing may be very different from those who do not choose to volunteer. Other biases could creep in due to nonrandom distribution of incentives. Therefore, encouragement should be given to methodological research in this area. One broad methodological endeavor that should receive attention is that of estimation bias.29 In addition, an existing methodological current—the use of counterfactual methods for comparative effectiveness research, which has a long history30—should continue to be encouraged, but with big data in mind.

To realize these opportunities and address the challenges, we recommend that the information be presented in a way that can be understood and applied by a wide range of users—including clinical informatics experts, providers, and patients. Transparency can increase provider and patient trust in the application of predictive analytics in health care.31,32 Transparency should promote a more active collaborative scientific and vendor community and should allow those that apply e-PHA to select the best models based on their environment and available data. Transparency in health care predictive analytics must be carefully implemented. Instead of a one-size-fits-all approach, a transparency framework should be adaptable to tailoring a predictive model’s prototype, complexity, and users.31

As part of the transparency framework, standards for the evaluation of predictive models need to be set.33 Due to a lack of regulatory framework and the rapid development of health care predictive analytics, the quality of available models varies widely.34–36 Published, peer-reviewed models provide a certain level of quality; however, the quality and extent of assessment of commercially available models is unclear.44 There is a need to specify standards around validation and transparency of predictive modeling to ensure that models are robust and of high quality.33,37

A possible strategy for standardization is to establish best practice guidelines to define and help incentivize best modeling practices, clinical coherence (consistency with clinical practice), standards for incorporating diverse data sources, and appropriate frameworks for validation as well as unbiased evaluation. An example of a transparency reporting guideline is the Consolidated Standards of Reporting Trials (CONSORT) for reporting clinical trial methods.38

We suggest that such standards be developed in consultation using a consensus process that includes the end users of predictive modeling including researchers, clinical informatics experts, clinicians, and patients. In particular, potential end users need nontechnical, “plain English,” information regarding the applicable population and performance characteristics of models. For example, rather than just reporting a single statistic (e.g., the c statistic),39 end users need to know how many patients would have been evaluated and correctly identified at different thresholds in their specific clinical setting. Such information should be made available to these end users to the maximum extent allowable under the current framework of intellectual property protection.

Ethics

Recommendation 3: Users of health care predictive analytics should develop a risk-benefit analysis approach for the use of health care predictive analytics at the individual, organizational, and societal levels to determine the adoption of these models.

Besides opportunities and challenges presented by the increasingly complex data and modeling techniques, the increasing sophisticated utilization of e-HPA models in assisting medical decision-making also presents new opportunities and challenges. For example, e-HPA has been used to identify when care can be escalated or de escalated without significantly compromising care outcomes, allowing for optimal resource allocation. In one example of such utilization, patients predicted to have a high risk of readmission may be targeted to receive more intense care management. So, resources are allocated to the patients that are most likely to benefit.26 While such applications of e-HPA are still emerging, the potential opportunities and benefits are huge and have been discussed elsewhere.26

On the other hand, using e-HPA to inform medical decision-making at the patient- and system levels raises important ethical issues that need to be addressed, especially when such models are used to escalate or de-escalate care for patients.

It is the responsibility of both the individual physician who uses the predictive models and the health care organization that embeds predictive models into their EHRs to ensure that the benefits of using the models outweigh the risks.33 To do so, they need to rigorously assess the benefits and risks of implemented e-HPA in various situations, which includes the actions recommended or triggered by the model’s results. Moreover, the distribution of benefits and risks across the variety of patients to whom the e-PHA is applied needs to be analyzed. It is possible that the model may benefit the population as a whole but be seriously harmful to some subgroups or individuals.

Additionally, developers of EHR with built-in predictive analytics should explicitly communicate the risks and benefits to their clients. Clinical providers and patients using predictive analytics should carefully consider risks and benefits of using or not using predictive models prior to their use. It is possible that, in the future, an institution foregoing the use of predictive models or a physician failing to appropriately use predictive models could be considered outside of practice norms when there is clear evidence that the use of e-HPA in such situations is beneficial.

With regard to consent, for the clinical applications of the predictive analytics (PA) model, the issues are whether use of the PA should be considered a condition of care, similar to the use of a clinical decision support tool, and what type of notification of PA should be given to patients.

Regulation and Certification

Recommendation 4: e-HPA Practitioners should implement a carefully constructed self-regulation and certification framework for e-HPA. The goal of the framework would be to realize the potential of e-HPA by addressing a multitude of concerns—such as the assessment of benefits and risks, transparency, and patient privacy—without stifling innovation. Voluntary organizations may help fulfill this need.

Given these complex issues in data, transparency, and model evaluation, and ethics, establishing the appropriate regulation and certification framework for e-HPA becomes crucial.

Health care predictive analytics offers an unprecedented opportunity and ability to leverage massive amounts of data to improve the health of patients and lower the cost of health care.9,14,40 This opportunity is not, however, without significant risks to patients.15,31 Underperformance of models and negligent operation of predictive analytics by health care organizations or clinicians, for example, are conceivable scenarios in an unregulated market of predictive models.41,42

Given the nascent state of the e-HPA era, the need to balance oversight without stifling innovation cannot be overstated. A strict regulatory framework for e-HPA may discourage developers and research institutions from investigating and developing novel models for use in various health care sectors. Instead, we recommend as a first step, the need for a carefully constructed self-regulation and certification framework within e-HPA.31,43 We define self-regulation as a process steered by key stakeholders, including health care researchers, predictive modelers, mathematicians, IT developers, manufacturers, clinicians, hospital executives, patients and their advocates, social workers and special societies aimed at minimizing patient risk, developing best practices, and setting standards for improving quality of care. Self-regulation may be applied in conjunction with some form of government regulation, or it could fill the vacuum in the absence of government oversight and regulation. Examples of voluntary organizations that may support self-regulation for e-HPA include the Joint Commission on Accreditation, Healthcare and Certification (JCAHO),44 National Association for Healthcare Quality (NAHQ),⁶⁷ National Quality Forum (NQF),45 and American Medical Informatics Association (AMIA).46

Certification standards based on model evaluation and documentation of performance metrics can further mitigate risks imposed on patients.31 Model evaluation and expanded documentation of a broader range of performance metrics could ensure quality application of predictive models in hospitals and other care settings.33,47,48 Model evaluation should include stress testing of models with respect to differing populations, missing and erroneous data, and clarity regarding validation procedures.33,37,49,50 One strategy to achieve this would be to have a third-party provider certify and evaluate models before their use in health care settings.33 Certification would require that the models achieve and maintain certain performance metrics.

Education and Training

Recommendation 5: Medical schools and other professional training institutions should incorporate e-HPA competency into medical, nursing, and paraprofessional curricula and training.

The growing sophistication and performance of predictive models in health care will create new training requirements of the medical professionals.51 An interdisciplinary workforce capable of evaluating and utilizing predictive models will be an essential ingredient in advancing the value of health care predictive analytics.52 Specifically, individuals with expertise in both health care and quantitative modeling techniques and informatics are already highly desired.53 There is also a need to attract individuals with sophisticated mathematical and modeling skill sets into the health care industry. Moreover, expertise from areas operating at the periphery of health care such as in biodesign, biomedical engineering, user experience and design, human factors engineering, and information security will be equally critical.

The panel voiced concern that the current workforce may not have the requisite professional training for an emerging predictive modeling paradigm in health care. Concerns range from clinicians’ and administrators’ poor understanding of concepts in probability, statistics, and heuristics54 to medical, nursing, and paraprofessional curricula and training that do not emphasize the value of accurate data collection and potential for reuse.52 An interdisciplinary trained workforce with a greater understanding of probability and statistics, not just predictive modeling techniques, could alleviate these concerns.55 This would require medical, nursing, and paraprofessional curricula and training to adopt e-HPA as a competency.

Conclusion

As with earlier generations of models and algorithms, e-HPA at the point of care has the potential to lead to significant improvement in the quality, efficiency, and convenience of health care. However, to realize such potential, challenges exist in key technical areas including increasingly big and complex data sources, increasingly complex modeling techniques, increasingly sophisticated ways of using e-HPA to guide medical decision-making, and key supporting infrastructures—including regulation and certification, and education and training. While earlier generations of frameworks are still instructive, the opportunities and challenges are sufficiently different that a new framework is urgently needed to provide guidance to the nascent field.

In this paper, a panel with 17 nationally and internationally known experts in diverse fields proposed a framework that includes developing an oversight strategy and standards for transparency, privacy, risk-benefit analysis, data availability, and health care workforce training and education. While the field of e-HPA is still in its early stages and the framework presented will grow and adapt as the field evolves, the panel believes that the framework presented provided the scaffolding and a starting point upon which to build future development of the field. We call upon the field to update this framework as the field evolves and new opportunities and challenges emerge.

Given the scope of topics covered and the limited space, we have not explored the implications or ramifications of these recommendations in detail. An in-depth discussions of these topics can be found elsewhere.7,8,56 It is the task of health care leaders, e-HPA practitioners, and other stakeholders to ensure an infrastructure that ultimately promotes effective use of predictive analytics to improve patient outcomes, satisfaction, and the value of health care resources.