Electronic health records based phenotyping in next-generation clinical trials: a perspective from the NIH Health Care Systems Collaboratory

Abstract

Widespread sharing of data from electronic health records and patient-reported outcomes can strengthen the national capacity for conducting cost-effective clinical trials and allow research to be embedded within routine care delivery. While pragmatic clinical trials (PCTs) have been performed for decades, they now can draw on rich sources of clinical and operational data that are continuously fed back to inform research and practice. The Health Care Systems Collaboratory program, initiated by the NIH Common Fund in 2012, engages healthcare systems as partners in discussing and promoting activities, tools, and strategies for supporting active participation in PCTs. The NIH Collaboratory consists of seven demonstration projects, and seven problem-specific working group ‘Cores’, aimed at leveraging the data captured in heterogeneous ‘real-world’ environments for research, thereby improving the efficiency, relevance, and generalizability of trials. Here, we introduce the Collaboratory, focusing on its Phenotype, Data Standards, and Data Quality Core, and present early observations from researchers implementing PCTs within large healthcare systems. We also identify gaps in knowledge and present an informatics research agenda that includes identifying methods for the definition and appropriate application of phenotypes in diverse healthcare settings, and methods for validating both the definition and execution of electronic health records based phenotypes.

Introduction

The US healthcare system is poised to significantly enhance the relevance, number, speed, and cost-effectiveness of clinical trials by embedding them directly within the healthcare delivery system. This transformation¹ will be enabled by capabilities offered by electronic health records (EHRs) and patient-reported outcomes (PROs), changes in the organization and delivery of healthcare, and cooperation toward the development of a learning health system^{2 3} in which evidence generated in clinical settings is routinely examined to inform research and practice. Widespread electronic collection of operational and clinical data⁴ have enhanced the potential for pragmatic clinical trials (PCTs), randomized controlled trials designed for broad generalizability, typically using multiple clinical sites and broader eligibility criteria.⁵ In contrast to explanatory trials, for which the goal is to detect biological effects of new treatments, PCTs are designed to support clinical decision-making by evaluating interventions in ‘real-world’ practice conditions.⁶ PCTs therefore recruit participants from heterogeneous practice settings, and pose challenges for reconciling the variation in healthcare operations, widely disparate information systems, and differences in data capture fidelity. The routine implementation of PCTs is a key element in achieving the vision of the learning health system,⁷ but achieving this on a global scale will require innovations, including new ethical frameworks to assess consent and risk,^{8 9} new methodologies to work with observational data, and more effective partnerships among healthcare systems.

Advancing our understanding and ability to conduct PCTs within healthcare systems using innovative approaches is a key focus of the NIH Collaboratory. The use of EHRs to support trial activities, including the identification of patient cohorts with precise clinical attributes, is an important component of this vision and the next generation of clinical trials. The Collaboratory is leveraging previous work in phenotype definition and execution, and adding new use cases and requirements to inform the practice of using EHRs for research, advancing the science for both informatics and evidence-based healthcare. In the following sections, we will describe the NIH Collaboratory and the Phenotype, Data Standards, and Data Quality (PSQ) ‘Core’ working group, including their early experiences with EHR data queries, standards considerations, and data quality activities. We will conclude with a proposed research agenda and suggested future directions.

The NIH Collaboratory

In 2012, The NIH Common Fund initiated the Health Care Systems Collaboratory (https://www.nihcollaboratory.org/) program to engage healthcare systems as partners in efforts to improve research efficiency and relevance to patients, clinicians, and the healthcare community—and in doing so, advance the national capacity for conducting large, cost-effective studies. The Collaboratory is not a traditional research network but rather a participatory forum, developing and integrating practices that allow diverse healthcare systems to participate in clinical research. It also supports the design and rapid execution of high-impact demonstration projects (see online supplementary appendix) addressing various conditions and treatments within the context of PCTs.

These goals are supported by two dovetailing strategies: (1) the reuse of EHR data, already collected during clinical practice and operations, for research purposes (eg, cohort identification, baseline data, outcome data); and (2) the integration of all aspects of research (eg, screening, enrollment, randomization and treatment assignment, protocol adherence, adverse events monitoring, and outcomes assessment) into routine practice workflows and data-capture mechanisms. By harnessing the large amount of data available from clinical practice, investigators will have access to larger, more heterogeneous research populations, presumably allowing them to detect smaller clinical effects, identify and study rare disorders, and produce robust, generalizable results. The stringent inclusion/exclusion criteria inherent in many clinical research protocols, while particularly important for efficacy studies, result in far more homogenous cohorts than are typical in real practice settings. However, the results of research studies embedded in clinical settings can be relevant to wider populations. Importantly, this expanded pool of research participants would potentially be available without the enormous expenses and challenges entailed by current randomized controlled trial screening and sampling methods.

The PSQ core

The Collaboratory's PSQ Core is one of seven ‘Cores’ (table 1) addressing challenges for engaging healthcare systems in PCTs. With representatives from demonstration projects and the Collaboratory Coordinating Center at Duke University, the Cores promote multidisciplinary discussion and collaboration on projects that use healthcare data to support clinical and health services research. The PSQ Core examines demonstration projects’ experiences in order to identify generalizable approaches and best practices for using EHR-based phenotypes—that is, clinical phenomenology of conditions and events—in research applications. In this paper, we use the term EHR-based phenotyping to describe activities and applications that use EHR data exclusively to describe clinical characteristics, events, and service patterns for specific patient populations. This term connotes the context of EHR systems, in which processes related to the care of patients are the fundamental driver behind data collection. We use ‘EHR-based phenotyping’ interchangeably with the broader term ‘phenotyping’, although the latter can include additional data sources (eg, research-driven data collection, patient reported data) collected outside of EHRs for characterizing observable characteristics of disease or genetic expression.

Table 1.

NIH Health Care Systems Collaboratory Core Groups

Core/working group	Key foci and deliverables for year 1
Phenotype, data standards, and data quality	Criteria for assessing utility of phenotypes and data quality from EHRs in PCTs; identification and validation of phenotype definitions for demonstration projects; methods for validating phenotype definitions in multisite trials; ‘Table 1 project’ for standardized clinical/research data reporting
Electronic health records	Technical approaches for extracting information from multiple EHRs and clinical systems for research use; explore existing distributed research networks and open-source tools for distributed queries
Patient-reported outcomes	Reusable and sustainable models for incorporation of patient-reported outcomes and assessments in EHRs and research study management tools
Provider–health systems interactions	Well-catalogued issues and challenges; guidance documents; strategies to minimize disruption and maximize engagement in healthcare systems; strategies for driving behavioral change to implement PCTs and cluster randomized trials (CRTs); updated and generalized HMO Research Network Toolkit resources ; updated CRT toolkit; guide to PCTs for stakeholder organizations
Regulatory and ethics	Fundamentals of patient informed consent for PCTs; fundamentals of determining minimal risk; use of central institutional review boards in PCTs
Biostatistics and study design	Key issues in extracting usable data from EHRs for PCTs; assessing statistical power in CRTs
Stakeholder engagement	Overcoming barriers to conducting PCTs within healthcare delivery systems

EHR, electronic health record; PCT, pragmatic clinical trial.

Early experiences with EHR-based phenotyping

EHR-based phenotyping uses data captured in the delivery of healthcare (typically from within EHRs) to identify individuals or populations (ie, cohorts) with conditions or events relevant to interventional, observational, prospective, and/or retrospective studies. Defining these patient cohorts requires explicit, standardized queries (consisting of logical operators, data fields, and value sets, often using standardized coding systems) that can be applied against different data sources to identify comparable populations. However, due to heterogeneity across data models, providers, and patient populations, designing phenotype definitions is complex and often requires customization for different settings.¹⁰ The validity of selected phenotype definitions and the comparability of patient populations across different healthcare settings has been reported.^10–13 However, the increasing emphasis on patient-centered research¹⁴ and comparative effectiveness will likely dramatically increase the number of needed phenotype definitions, and a standard approach to defining, validating, and identifying appropriate phenotype definitions is desperately needed.

The PSQ Core's year 1 aims include identifying or defining phenotypes that can be executed and replicated in different healthcare settings. The Core will advance the practice of EHR-based phenotyping by defining generalizable criteria for the utility and quality of EHR-based phenotypes for PCTs, and methods for validating phenotype definitions across different disease conditions and organizational contexts. The Core is also exploring designs for an inventory of phenotype definitions, descriptive documentation, and (perhaps) executable code that can enable investigators to identify, evaluate, and reuse definitions to support a broad range of purposes for multisite research. An accessible set of standardized and explicit phenotype definitions can support the aggregation and valid comparison of study outcomes, and stimulate consistency in reporting across studies. This would support the Core's goal to clearly define a set of ubiquitous baseline characteristics (eg, demographics; comorbidities) necessary for standardized trial results reporting across multisite demonstration projects (a current PSQ Core activity called the ‘Table 1 Project’).

Most of the demonstration projects in the Collaboratory are using EHR data and phenotypes to identify patients as potential participants for multisite trials. Some Collaboratory researchers are using phenotypes to longitudinally identify the clinical presence of comorbidities, including diabetes, hyperlipidemia, ischemic heart disease, heart failure, cerebrovascular disease, and renal insufficiency, all of which entail complex characterization and detailed algorithmic logic. Other Collaboratory use cases for EHR-based phenotyping are shown in box 1. These use cases provide a foundation for identifying successful strategies for phenotype selection, execution, and validation in various contexts.

Box 1. Use cases for electronic health record (EHR)-based phenotyping in the NIH Collaboratory.

Use cases

• Identifying patients for recruitment of prospective trials.

• Describing patient cohorts for analysis of existing data for comparative effectiveness or health services research.

• Presenting baseline characteristics or conditions to describe research populations by demographics, clinical features, and comorbidities for clinical trials.

• Presenting primary outcomes to test the trial hypothesis.

• The implementation of supportive tools for providers that are embedded within EHR systems and clinical workflows (eg, routine assessment for suicide risk and provider-directed information to understand epidemiology and possible handling of incidental findings from lumbar imaging studies).

Although not exhaustive, the work of the demonstration projects provides a way to categorize phenotypes in terms of use (eg, eligibility screening, cohort identification, or algorithm for coding independent and dependent variables or intervention); construct of interest (eg, disease indicators, disease progression, service utilization); type of data used (eg, medications, procedures, labs), and whether target data are structured or unstructured. Together, these attributes and intended uses of phenotypes will drive data requirements, acquisition methods, and quality thresholds. The PSQ Core will continue to refine this phenotype classification to support the identification and evaluation of approaches for querying data across systems in ways that are reproducible and will identify comparable patient populations. These efforts will in turn inform metrics of data quality and drive requirements for data standards.

Data standards considerations for the core

In lieu of universally adopted clinical data elements, the Collaboratory demonstration projects are using readily available electronic data sources (eg, encounter-based diagnosis and procedure codes, drug codes, and lab codes and results) for phenotyping, and are in a position to identify and promote clinical data elements that will better support healthcare delivery and secondary research use. Current data are insufficient for characterizing many conditions, especially those not strongly associated with defined treatments or diagnostic procedures. For example: chronic pain can be defined from a series of clinical encounters, leading to complex queries involving multiple time-points and medication histories. Although widely used, the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) as a classification by definition excludes certain conditions, such as rare disorders. In addition, different clinicians might use codes differently (eg, specialty practices with greater diagnostic precision will tend to use greater specificity in coding), and the context of use will likely necessitate local customization of ‘standardized EHR-based phenotype queries’ to address issues emerging from specific features of providers, provider specialization, urban vs rural patient populations, type of EHR, and length of time it has been implemented. Psychiatric and behavioral disorders pose particular challenges because of diagnostic subjectivity and potential reluctance of providers to record diagnoses that may carry a risk of stigmatizing a patient.¹⁵ Preliminary experience in the Collaboratory suggests that using EHRs as a source of patient outcomes for analysis as dependent variables will require rich clinical data, collected consistently and organized to render transparent the subjective issues related to each study site, so that data can be represented in a ‘standard’ fashion for each multisite PCT.

Moving forward, adoption of clinical data elements with explicitly defined value sets will facilitate the use and aggregation of healthcare data from many sites. There are multiple sources for data elements (eg, consensus measures for Phenotypes and exposures (PhenX),¹⁶ the Cancer Data Standards Repository (caDSR),¹⁷ the clinical portion of the LOINC (Logical Observation Identifier Names and Codes) vocabulary,^{18 19} the Patient Reported Outcomes Measurement Information System (PROMIS),²⁰ and the NIH Common Data Elements inventory²¹) which can be adopted for both healthcare delivery and research, but each is ‘context dependent’ and represents a limited number of domains. Ultimately, the Collaboratory will need to promote computable representations for phenotype definitions that are valid (reflect phenomena of interest) and reliable (can be aggregated and compared across institutions). While it may be feasible to develop formal phenotype representations that can be executed across healthcare organizations, the informatics and healthcare communities must first determine whether sharing executable code for specific platforms (eg, Epic) will be prudent or desired by potential consumers.

Emerging discussions surrounding approaches and EHR data requirements for PCTs can advance the development of standards for primary EHR data capture, and in turn spawn novel secondary (research) uses of EHR data if leadership from the various health systems remain engaged. As Meaningful Use advances, the timing is also ripe for guidance on implementation/deployment of solutions that can support both primary and secondary data capture. The Collaboratory can convene multiple stakeholders to identify specific requirements for and approaches to standard data elements and value sets, while also explicitly working to prevent the emergence of a ‘Tower of Babel²²’ among researchers and systems involved in PCTs. Informaticians and policy leaders, supported by the National Library of Medicine and other NIH institutes and centers, can identify sociocultural strategies for promoting dialog and cooperation among professional societies and organizations representing clinical perspectives.

The PSQ Core can also identify and promote strategies to reuse existing data despite inconsistencies and limitations. For instance, the Mini-Sentinel Project, which has demonstrated use of multi-institutional electronic (claims) data for monitoring safety of FDA-regulated medical products on a national scale, uses a distributed-data approach and a common data model to aggregate existing data encoded by standardized coding schema (eg, ICD-9-CM, Healthcare Common Procedure Coding System (HCPCS), Current Procedural Terminology (CPT), and National Drug Codes.²³ The ‘Collaborative Care for Chronic Pain in Primary Care’ demonstration project is using the HMO Research Network (HMORN) Virtual Data Warehouse (VDW) to support aggregation of data from different Kaiser Permanente health plans. The VDW is a distributed data model that has been used to unify heterogeneous data across 17 different EHR systems to support a broad range of research and enable evaluation of clinical interventions and comparative effectiveness research.

The Strategic Health IT Advanced Research Projects’ Secondary Use of Electronic Medical Record Data (SHARPn) project uses common ‘disease-agnostic’ data models that support multiple secondary use cases.²⁴ The SHARPn approach could increase efficiencies by eliminating the need to develop new data models to support specific research questions or use cases, although building such multipurpose clinical data models is a complex task, one yet to be demonstrated on such a grand scale.

We note that substantial development has already occurred. For example, since 2008 the FDA invests approximately $10 million annually in the infrastructure and core operations for Mini-Sentinel. The clinical element models underlying SHARPn required a year of dedicated development, drawing on decades of evolution in the conceptualization of clinical data. These models are now in active development in the Clinical Information Modeling Initiative (CIMI), which includes representatives from healthcare systems, EHR vendors, the US Office of the National Coordinator (ONC), and several countries.²⁵ Similarly, the HMORN VDW model has evolved since 2002 and has supported a number of research networks, including the Cancer Research Network, the Cardiovascular Research Network, and the Mental Health Research Network.

Data quality activities in the core

Although the re-use of data generated during the care of patients to support phenotyping for clinical research offers numerous benefits, concerns about quality and precision linger, given that most therapies have modest effects, so that trials typically target effect sizes of 15–20%.²⁶ The PSQ Core is addressing these issues by surveying the methods used across demonstration projects, identifying overarching methodological considerations, and synthesizing practices in the context of the published literature.

Data quality is most commonly defined as ‘fitness for use’.²⁷ Its assessment, therefore, is use-dependent and requires detailed understanding of data collection features and procedures across multiple dimensions,²⁸ the most pertinent of which (for secondary use) are completeness and accuracy.^{29 30} Data completeness can be measured as the percentage complete of any variable, the presence of all needed data elements, or the percentage present of expected values (ie, case ascertainment).

Data accuracy implies that the EHR data correspond to a concept of research interest. Assessing data accuracy in research typically involves comparison, either of individual values to an independent measurement, validated indicator, or valid (ie, expected or plausible) values^{31 32} (commonly done in clinical trials³¹ and registries²⁹) or of aggregated values to expected, estimated, or ‘gold standard’ values.^{31 33} Evaluating data accuracy through aggregate comparison involves comparing summary data (eg, mean, median) from a research dataset to values obtained from comparable populations. Other aggregate methods use comparisons within a dataset³⁴ (eg, among multiple sites in a multicenter study³³)_, or among subgroups defined by sex or age. The seven Collaboratory demonstration projects use various combinations of these methods.

The PSQ Core will integrate the experience and results from Collaboratory demonstration projects and disseminate findings to inform further discussion on practical, validated methods for measuring data quality, including defining denominators for case ascertainment, and gold standards for accuracy of data extracted from heterogeneous EHR systems. Methods for measuring the quality of phenotypes derived from EHR data undoubtedly will tie back to the concept of ‘fitness for use’ and to the emerging taxonomy of phenotype uses, and will likely include a variety of metrics and thresholds appropriate for various research applications.

A research agenda for EHR-based phenotyping in PCTs

The greatest challenge for effective and reusable EHR-based phenotyping is ensuring consistent collection of relevant data by identifying defined study populations and their features (genetic, biological, psychological, and social), characterizing disease in terms of progression and patient impact, and unambiguously identifying procedures and treatments. But until these data are collected and represented uniformly across all EHR systems, we need methods for understanding the semantics, precision, and limitations of existing EHR data and strategies for meaningful aggregation of data across institutions. This includes exploration of data quality models and metrics related to phenotypes, and theory-driven models for selecting and evaluating potential data sources. The evaluation of potential data sources will be facilitated by exploring and explicitly defining the various ‘clinical’ and ‘administrative’ data constructs that current pragmatic trials use to identify participants. For example, ‘people with suicidal ideation’ and ‘people at risk for suicide attempt’ are clinical constructs for which EHR data can serve as a proxy (eg, patient reported data collected in EHR or ICD-9-CM diagnosis codes V62.84, ‘Suicidal ideation’, 296.3, ‘Major Depressive Disorder Recurrent Episode’, or E950–E958, ‘Definite self-inflicted injury’). In contrast, ‘people initiating dialysis treatment’ is an administrative construct, and EHR data (eg, admission to a dialysis unit, CPT procedure codes for dialysis) are not a proxy for the construct—rather, they define the construct.

EHR data have their own ‘fingerprint’—types of data, levels of structure, patterns of availability, and data quality, all driven by patient care activities, which is a markedly different context than traditional clinical research. EHR data do not always reflect a ‘true’ patient state, but rather interactions between a patient, their environment, and the healthcare system, including provider encounters as well as unrelated business and operations processes and transactions. PSQ Core members support Hripsack's suggestion that the exploration of organizational healthcare data can help us identify these interactions and ultimately understand population characteristics and phenotype performance in different healthcare organizations.³⁵ This information can also help identify data anomalies that do not reflect true differences in patient populations, but rather reflect coding variation across healthcare organizations and regions. Coding variation can result from a variety of factors including changes in coding systems or versions, learning effects as awareness of codes increases over time, organizational culture and policies (eg, reimbursement) that impact coding practices, and different EHR systems and coding interfaces (eg, the display and order of codes presented to users). For example, using data from all state-based morbidity data systems, the Centers for Disease Control demonstrated significant variability across geographic regions and health system in the use of ICD-9 e-codes to indicate cause of injury.³⁶ Therefore, the use of those data (as outcome measures) to evaluate the effectiveness of suicide attempt prevention programs is only feasible in health systems with high rates of e-code recording. Such experience indicates that future reporting standards for PCTs should include not only baseline characteristics of study populations (ie, intervention and control groups), but also baseline characteristics of system characteristics and coding profiles that might highlight coding biases between sites or across time. For example, these characteristics could identify sites or systems that use ICD-9-CM (injury) e-codes less frequently, and these data could help reviewers assess biases in reported studies.

The PSQ Core has also identified the need for computable representations for phenotype definitions that address variations in data collection and representation across different organizations. These representations must address parameters for local customization that will be required to support their execution in heterogeneous settings. Examples include the institution-specific reference ranges for lab values, availability of data as structured elements versus free text, and adoption of ICD-10 and handing of retrospective data in ICD-9-CM.

Although common-model approaches may ultimately realize this goal, they require substantial resources and time to implement. Meanwhile, other solutions offer incremental approaches to coalescing heterogeneous data for research. By applying existing phenotypes (eg, Mini-Sentinel, eMERGE, HMORN) in different settings, we can refine algorithm query logic and build guidance documents so that existing phenotypes can be easily and consistently leveraged by other healthcare organizations. Data on the use of existing algorithms (including performance measures such as sensitivity and positive predictive value, and with institution-specific customizations) can be presented in libraries such as the Phenotype KnowledgeBase (PheKB)³⁷ to inform selection of phenotype algorithms. These usage analyses can also help us establish whether the reuse of complex algorithmic logic and approaches such as ‘high-throughput phenotyping’³⁵ are valid and feasible methods for overcoming heterogeneities in the collection and representation of healthcare data. Standard methods and approaches to validation are also needed. This includes delineating the validation of the face validity (eg, algorithm logic, data sources, and institution-specific customizations) from the discriminant ability of the phenotype. Ensuring face validity requires a sound process for identifying appropriate institution-specific data sources for variables specified in the algorithm, translating the logic to EHR system queries, and ensuring that these queries are consistent with the specifications, intent, and parameters of the phenotype definition (eg, good practice is at least two independent programmers review the code.) Measuring a phenotype's discriminant ability involves calculations (eg, sensitivity, specificity, positive predictive value, negative predictive value) that compare features of cases returned against a gold standard, although the nature of the gold standard is difficult to define across heterogeneous systems. Physician expert review and structured chart abstraction are common approaches, but the methods used have varied across studies and organizations.¹⁰ At present, there is no consensus for how to validate the populations returned from EHR-based phenotyping, including the numbers of patient records that need to be sampled and how best to confirm the diagnosis or phenotype of interest. The Collaboratory demonstration projects are each developing data quality plans that will address the issue of phenotype validation by applying and extending methods used in previous EHR-based phenotyping projects, such as eEMERGE, and also by leveraging other healthcare systems in the Collaboratory as possible testing sites for validation methods.

The NIH Collaboratory can help create new capabilities for evaluating, using, and sharing electronic health data, iteratively defining standards and best practices for a new system of evidence generation, and then broadly disseminating them to the many entities interested in health research. Ultimately, this will yield high-quality, relevant, and generalizable evidence at a faster pace and lower cost, as health systems apply the results of their own investigations.

In the coming year, the Collaboratory demonstration projects and Cores will engage health systems as they leverage existing data to benefit patients, clinicians, and organizations. Through public sessions, such as the Grand Rounds available at https://www.nihcollaboratory.org, they will share lessons learned, suggest best practices, and explore tools, procedural guidelines, and research standards that can advance the conduct of PCTs as part of a learning healthcare system. Broad engagement and varied perspectives will be critical to identifying and implementing solutions, and we encourage participation by Collaboratory partners, clinical researchers, informaticians, and representatives of healthcare organizations and research sponsors with an interest in developing better evidence to inform healthcare decisions and practices.