Framework of the Centralized Interactive Phenomics Resource (CIPHER) standard for electronic health data-based phenomics knowledgebase

Abstract

The development of phenotypes using electronic health records is a resource-intensive process. Therefore, the cataloging of phenotype algorithm metadata for reuse is critical to accelerate clinical research. The Department of Veterans Affairs (VA) has developed a standard for phenotype metadata collection which is currently used in the VA phenomics knowledgebase library, CIPHER (Centralized Interactive Phenomics Resource), to capture over 5000 phenotypes. The CIPHER standard improves upon existing phenotype library metadata collection by capturing the context of algorithm development, phenotyping method used, and approach to validation. While the standard was iteratively developed with VA phenomics experts, it is applicable to the capture of phenotypes across healthcare systems. We describe the framework of the CIPHER standard for phenotype metadata collection, the rationale for its development, and its current application to the largest healthcare system in the United States.

INTRODUCTION

Electronic health records (EHRs) are routinely leveraged to generate phenotypes for use in clinical research and healthcare operations. The Department of Veterans Affairs (VA) EHR system supports the largest healthcare network in the United States, consisting of over 1290 facilities and 6 million Veterans receiving care annually, totaling 24 million users in the last 20 years.¹ The VA EHR contains a wide breadth and depth of data from outpatient and inpatient settings, including unique content for the Veteran population such as screening for military environmental exposures. The availability of over 20 years of structured and unstructured data has been a valuable asset for supporting VA research.² Linkage to external data sources from the Department of Defense, Centers for Medicare and Medicaid Services, and National Death Index, as well as internal sources such as clinical trials data, supplement the VA EHR to provide a more complete picture of Veteran health. Knowledge extracted from these rich data sources can also greatly benefit EHR research and downstream clinical studies at large.

Expertise in both phenomics science and the intricacies of VA data are needed to develop EHR-based phenotypes. For example, multiple definitions of binary post-traumatic stress disorder (PTSD) status have been identified using the frequency, source (such as mental health provider) and location (inpatient or outpatient) of PTSD diagnosis codes. Harrington et al³ built upon this work by developing a model to predict the probability of having PTSD, providing a flexible and adaptable definition for future applications. Given the time and resources required to develop phenotypes, the resulting algorithm metadata need to be captured and made available for future use and application, which is rarely done.^4–6 To meet these needs across the national healthcare system, VA has developed a phenomics knowledgebase library, the Centralized Interactive Phenomics Resource (CIPHER), which enables reuse of EHR-based algorithms and increases efficiency and innovation in phenomics (Figure 1).

Figure 1.

CIPHER knowledgebase workflow. The CIPHER knowledgebase intakes phenotype definitions from contributors using the metadata collection standard and reviews content for completeness and clarity before publishing to the knowledgebase for community use. Abbreviations: CIPHER: Centralized Interactive Phenomics Resource; CPT: current procedural terminology; CUI: concept unique identifier; EHR: electronic health records; HCPCS: Healthcare Common Procedure Coding System; ICD: International Classification of Diseases; LOINC: Logical Observation Identifiers Names and Codes; NDC: National Drug Code.

The challenges of phenotyping are not unique to the VA and several libraries have been created to store phenotype definitions. The Phenotype KnowledgeBase (PheKB) hosts phenotype definitions from multiple Electronic MEdical Records and GEnomics (eMERGE) Network sites in the United States.⁷ The HDR UK Phenotype Library is focused on collection of phenotypes from the United Kingdom EHR.⁸ These libraries demonstrate that phenotyping algorithms can be centralized and cataloged for reuse. However, current metadata collection approaches need a more systematic metadata capture that provides richer context for users and improves transportability across health systems. Through VA’s integrated healthcare system, CIPHER has applied and developed standards for metadata collection which set the foundation for the CIPHER phenotype knowledgebase.

The objective of this report is to provide the framework of the CIPHER standard for EHR-based phenotype metadata collection based on our experience through its application in the VA healthcare system.

MATERIALS AND METHODS

The development of the CIPHER standard for phenotype metadata collection was an iterative process. The earliest version of the standard was developed in 2017 to support phenotype collection in the Million Veteran Program (MVP) and this scientific community contributed heavily to CIPHER’s development.⁹ Since then, insights and experience from investigators, data analysts, and other VA stakeholders have been incorporated to refine cataloging standards used in CIPHER. During the development process, existing phenotype libraries, guidelines, and desiderata from the literature were evaluated to understand the current landscape of standards and identify remaining gaps.^7,8,10–13 The following principles were identified during the development process and informed our iterative approach:

Know the audience

The primary audience for reviewing phenotype metadata includes principal investigators, project managers, clinicians, statisticians, and data scientists seeking to leverage existing phenotype definitions. The secondary audience includes stakeholders from healthcare administration and operations such as program managers, center directors, and scientific officers who query and access available phenotypes developed within the healthcare system to aid in policy decisions and guidelines. Development of these standards catered to the primary audience who are well versed in phenomics science but also includes high-level information for all audiences. Phenotype elements collected must be clear, concise, searchable, and interpretable by our primary audience.

Provide context

The scope and purpose of phenotype development must be clearly defined, so that users can determine whether the phenotype algorithm is generalizable for their use case. For example, a definition for diabetes may be created to optimize sensitivity and identify all possible cases of diabetes as an exclusion criterion for a study of new-onset diabetes. This is an informative starting point for a case-control study seeking high specificity, but further refinement of the definition is advised.

Facilitate reproducibility

The standard must provide enough information including data provenance, process, and methods for a user to replicate the phenotype definition. Reproducibility is key to the utility of collected phenotype metadata. It also enables direct comparison of phenotype prevalence in different settings.

Enable adaptability

The standard allows the collection of granular detail to enable reproducibility, but some fields may not be applicable across all phenotypes provided by users. For example, many phenotypes are not formally evaluated for performance, but this is still useful to collect. Additionally, the standard allows the collection of multiple definitions for one phenotype, which captures the nuance between the definitions such as the role of the phenotype in the analysis. Users may then evaluate multiple definitions to determine which one best meets their needs.

RESULTS

The CIPHER standard for phenotype metadata collection consists of 7 domains (Table 1). Each domain consists of standard fields (see Supplementary Material for collection form). Seven of the fields use standard categories for cataloging.

Table 1.

CIPHER phenotype metadata and standards for cataloging

Metadata domain	Contents	Standard for phenotype metadata cataloging
Phenotype identification	Unique phenotype namea	Free text
	Abbreviations and keywords	MeSH term or free text
Algorithm overview	Data classification	Combat related	Health Services and Programs	Medications
		Demographics	Laboratory Tests	Procedures
		Diseases	Lifestyle/Environmental Factors	Vital Signs
		Health Access and Metrics
	Related disease domain	Cardiovascular	Genitourinary	Neurology
		Congenital Anomalies	Geriatric	Obstetrics/Gynecology
		Dental	Hematology	Oncology/Neoplasms
		Dermatology	Infectious Disease	Respiratory
		Endocrine/Metabolic	Injuries/Poisonings	Rheumatology
		ENT/Ophthalmology	Mental/Behavioral Health	Symptoms
		Gastrointestinal	Musculoskeletal	Women’s Health
	Algorithm descriptiona	Free text
	Method useda (phenotyping approach)	Rules based	Machine learning—Unsupervised	Other
		Machine learning—Supervised	Machine learning—Other
		Machine learning—Semi-Supervised
	Population in which phenotype was developed	Free text
	Author and contact information	Free text
	Date created	Date
Acknowledgment and Publication	Publication	Citation and hyperlink
	Acknowledgment	Free text
Algorithm Components	Algorithm component list and directions for usea	ICD-9/10 Codes	Medications
		ICD-9/10 Procedure Codes	Lab tests
		CPT Procedure Codes	Text snippets
		Clinic Stop Codes	Other
	Description of approach and rationale	Free text
Validation	Description of validation	Free text
	Performance metrics	Sensitivity	PPV
		Specificity	AUC
		NPV	Other
Source of Phenotype Data	Data sourcea	Free text
	Role of phenotype in analysis	Primary Outcome/Exposure	Inclusion/Exclusion Requirement	Other
		Secondary Outcome/Exposure	Comorbidity/Covariate
Additional Information	Programming code or public code repository link	Free text
	Tables, figures, slides, and associated files	Images
	Prevalence data	Attachments

^aIndicates required fields.

Abbreviations: AUC: area under the ROC curve; CPT: current procedural terminology; ICD: International Classification of Diseases; MeSH: medical subject headings; NPV: negative predictive value; PPV: positive predictive value.

The domains of the CIPHER standard are described below:

Phenotype identification

Phenotypes are uniquely identified using the convention “Phenotype Name, subtype (Author).” Abbreviations and keywords including Medical Subject Headings terms are collected to facilitate cataloging.

Algorithm overview

This section is intended to provide a high-level summary to the reader of the key components of the phenotype algorithm. The “Data Classification” and “Related Disease Domain” fields are used to catalog phenotype categories. The method used to develop the algorithm is stated so that the user can quickly determine its complexity. The description of the population used to develop the algorithm informs the user whether the algorithm is generalizable to their population of interest. The date of algorithm creation is included to denote version and the author contact is also provided.

Acknowledgment and publication

If the algorithm is affiliated with a published manuscript, PubMed or preprint journal links are collected. In the absence of a citation, the author will provide an acknowledgment so that work may be attributed to the author by future users.

Algorithm components

This domain contains a more descriptive explanation of the data elements used to construct the algorithm and how to use them. The CIPHER standard lists commonly used algorithm components including diagnosis codes, procedure codes, lab tests, and medications, but other data elements may be included as well. The phenotype author provides the code list for the definition and describes any inclusion, exclusion, frequency, or other requirements for each code set. A description of the entire algorithm is provided which details how each code set is used to create the final phenotype definition. The author also provides rationale for the use of the approach if it is not available in a published manuscript.

Validation

This section describes the validation of the phenotype, if performed, and performance metrics. Validation practices may range widely from using chart review as a gold standard, comparing against patient-reported data, or replicating a known association (such as replicating expected results from a genome-wide association study [GWAS]). Standard fields used for reporting performance are listed in Table 1. This section can also be used to capture performance metrics from validation studies applying the algorithm to other cohorts.

Source of phenotype data

The source data for phenotype development is listed, which may include VA data and other linked sources. The role of the phenotype in the analysis is also captured using standard responses and the collection method accommodates description of phenotype use for other cases, such as healthcare operations.

Additional information

We allow the author to share other resources which may provide context and clarity for the user such as prevalence statistics for applied phenotypes, figures, or attached files. This section also contains programming code or a link to a public code repository.

The CIPHER standard has been used to collect phenotype metadata for over 5000 phenotypes from across the VA user community (Figure 2, see Supplementary Material for metadata capture summary). Contributors must share a minimum number of fields for a phenotype to be accepted in the library and to enable replication (Table 1).

Figure 2.

CIPHER platform content. Other disease domains include congenital anomalies, dental, geriatric, mental/behavioral health, neurology, obstetrics/gynecology, rheumatology, symptoms, and women's health. Other data classifications include combat related, health services and programs, lifestyle/environmental factors, procedures and vitals. Phenotypes may fall into more than one disease domain. Abbreviations: CAD: coronary artery disease; COPD: chronic obstructive pulmonary disease; ENT: ear, nose, and throat; NAFLD: non-alcoholic fatty liver disease.

DISCUSSION

The CIPHER phenotype collection standard is an adaptable metadata collection method that enables reproducibility of EHR-based phenotypes. This standard was iteratively developed with VA community members engaged in phenotype development and provides both detailed information on the phenotype algorithm and a high-level picture of the development and validation process.

Our standard builds on the structure of existing phenotype libraries but requires a more systematic collection of metadata from authors. A comparison against the PheKB and HDR UK phenotype library standards shows the gaps that the CIPHER standard aims to fill (Table 2, see Supplementary Material for further description). Several components including overall algorithm description and purpose, phenotyping approach, and validation description are irregularly captured in PheKB and not captured in HDR UK. For example, a PheKB page may describe the positive predictive value of a phenotype, but the validation approach is not described. For many data elements users must read a publication, if available and with sufficient details, to understand the role and method of development for the phenotype. In the case of the PTSD phenotype example described above, the variability of the performance metrics across definitions can partly be attributed to the use of different reference standards used to generate performance metrics including clinician-administered surveys, self-report, and chart review. The CIPHER standard aims to showcase this information directly to expedite the user’s understanding of the phenotype and ensure that these fields are systematically captured.

Table 2.

Comparison of phenotype libraries

	CIPHER	PheKB	HDR UK
Unique phenotype name	✓	✓	✓
Abbreviations and Keywords	✓
Phenotype validation status	✓	✓
Data classification	✓	✓	✓
Related disease domain	✓
Algorithm description	✓	Sometimes captured
Phenotyping method	✓
Population in which phenotype was developed	✓ Free text field	✓ Age, race, gender, and ethnicity fields collected	✓ Sex collected
Author and contact	✓	✓	✓ Author only
Date created	✓	✓	✓
Publication and acknowledgment	✓	✓	✓ Publication only
Algorithm components	✓	✓	✓
Description of approach and rationale	✓	Sometimes captured	Sometimes captured
Validation and performance metrics	✓	Validation approach sometimes captured
Source of data	✓	✓	✓
How phenotype was used	✓
Programming code/pseudocode	✓	✓	✓
Search method	✓ Browse through catalog by multiple categories and searchable	Entire library listed	Several categories of browsing available and searchable

Abbreviations: CIPHER: Centralized Interactive Phenomics Resource; PheKB: Phenotype KnowledgeBase; HDR UK: HDR UK Phenotype Library.

Our approach enables a standardized capture and dissemination of phenotypes developed by the research and clinical operations communities (Figure 1). User feedback has been integrated into the development of the CIPHER standard and ongoing quality review ensures metadata completeness and clarity. The standard has evolved over time and will continue to be updated. While the CIPHER standard has been implemented in the context of the VA EHR, its framework includes standard vocabularies and thus enables the interoperability of the phenotype knowledgebase across various healthcare systems.

The major limitations of the CIPHER standard pertain to its specific use in the VA healthcare system and limited access. The form was developed for use in the VA which may limit its generalizability to other systems or pediatric care populations. While the VA EHR uses standard vocabularies for structured data capture, there are VA-specific data elements that needed to be included in the definition of the standard. The current version of the CIPHER VA phenomics library uses MediaWiki software and is accessible only on the VA internal network. However, CIPHER’s future plans include extending access to the wider phenomics community via a public website. The CIPHER website will be connected to relational databases that support phenotype comparison, reproduction of computable phenotypes, and connection to visualization tools that display relationships between medical concepts. The standardization of phenotype metadata elements support development of this infrastructure, which will provide a significant improvement over currently accessible libraries. The next generation of the library is currently under development and will be accessible at https://phenomics.va.ornl.gov (Figure 3).

Figure 3.

Current CIPHER website landing page. The CIPHER library will be accessible to the public via https://phenomics.va.ornl.gov.

CONCLUSION

The CIPHER standard aims to make EHR-based phenotyping scalable and efficient by enabling reuse and ensuring reproducibility. The standard and its underlying principles for phenotype metadata collection build upon existing phenotype libraries and have been implemented in the VA healthcare system. This framework can be applied to other EHR systems and allows interoperability across various systems. CIPHER plans to expand access to its phenotype library beyond the VA and enable all healthcare researchers to utilize this resource.

FUNDING

This work was supported by the U.S. Department of Veterans Affairs Office of Research and Development. The contents do not represent the views of the U.S. Department of Veterans Affairs or the United States Government.

AUTHOR CONTRIBUTIONS

JH and YH equally contributed to the conceptualization, data curation, methodology, visualization, working draft, and final approval. FF, JG, RS, AG, MMa, and MMu contributed to data curation, methodology, and manuscript editing. AJZ, TC, and KPL contributed to methodology and editing. SBW and JPC contributed to manuscript editing. DRG and MG contributed to conceptualization and methodology. SM and RBR provided program resources, project administration, edits, and final approval. KC led manuscript supervision, contributed to conceptualization, methodology, project administration, visualization, manuscript drafting, editing, and final approval.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.

CONFLICT OF INTEREST STATEMENT

None declared.

DATA AVAILABILITY

The phenotype definition metadata described in this article will be made available on https://phenomics.va.ornl.gov.