A Standardized Guideline for Assessing Extracted Electronic Health Records Cohorts: A Scoping Review

Abstract

Assessing how accurately a cohort extracted from Electronic Health Records (EHR) represents the intended target population, or cohort fitness, is critical but often overlooked in secondary EHR data use. This scoping review aimed to (1) identify guidelines for assessing cohort fitness and (2) determine their thoroughness by examining whether they offer sufficient detail and computable methods for researchers. This scoping review follows the JBI guidance for scoping reviews and is refined based on the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for scoping reviews (PRISMA-ScR) checklists. Searches were performed in Medline, Embase, and Scopus. From 1,904 results, 30 articles and 2 additional references were reviewed. Nine articles (28.13%) include a framework for evaluating cohort fitness but only 5 (15.63%) contain sufficient details and quantitative methodologies. Overall, a more comprehensive guideline that provides best practices for measuring the cohort fitness is still needed.

Introduction

EHR-based cohorts for retrospective study designs are created by extracting the relevant patient data from a database. The term “Electronic Health Records (EHR) cohort” or “EHR-based cohort” typically refers to a group of individuals whose health information is collected and stored electronically within a healthcare system’s EHR system¹ – ³. Cohorts can be thought of as a group of individuals who share a common characteristic or experience and are studied over a period of time to observe outcomes or trends related to that characteristic or experience. The purpose of creating a cohort is to have a dataset that is representative of the target population of the study and provide results that are generalizable⁴,⁵. Patient selection from Electronic Health Records should be data-driven and grounded in sound clinical and scientific reasoning. Although the process of building a cohort for an EHR-based retrospective study is intricate, it is essential to research and warrants careful attention to ensure accurate and meaningful results⁶.

Building a cohort from EHR data is complex, especially when researchers can only work with what’s available in the system. Various disease descriptions can lead to different patient groups, highlighting the importance of standardized phenotype definitions to ensure accuracy and reproducibility⁷,⁸. Inclusion and exclusion criteria are key to shaping the cohort, but the abundance of codes such as ICD, CPT, LOINC, and SNOMED⁹ – ¹⁴ for a single disease can complicate this process⁸,¹⁵. Researchers may combine different codes, but without expert knowledge or clear guidelines, they risk including false positives or negatives. Additionally, balancing cohort specificity and result generalizability is essential. Sample selection based on record completeness can result in bias, excluding patients with less data and potentially over-generalizing findings¹⁶ – ¹⁸.

Creating a cohort representing the study target population is challenging, even for experienced researchers. Ostropolets et al.¹⁹ asked nine research teams to reproduce an observational study, aiming to recreate the target cohort. The results showed the teams were unable to create consistent cohorts, with at least four criteria deviations from the master implementation. The authors noted that eligibility criteria based on phenotypes often require further refinement, leading to investigator-induced errors. Despite using the same tools, individual logic affected cohort creation, suggesting that knowledge of data tables, columns, and elements is insufficient. Researchers need detailed guidelines that aids in creating consistent and valid cohorts.

According to Gatto et al.²⁰, identifying study designs and data that are fit for the research purpose help ensure validity, transparency, and reliability. The fitness of an extracted cohort can be defined as the extent to which the collected data represent the target population or the alignment between cohort characteristics and the intended target population²¹. An unfit cohort has downstream effects on analyses as well as results interpretation, generalizability, and implementation¹⁶. Therefore, ensuring cohort fitness by carefully selecting and validating study cohorts from EHR data is essential for generating reliable and meaningful insights from secondary analyses.

To date, manual chart review is still considered the gold standard for validating a cohort; however, it may not be feasible for the large amount of information available in the EHR system. Most importantly, this method does not answer the question of how researchers can know that the cohort they extract from the EHR system reflects their intended target population, or whether the cohort “fits”. To our knowledge, there does not exist a tool that researchers can use to test the fitness of a cohort. The detailed steps of cohort creation are not generally disclosed by researchers during publication. Therefore, there is no sure way for independent researchers to assess how well the cohorts used in the study are actually suitable for their research questions.

The purpose of this scoping review is to search and evaluate the current state of literature in order to identify any existing guidelines or frameworks that can be used to verify the fitness of an extracted cohort from the EHR. The guidelines, frameworks, or tools should contain sufficient details and methods that are computable quantitatively and that sufficient details are disclosed so that a researcher can implement the guideline. The scoping review also describes aspects of EHR data usage that are already available and identifies any gaps that may still exist. The information from this scoping review will provide a foundation for creating a new and improved guideline that will help researchers assess cohort fitness. For the rest of the paper, the terms guideline and framework will be used interchangeably.

Methods

Database search

The methodological framework described by Arskey and O’Malley²² was used to guide our search and evaluation of the literature. The protocol was developed using the JBI guidance for scoping reviews²³ and refined based on the Preferred Reporting Items for Systematic reviews and Meta-Analyses extension for scoping reviews checklist²⁴ . The completed protocol²⁵ for the scoping review was registered with OSF on November 27, 2023. Final database searches to identify relevant literature were constructed by an experienced health sciences librarian (EW) and conducted in Medline via Ovid, Embase via Ovid, and Scopus on 12/1/2023. Records were imported into the Covidence systematic review software²⁶ for deduplication and management of the screening and review process. Two references from the FDA²⁷ and the European Medicines Agency²⁸ were independently added.

Search terms

In October and November of 2023, preliminary searches were conducted to identify relevant terms and to pilot test approaches to the question. The basic search approach was to combine three concept groups: cohort selection AND electronic health records AND standards. The research team discussed different results sets and validated iterations of the search strategy against articles known to fit these criteria, refining terms to reflect the most relevant aspects of the question. A combination of subject headings and keywords were used to diversify the search methods and increase reliability of retrieval, and the search syntax was optimized for each individual database. Additionally, reference lists of included studies and related reviews were also searched to identify additional potentially eligible studies. Search results were limited to English language and publication year 2013-2023, due to the FDA supporting the use of real world evidence in 2018 and concerns about practices before 2013 being outdated. A list of full search terms and strategies are presented in Table 1.

Table 1.

Full Terms and Strategies for Database Search

Search terms	Number of Articles
Ovid MEDLINE(R) All
1) exp Research/ or exp Data Collection/ or exp Research Design/ or exp Epidemiologic	12,704,103
Methods/ or ((patient* adj2 select*) or assess* or evaluat* or cohort* or (data adj3 (collect* or quality))).mp.
2) limit 1 to (english language and yr=“2013 -Current”)	6,053,814
3) ((routinely adj1 collected) or (real world or noninterventional or non-interventional or postintervention* or post-intervention* or observational or EHR or EHRs or EMR or EMRs or electronic health record* or electronic medical record*)).ti.	92,101
4) (guideline* or standard* or reporting or best practi* or framework* or template* or solution* or propos* or reproducib* or replicat*).ti.	562,516
5) 2 and 3 and 4	1,583
Scopus
( TITLE ( ( routinely W/1 collected ) OR ( “real world” OR noninterventional OR “non- interventional” OR postintervention* OR “post-intervention*” OR observational OR ehr	1,190
OR ehrs OR emr OR emrs OR “electronic health record*” OR “electronic medical record*” ) ) AND TITLE ( ( guideline* OR standard* OR reporting OR “best practi*” OR framework* OR template* OR solution* OR propos* OR reproducib* OR replicat* ) )
AND TITLE-ABS-KEY ( ( patient* W/2 select* ) OR assess* OR evaluat* OR cohort*
OR ( data W/3 ( collect* OR quality ) ) ) ) AND PUBYEAR > 2012 AND PUBYEAR <
2024 AND ( LIMIT-TO ( DOCTYPE , “ar” ) ) AND ( LIMIT-TO ( LANGUAGE ,
“English” ) )
Embase
1) exp Research/ or exp Methodology/ or exp Epidemiology/ or ((patient* adj2 select*) or assess* or evaluat* or cohort* or (data adj3 (collect* or quality))).mp.	17,686,654
2) limit 1 to (english language and yr=“2013 -Current”)	9,531,313
3) ((routinely adj1 collected) or (real world or noninterventional or non-interventional or postintervention* or post-intervention* or observational or EHR or EHRs or EMR or	138,496
EMRs or electronic health record* or electronic medical record*)).ti.

Study Selection

Papers selected in this scoping review were published between the year of 2013 and 2023 and were published by health-related organizations only which may include research institutions, academic institutions, or hospitals. The abstracts should mention the Electronic Medical Records, the Electronic Health Records, or observational data. Papers are excluded if they were not written in the English language. Dissertations and conference proceedings are also excluded. Additionally, clinical trials, randomized control trials as well as grey literature are excluded. Screening for both title/abstract and full-text papers was conducted by two independent reviewers [NS, RJ]. After conducting a review of the included papers on which reviewers had divergent assessments, a consensus was achieved to either exclude them or advance them to full-text screening

Data Extraction

Data extracted from Covidence include the name of the authors, institution affiliations, year of publication, where the study was conducted, objectives of the publication, journal of the publications, how the guideline/framework was created, description of the guideline, the key goal of the guideline, and key findings related to the scoping review question. All extracted data were recorded on an Excel sheet.

Main Topics Covered by Guidelines

The topics identified in the included guidelines are compared to best practices and steps for using healthcare data in studies as recommended in Artificial Intelligence and Machine Learning in Health Care and Medical Sciences: Best Practices and Pitfalls²⁹. Although these steps do not directly provide methods for researchers to evaluate the fitness of the cohort, they describe a series of actions necessary before the data should be analyzed. The steps in the life cycle show that poor handling of data can affect downstream work of modeling as well as the results generated from these models. For this review only the best practices related to data design are of interest. The topics include: Study Design, Data Source Selection, Data Collection, Data Extraction, Data Cleaning, Cohort Formation, Data Quality, Reporting and documentation, Reproducibility, Comprehensive regulatory guideline, and Cohort Fitness

Guideline Evaluation

Even if papers were identified based on search criteria and full-text review, the crucial step in assessing the included frameworks is to judge the alignment or closeness of the guidelines in relation to what the objective of the scoping review is. Based on the paper objective, the desired guidelines, frameworks, or tools must contain enough details that can sufficiently guide the users to understand the recommendations and follow the process as well as allowing for quantitative assessment for cohort fitness. These guidelines will be categorized as “Detailed Quantitative”. Other detailed guidelines that alternatively offer qualitative assessment methods are categorized as “Detailed Qualitative”. Other guidelines that include recommendations without detailed guidance will be categorized as “Brief”. Any other guidelines will be categorized as “Not Applicable”.

Results

The scoping review took place between October 2023 and January 2024. Overall, 4,232 papers and 2 references from other sources were imported for screening. After deduplication 1,904 articles were screened for titles and abstracts, giving 47 articles for full-text review. Articles were excluded for several reasons including: methods are based on machine-learning or artificial intelligence (n=2), insufficient guideline description (n=12), and guideline is still in development (n=1). As a result, 30 articles and 2 additional references were used for data extraction.

Article characteristics/general description of included studies

The included studies were published between the year 2015 and 2023 with most publications occurring in 202320,30–35 (n=7, 21.88%), followed by 2016³⁶ – ⁴¹ (n=6, 18.75%), and 2020³⁰ – ³³ (n=4, 15.5%). Most studies^{20,27,35,36,38,39,42–51}(n=16, 50.0%) are identified as frameworks by the author. The remaining sources were identified as checklists⁵²,⁵³ (n=2; 6.25%), reporting guidelines³²,⁵⁴ (n=2; 6.25%), templates⁵⁵,⁵⁶ (n=2, 6.25%), while the rest are mixtures of different types of guideline documents and tools. The included references came from the United States20,27,30,33,35–38,38– 43,45,47,48,51,55,56 (n=20; 62.50%), the United Kingdom^34,46,54,57 (n=4; 12.5%), Canada⁵²,⁵³ (n=2; 6.25%), Switzerland³² (n=1; 3%), South Korea⁴⁹ (n=1; 3.13%), and collaboration between multiple nations²⁸,⁵⁰,⁵⁶(n=3; 9.38%). Five articles^{20,41,44,47,48} were published in Clinical Pharmacology and Therapeutics (15.63%), four papers^36,38,39,45 came from eGEMs (12.50%). A total of three papers⁵²,⁵⁴,⁵⁵were published in BMJ and two papers³⁰,³⁴ were published in JAMIA. One paper was published in each of the following journals: BMC Pediatrics⁴⁰, Pharmacoepidemiology and Drug Safety⁵⁶, BioData Mining⁵⁷, American Psychiatric Association⁴³, International Journal of Medical Informatics⁴², Annals of Internal Medicine⁵⁸, Journal of Clinical and Translational Science⁴⁶, Drug Safety³³, Lancet Digital Health⁵⁰, PLOS Medicine⁵³, Annals of Oncology³², Therapeutic Innovation and Regulatory Science⁵¹, Health Services and Outcomes Research Methodology³¹, Journal of Biomedical Informatics³⁷, and Information Systems³⁵. Non-journal sources include the European Medicines Agency²⁸ and the U.S. Food and Drug Administration²⁷.

Topics Covered by Guidelines

While only 9 of 32 papers directly addressed cohort fitness, all included papers contribute to using EHR data effectively. These topics were extracted based on the objectives of the frameworks stated in each paper. The most common topics after cohort fitness were data quality, reporting/documentation, and cohort formation (N=8; 25.0% each), followed by data source selection (N=7; 21.88%). Other topics included data extraction (N=6; 18.75%), reproducibility (N=5; 15.63%), study design (N=3; 9.38%), data cleaning (N=3; 9.38%), comprehensive regulation (N=3; 9.38%), and data collection (N=1; 3.13%). These guidelines cover various aspects of EHR data use, from selection and extraction to reporting and reproducibility. Table 2 represents the main topic areas directly covered.

Table 2:

Guidelines and Topics Addressed.

Guideline	SDa	SSb	Dcoc	Dexd	Dcle	CFf	DQg	RDh	RPi	CGj	Cfitk
ACE-IT33		X				X					X
ATRAcTR31		X					X
Callahan 202058								X			X
Castillo 201543	X	X		X
CODE-EHR50										X
CONSORT-ROUTINE52								X
DAQCORD 46		X					X	X
DataGauge45				X	X	X	X				X
Denaxas 201757									X
Dziadkowiec 201638					X		X
EnCePP (rev 11)28										X
ESMO-GROW32								X
FDA’s RWE Framework27										X
Generalizability Score42						X					X
GIST 2.037				X		X					X
Graul 202334						X
Han 202249			X			X					X
Haneuse 201639						X
HARPER56								X	X
Kahn 201636							X
KEEPER30				X							X
Knake 201640				X
Miao 202335				X	X	X	X				X
MVET criteria41									X
PICOTS51		X
RECORD53								X
RECORD-PE54								X
SPACE47	X								X
SPIFD48		X					X
SPIFD220	X	X					X
STaRT-RWE55								X	X
The Certainty Framework44											X

^aSD = Study Design, ^bSS = Data Source Selection, ^cDCo = Data Collection, ^dDEx = Data Extraction, ^eDCl = Data Cleaning, ^fCF = Cohort Formation, ^gDQ = Data Quality, ^hRD = Reporting and documentation, ⁱRP = Reproducibility, ^jCG = Comprehensive regulatory guideline, ^kCFit = Cohort Fitness

Study design guidelines included in this review provide checklists emphasizing design structures, research objectives, and methodological rigor. Castillo et al.⁴³, SPACE⁴⁷, and SPIDF2²⁰ offer outlines and checklists that focus on design structures, research objectives, and methodological rigor assessment. Three additional frameworks offer a more comprehensive approach for regulatory guidelines for study designs. Code-EHR⁵⁰ provides best practices for minimum information in research, focusing on documentation and reporting. The EnCePP²⁸ and the FDA²⁷ developed by government organizations offer a broad perspective. While these comprehensive guidelines cover much of the life cycle, they lack detailed methodologies for each step of the cycle.

Frameworks for data source selection emphasize the relevance and quality of sources, with several studies discussing frameworks for data extraction as either standalone processes or within broader frameworks. For example, PICOTS⁵¹ aids researchers in evaluating the suitability of data sources for specific research aims, while DAQCORD46(p20) provides guidance on identifying appropriate sources and addressing ethical concerns in data acquisition. Han et al.⁴⁹ developed a systematic framework for real-world data (RWD) collection within the Korean health system, specifically targeting cancer drug safety and effectiveness. A framework by Knake et al.⁴⁰ promotes consistency in data extraction, though the framework did not provide specific methodologies. The KEEPER Profile³⁰ assists researchers by offering a general workflow for data extraction, starting with the identification of phenotypic information, followed by data standardization and algorithmic extraction. Dziadkowiec et al.³⁸ focus their entire framework on data cleaning, developing strategies to address identified quality issues, correct errors, review inconsistencies, and perform data imputations. Haneuse et al.³⁹ systematically address selection bias by examining missing or available data and providing guidance on patient selection. Additionally, frameworks such as DataGauge⁴⁵, ATRAcTR³¹, and Dziadkowiec et al.³⁸ evaluate extracted data against specific quality aspects to evaluate both data source fitness and study design.

Reporting and documentation frameworks are designed to ensure consistent and detailed methodologies, thereby enhancing research reproducibility and generalizability. Reproducibility is a central concept driving the development of many of these frameworks. For instance, a framework by Callahan et al.⁵⁸ includes action items for reporting, while others provide standards that researchers should follow when publishing their work^32,46,52,54. The START-RWE⁵⁵ template is specifically designed to improve transparency and consistency, while enhancing the credibility and reproducibility of RWE through transparency. HARPER⁵⁶ serves as a protocol template offering standardized operational details, context, and rationale. The MVET criteria⁴¹ guides researchers in using real-world data (RWD) for decision-making, emphasizing the importance of reproducibility for replication and robustness checks.

Cohort Fitness Guideline Evaluation

Table 3 represents the summary of all of the included papers. Out of the 9 papers that address cohort fitness, 5 papers (15.63%) offer quantitative methodologies and contain enough details that can sufficiently guide the users to understand the recommendations and follow the process. Three papers (9.38%) also offered sufficient detail that researchers can implement; however, the assessment methodologies are qualitative. One paper (3.13%) offers brief recommendations without clear steps on how users should approach cohort fitness assessment. More than half of the included papers do not at all describe the kind of framework we had hoped to find (71.88%) as they do not address the topic of cohort fitness in any way or simply present the topic of cohort fitness without offering any practices, recommendations, or methodologies.

Table 3.

Guideline Evaluation Categories.

Categories and Summary of Methods		N (%)
Detailed Quantitative | Quantitative methodologies for cohort fitness assessment are described in sufficient detail that researchers can implement. DataGauge34 An iterative process to systematically implement quality assessment of repurposed clinical data. The tool relies on concepts from Wang and Strong’s classification of data quality dimensions48 and Borek et al.’s classification of data quality assessment methods49. Because the data quality analysis is dependent on the predeveloped data needs model specific to the analysis questions, the tool allows users to evaluate the data quality requirements that align with their research questions. DataGauge also calculates the percentage of compliance for each data quality aspect and flags all data items that do not comply with the criteria. The total number of non-flagged patients is also calculated. Generalizability Score31 The Generalizability Score represents how well the cohort reflects the “general population” in the context of a clinical trial. The tool uses computer algorithms to analyze the characteristics of trial participants and compare them with the potential target population. For each characteristic, the algorithm produces a ratio that represents likeness or generalizability. Han 202238 A data collection framework that includes reliability analysis for collected data. The authors presented that consistency between different cohorts reveals how data are collected, which includes how investigators describe their data elements and data element definitions. Using concordance rates between previously collected data and data collected by the independent investigators, the reliability and consistency of the cohorts can be assessed.	5	(15.63%)
Miao 202324 A data preparation framework where the evaluation of extracted EHR data is done concurrently with data cleaning and quality assessment. The framework uses data quality aspects as a foundation and focuses on the quality of the data and suitability of individual variables. During variable domain assessment, the framework allows users to calculate various statistical metrics such as the mean, median, and standard deviation in order to create visualizations for each variable that can be used to compare to external data or reviewed by experts. GIST 2.026 GIST 2.0 quantifies population representativeness, focusing on generalizability. The metric uses the weighted scoring system to assign importance to traits based on their relevance to the research question. It also allows users to calculate clinical significance of each eligibility criteria through a trial-specific significance scale where greater stringency correlates to higher significance.
Detailed Qualitative | Qualitative methodologies for cohort fitness assessment are described in sufficient detail that researchers can implement ACE-IT22 The ACE-It tool was designed to qualitatively assess electronic medical records or claims-based algorithms in relation to a specific decision context through a series of questions. The Certainty Framework33 The framework provides level-based assessment for determining the degree of certainty in which the data source and algorithms derived from real-world data are reliable. The framework presents that certainty of the data can be categorized as either “optimal”, “sufficient”, or “probable” depending on the importance of study variables and intended use. Users can use the framework’s recommendations for each category to determine the appropriate data processing techniques. KEEPER19 The KEEPER profile was designed for phenotype evaluation in replacement of manual chart review. The profile was developed to offer a standardized solution for extracting and evaluating data elements in relevance to the steps clinicians follow when diagnosing a patient. The extracted data elements correspond to common disease concepts or ideas such as symptoms, family history, or vital signs. In the background, the profile translates the symptoms into condition codes. The output contains data elements defined by the user along with their corresponding phenotype definitions and specific codes.	3	(9.38%)

Discussion

The goal of this scoping review is to identify current frameworks that offer quantitative assessment methodologies that are in sufficient detail in which users can implement. While some included guidelines in this scoping review offer quantitative methods that could be adapted to evaluate extracted EHR-based cohorts, they were originally designed for other purposes. At the time of writing this scoping review, there are no guidelines that were designed specifically for the purpose of obtaining a quantitative measure that allows researchers to know if their cohort fits their research objectives. Therefore, a new guideline is needed in order to fulfill this gap. The design of a new guideline should also consider the following characteristics:

• Flexibility and adaptability. Because health research and health data can vary greatly depending on factors such as the disease of interest, the setting of study, or data collection habits, guidelines should be flexible enough that the researchers can use with ease. Guidelines that are too restrictive may be too difficult to use and may add more burden to users.

• Clarity. Many health data users may not be adequately trained to use complex programs or tools; therefore, guidelines should be straightforward and have clear instructions to avoid misuse. Clear instructions should specify when to use the guideline in the data life cycle. For instance, a data extraction tool should be used after the specification of cohort definition and before cohort evaluation, while reporting guidelines should be applied after analytics validation but before publication preparation.

• Updatability. Due to the ever changing nature of the disease and the structure of the EHR system, changes over time such as the addition of new diseases, codes, procedures, or medications have to be considered during the development of a guideline.

To our knowledge, this is the first attempt to identify detailed guidelines for assessing the fitness of extracted EHR cohorts. This review provides a foundation for developing improved guidelines for evaluating cohort quality. We conducted a broad search across various health and non-health databases to provide diverse perspectives. Articles were restricted to those published in English from 2013 to 2023 to capture recent developments while avoiding outdated information. Key limitations include the exclusion of articles published in other languages, methodologies predating 2013, and new approaches from 2024. Guidelines for unstructured data and those based on machine learning or artificial intelligence were also excluded. While acknowledging the growing shift toward automation, this review prioritizes basic methodologies, recognizing that many EHR users lack extensive backgrounds in machine learning. These gaps highlight opportunities for future research to address unstructured data and automated processes.

Conclusion

This scoping review aimed to identify frameworks, guidelines, or best practices for assessing the fitness of extracted EHR cohorts and their alignment with the desired population. Of the 32 articles reviewed, only 9 provided sufficient detail that researchers can implement. However, only 5 papers provided methodologies that allow for quantitative computations. Although these guidelines were not originally designed for cohort fitness evaluation, they can be adapted to indirectly assess the fitness of the extracted EHR-cohorts. Additionally, existing guidelines, while covering various aspects of EHR data use, still struggle with flexibility, adaptability, clarity, and updatability. To effectively assess cohort fitness, a more comprehensive framework that addresses these gaps and meets all necessary criteria is required.

Figures & Table

Figure 1.

Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Flowchart.