Apples and Oranges? Considerations for EHR-Based Analyses Aggregating Data From Interventional Clinical Trials and Point-of-Care Encounters in Oncology

The use of real-world evidence—defined by the US Food and Drug Administration (FDA) as “data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources”—is gaining momentum.^1-5 Information systems that were historically siloed, such as pathology, imaging, and genomic records, are becoming increasingly integrated into electronic health record (EHR) systems. This integration facilitates queries of the EHR for patients who meet specified inclusion criteria and allows for data to be aggregated into a form that is suitable for analysis, providing an opportunity to generate real-world evidence.

EHR-based cohorts may include patients who ever enrolled on an interventional clinical trial in conjunction with patients who never enrolled on an interventional clinical trial, and this is not often acknowledged or accounted for in analyses. Stipulations in clinical trial agreements, which may restrict access to data or portions of data (in arbitrary ways), can impact the completeness of the real-world data in ways that would be difficult to understand without specific knowledge of trial participation. Despite the increased integration of many clinical data elements, clinical trial enrollment status is not available by default in many EHR systems. A variable that indicates a patient’s clinical trial enrollment status along with details on restrictions for data access specified by the clinical trial agreement would provide important utility in determining the appropriate analysis of these aggregated data.

Oncology patients enrolled on interventional clinical trials differ systematically from nontrial patients with respect to a number of important factors, such as demographic characteristics and health status.^6-8 For example, oncology trial inclusion criteria often specify a minimum ECOG performance status score,⁹ which measures how a disease impacts a patient’s ability to carry out daily living activities. Alternatively, some trials may enroll patients who are less healthy but have exhausted other treatment options. With respect to demographic characteristics, Black and Latino patients and older patients are underrepresented in oncology trials.^10-16

There is also variation between clinical trial and point-of-care encounters with respect to the reason, frequency, and extent of medical assessments. Patients enrolled on a clinical trial follow a prespecified visit schedule, whereas point-of-care encounters typically indicate a need for medical attention.^17,18 Conversely, in mobile health studies, constant monitoring by mobile health devices may trigger a health care encounter. With respect to the frequency of assessments, in studies estimating progression-free survival, patients following a more frequent scanning schedule will have progression detected earlier compared with patients on a less frequent scanning schedule. Additionally, patients enrolled on a trial are more connected with the healthcare system due to intensive monitoring, with assessments beyond the standard of care.

These differences in patient case mix and in the reason for, frequency of, and extent of medical assessments between patients enrolled versus not enrolled in a clinical trial can lead to violation of modeling assumptions, biased estimates, and inflated type I error rates. Furthermore, because of these differences, specific analytic methods for analyzing EHR data, such as conditioning on the number of visits,^19,20 joint modeling,²¹ reweighting estimators,²² misclassification adjustment,²³ and pseudolikelihood estimation,²⁴ may not be appropriate when aggregating clinical trial and point-of-care encounter data. For example, the number of point-of-care encounters has been used as a proxy for a patient’s health status in EHR-based analyses^19,20; patients with a higher number of visits are likely to be sicker than patients with fewer visits. However, given that clinical trial patients follow a prespecified visit schedule, this relationship no longer holds. If relatively few clinical trial patients are included in an EHR-based cohort, using the number of visits as a proxy for health status may have few implications for statistical inference, but if the cohort includes a high proportion of patients who were on a clinical trial, inference will likely be inaccurate. It is not currently possible to easily assess the likelihood of such violations without a structured data field for clinical trial enrollment status in the EHR.

Capturing clinical trial enrollment status also has utility in addressing FDA real-world evidence guidelines. Current FDA draft guidance on submitting real-world evidence for drugs and biologics indicates that the real-world data sources that were used to derive real-world evidence should be described.²⁵ While clinical trial and point-of-care encounter data are all derived from the EHR, the distinctions between the two types of data outlined above highlight the importance of distinguishing the specific source of data (clinical trial v point-of-care encounter) within the EHR system. Attention to this detail by informaticists and statisticians is one component that can improve the transparency and reproducibility of real-world data analyses.

Apart from evaluating the potential implications of an analysis that aggregates two sources of EHR data, the secondary analysis of clinical trial data requires careful consideration in and of itself.^26,27 Issues of bias and multiplicity arise, highlighting the importance of an analyst being aware of whether an EHR-based cohort includes patients who were enrolled on a clinical trial. For patients who were enrolled on a clinical trial, the analyst may need further information, such as the randomization schema, stratification factors, and the study schedule, to properly conduct and interpret the analysis.²⁸

Without the routine capture of clinical trial enrollment status, formal evaluation of the appropriateness of the established methods for analyzing EHR data, adherence to current FDA draft guidance, and appropriate consideration of the implications of a secondary analysis of clinical trial data are not possible. A deeper investigation into the implications of aggregating clinical trial and point-of-care encounter data with respect to their effect on statistical inference is beyond the scope of this paper; it would depend on the specific research question being addressed using the EHR-based cohort, the proportion of patients in the real-world EHR cohort who were ever enrolled on a clinical trial, the timing of their clinical trial enrollment with respect to the time period of interest, as well as specific characteristics of the clinical trial itself (eg, randomization schema, stratification factors, and visit schedule). Our intention is to highlight the importance of capturing clinical trial enrollment status as a key first step in being able to begin to fully quantify the implications of aggregating clinical trial and point-of-care encounter data.

In conclusion, EHRs are a complex but rich source of real-world data. With the development of EHR data standards and data governance frameworks to generate reproducible processes for data-intensive research, there is an opportunity to leverage custom EHR fields to ensure uniform documentation of clinical trial enrollment status in the EHR.^29,30 Being able to easily access this information will allow investigators to consider the analytic implications of aggregating data from clinical trial and point-of-care encounters, ensure compliance with clinical trial agreements, and sufficiently assess the quality and appropriateness of real-world evidence generated from real-world data.

AUTHORS’ DISCLOSURES OF POTENTIAL CONFLICTS OF INTEREST

The following represents disclosure information provided by authors of this manuscript. All relationships are considered compensated unless otherwise noted. Relationships are self-held unless noted. I = Immediate Family Member, Inst = My Institution. Relationships may not relate to the subject matter of this manuscript. For more information about ASCO’s conflict of interest policy, please refer to www.asco.org/rwc or ascopubs.org/cci/author-center.

Open Payments is a public database containing information reported by companies about payments made to US-licensed physicians (Open Payments).

Jessica A. Lavery

Research Funding: American Association for Cancer Research (AACR) Project Genomics Evidence Neoplasia Information Exchange Biopharma Collaborative (GENIE BPC)

Margaret Callahan

Consulting or Advisory Role: AstraZeneca, Moderna Therapeutics, Merck, Immunocore

Research Funding: Bristol-Myers Squibb

Other Relationships: Clinical Care Options, Potomac Center for Medical Education, Bristol-Myers Squibb

Katherine S. Panageas

Research Funding: American Association for Cancer Research (AACR) Project Genomics Evidence Neoplasia Information Exchange Biopharma Collaborative (GENIE BPC)

Stock and Other Ownership Interests: Catalyst Biotech, Dynavax Tech, Sunesis Pharmaceuticals, Viking Therapeutics

No other potential conflicts of interest were reported.

AUTHOR CONTRIBUTIONS

Conception and design: All authors

Manuscript writing: All authors

Final approval of manuscript: All authors

Accountable for all aspects of the work: All authors