Systematic Evaluation of Data and Trial Fitness for Oncology Trial Emulation: Empirical Findings from the CARE Initiative

Abstract

The Coalition to Advance Real‐World Evidence through Randomized Controlled Trial Emulation (CARE) Initiative seeks to advance understanding of when real‐world data (RWD) can generate valid treatment effectiveness estimates by emulating completed oncology randomized controlled trials (RCTs). A prerequisite for meaningful RCT emulation insights is the identification and use of RWD with sufficient fitness to satisfy RCT‐specific data elements. We conducted a systematic, multi‐stage feasibility assessment of six commercially available US electronic health record‐based RWD sources across 23 candidate oncology RCTs. Each potential RCT‐RWD combination was first screened for availability of the RCT indication, outcomes, and sample size ≥ 1.5‐times enrollment for each trial arm. Combinations passing this screen underwent more detailed evaluation of RCT design elements including eligibility criteria, outcomes, and potential confounders. Each data element was rated with respect to availability, missingness, and validity. Final feasibility determination was informed by ratings of essential element capture and refined sample size estimates. Of 54 candidate RCT‐RWD combinations assessed, nine advanced to detailed feasibility assessment and three were selected for emulation protocol development. Fit‐for‐emulation constraints included complex eligibility criteria, biomarker requirements, performance status requirements, and outcome ascertainment. These findings highlight the importance of systematic feasibility evaluation before conducting emulations and may inform data selection for future RWD studies in oncology. Data fitness for oncology RCT emulation could be improved by linking high‐quality, oncology‐specific RWD sources to broader EHR and claims data sources or through customized data abstraction.

Study Highlights.

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Target trial study design framework and structured processes for identifying fit‐for‐purpose data are important tools to guide observational research.

• WHAT QUESTION DID THIS STUDY ADDRESS?

What modifications to fit‐for‐purpose identification processes are required for oncology RCT emulation studies?

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

Specific data feasibility considerations are required when the goal of a real‐world study is to mirror an RCT as closely as possible, and these may differ from considerations needed to identify a data source for evaluating real‐world comparative effectiveness. Lack of information on non‐cancer diagnoses and treatments in data sources with rich oncology data limits our ability to emulate clinical trial eligibility criteria and control for confounding.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

The same RWD source may not be fit‐for‐emulation of an RCT but could be fit‐for‐purpose for other types of RWD studies; differences between the goals of emulation and real‐world inference must be top of mind when choosing a real‐world data source.

Randomized controlled trials (RCTs) are the gold standard for demonstrating the efficacy and safety of biomedical products. However, RCTs in oncology face well‐recognized practical and ethical barriers, including enrollment difficulties due to small populations and restrictive eligibility criteria ¹ , ² , ³ ; high rates of treatment crossover or study dropout ⁴ , ⁵ ; rapid evolution of standards of care and off‐label use. ⁶ , ⁷ Clinical and regulatory decision makers increasingly recognize the potential for non‐interventional studies using healthcare data generated during routine clinical practice, that is, real‐world data (RWD), to produce complementary evidence about the effectiveness and safety of cancer treatments. ⁸ , ⁹ , ¹⁰ Real‐world evidence (RWE) complements RCTs in a variety of ways, such as providing context to support the interpretation of single‐arm trials, generating new hypotheses, producing insights more quickly and efficiently, and understanding treatment effects in broader patient populations under real‐world, non‐protocolized clinical care patterns and as measured by longer term outcomes. ¹¹ , ¹² , ¹³ , ¹⁴ At the same time, causal inference from non‐interventional studies using RWD may be limited due to threats to internal validity, such as uncontrolled confounding due to lack of randomization and measurement error arising from varying quality and completeness of data.

The Coalition to Advance Real‐World Evidence through Randomized Controlled Trial Emulation (CARE) Initiative was established to inform oncology real‐world study best practices by systematically emulating completed oncology RCTs using RWD and evaluating concordance ¹³ with RCT results as the gold standard. ¹⁵ , ¹⁶ Trial emulations replicate trial design features using RWD to generate insight into the circumstances under which observational methods can approximate RCT causal effect measures. Meaningful methodological insight from emulation therefore depends on the availability of data capable of accurately operationalizing trial‐specific populations, exposures, outcomes, and confounders.

In published guidance, the United States (U.S.) Food and Drug Administration (FDA) emphasizes that selecting reliable and relevant fit‐for‐purpose data is “critical for making appropriate causal inferences” from non‐interventional studies. ¹⁷ Several frameworks operationalizing these and other guidelines for RWD selection have been published, ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² including oncology‐specific guidance. ²³ , ²⁴ , ²⁵ Researchers seeking fit‐for‐purpose RWD for oncology studies face particular challenges due to the need for specific clinical data elements such as tumor histology, disease stage, ECOG performance status, biomarker status, complex treatment regimens, multiple lines of therapy, drug toxicity, and sites of metastasis. Real‐world oncology RCT emulation additionally requires measures capturing extensive trial eligibility criteria and comparable clinical study endpoints. ²⁶ A recent review of published oncology RCT emulations found that few provided details on whether or how RWD sources were assessed for fitness or discussed why a given source was ultimately selected for the study. ²⁷

Rigorous and transparent evaluation to identify fit‐for‐emulation RWD is foundational to the interpretation of any RCT emulation study and determines whether the results will generate novel insights into the impact of data fitness, study design, and analytic methods on the validity of RWE. Because each RCT design incorporates distinct study populations, exposures, and outcomes, data fitness for trial emulation is inherently trial‐specific. The CARE Initiative study team therefore undertook a systematic assessment of six commercially available data sources to assess their feasibility for emulating oncology RCTs across diverse trial designs. The objective of this effort was to characterize oncology RCT emulation feasibility across these multiple combinations, describe how each step of this data feasibility process was applied to candidate RCTs and available datasets for the CARE Initiative, and describe patterns and key drivers of overall ratings that highlight the strengths and weaknesses of available RWD sources for oncology emulation. The findings provide empirical insight into the substantial data demands imposed by oncology RCT designs and inform best practices for transparent data fitness assessment in emulation research.

METHODS

Selection of candidate oncology RCTs for emulation

We searched CenterWatch ¹⁹ and FDA ²⁰ repositories to identify candidate RCTs for emulation by targeting completed trials that evaluated the efficacy of oncology drugs, were conducted during 2015–2020, led to an FDA drug approval, included an active comparator arm, and, in anticipation of sample size requirements, involved commonly treated tumor types (breast, colorectal, non‐small cell lung, pancreatic, prostate, renal or urothelial cancer; acute myeloid or chronic lymphocytic leukemia; follicular lymphoma). Trials with design features that would be difficult to emulate in any RWD source (e.g., very recent drug approvals or use of a new biomarker not routinely assessed in real‐world practice) were excluded. The remaining qualifying RCTs proceeded to data feasibility assessment.

Feasibility assessment of candidate RCT emulations in available RWD sources

Six US electronic healthcare‐based RWD sources, provided by data partners participating as non‐voting members of the CARE Steering Committee, were considered as candidate data sources for each of the qualifying trials. ¹³ These included ConcertAI, COTA, Loopback, Tempus, TriNetx, and XCures. Every RCT‐RWD source combination was considered for a potential emulation study. Given the large number of RCT‐RWD combinations under consideration, we approached feasibility in a stepwise manner for efficiency. The initial data feasibility screen (Screen 1) assessed whether the indication and treatments evaluated in each trial were identifiable in each eligible data source. All RCT‐RWD combinations remaining after Screen 1 proceeded to the next screen.

Screen 2 filtered out RWD‐RCT combinations unlikely to meet the emulation study target sample size. Given the emulation objective, Screen 2 evaluated the number of patients with both the indication and the RCT treatments of interest. A minimum of 1.5 times the number of patients in each RCT arm was required in anticipation of further attrition once additional inclusion/exclusion criteria were applied. The indication and treatment of interest were defined using tumor site and line of therapy variables from curated data sources or using relevant diagnosis and drug identifier codes from claims/EHR data sources. Screen 2 was intentionally broad to facilitate rapid, efficient, and objective screening of the RCT‐RWD pairs without the analytic burden of operationalizing numerous, complex inclusion/exclusion criteria.

Data source‐RCT combinations remaining after Screen 2 entered a more detailed assessment of study data elements based on the Structured Process to Identify Fit‐For‐Purpose Data (SPIFD2) process, with a priori modifications to explicitly address specific data requirements for emulation. ²⁰ , ²¹ Briefly, SPIFD2 facilitates a clear articulation of a research question and objectives, and outlines a step‐by‐step process for systematic and transparent evaluation, documentation, and selection of candidate RWD sources based on capability to operationalize the design elements of a real‐world study designed as an emulation of a hypothetical target trial. ²⁰ , ²¹ To identify minimum data elements, we first extracted pertinent information about candidate RCT study designs from the published article, including the research question, primary objectives, treatment regimens, median follow‐up time, enrolled sample size, inclusion and exclusion criteria, and outcomes. We identified potential confounders based on patient characteristics presented in the trial publication and subject matter expertise. Given the application to RCT emulation, we a priori identified RCT elements that were inherently infeasible to emulate in RWD. This included elements that are not typically assessed by health care providers in the course of routine clinical care (measurable disease according to the Response Evaluation Criteria in Solid Tumors (RECIST), agreement to provide tumor sample, and life‐threatening extensive or advanced disease) ²⁸ and preference for overlapping study time periods. These criteria were designated as “not applicable” for the feasibility assessments, since requiring them would rule out all emulations.

Each retained applicable design element was then ranked as either “essential” or “supplemental,” depending on whether the element was considered critical for a valid emulation as described in SPIFD2. Elements required to identify the indicated population, treatments of interest, and outcomes in a dataset were considered essential. For example, a measure of Eastern Cooperative Oncology Group (ECOG) performance status was considered essential given the strong relationship between ECOG and oncology study outcomes (e.g., survival). In contrast, RCT exclusion criteria based on previous infections or adverse reactions to other drugs were classified as supplemental as these criteria are often intended to protect patients at elevated risk of serious adverse events from exposure to an experimental treatment, rather than being critical to study validity.

Remaining RCT‐RWD combinations underwent detailed assessment of essential and supplemental design elements with respect to availability, completeness, and validity. Additionally, sample sizes were further refined in consideration of age requirements and key exclusion criteria. Maximum follow‐up time was determined.

Availability was assessed as the existence of variables required to operationalize data elements such as treatment regimens, for example, presence of generic or brand drug names or National Drug Codes and dates of prescription and/or administration. Completeness was assessed as the degree of known missingness for each element with a defined denominator. For example, completeness of performance status was measured as the proportion of patients for whom a non‐missing performance status was present in the data, with the lack of a recorded value interpreted as a true unknown since this measure theoretically has a value for all oncology patients (i.e., there is no “absence” of performance status). Conversely, missingness could not be determined for the presence of other, non‐cancer medical conditions since missing values could indicate the absence of a condition or unknown status. Validity was assessed based on variable curation status or external validation of variable or algorithm. Maximum potential follow‐up time in each data source was estimated as the time between the earliest observed diagnosis date for the study indication and the dataset end date and was assessed relative to the duration of follow‐up for the RCT endpoints. Preliminary sample size estimates from Screen 2 were refined by restricting to adult patients and considering only treatments received in the relevant line of therapy.

Similar to the original SPIFD2 process, for each RWD source, we then assigned ratings to each data element with respect to availability, completeness, and validity. Ratings ranged from a low of one, indicating that minimum data requirements were not met, to a high of five, indicating that nearly all data requirements needed to operationalize a given element were met (Table 1 ). At least two study team members independently rated elements for a given emulation. Steering Committee members provided subject matter expertise when requested. Interrater agreement and adjudication metrics were not calculated, as study team consensus was ultimately required for the final ratings of each study design element to ensure consistency across assessments. An overall rating summarizing the degree to which an RCT could be emulated in a given RWD source was then assigned to each RCT‐RWD source combination based primarily on the ratings of essential design elements and sample size. Supplemental elements did not count directly towards the rating but were considered qualitatively. Feasibility assessment results with anonymized references to data sources were provided to the CARE Steering Committee to facilitate selection of the final RCT‐RWD source combinations for emulation by consensus.

Table 1.

Criteria for scoring data elements

Overall rating		Components of overall rating
Rating	Interpretation	Availability	Missingness	Validity
1	Data Requirements are not met	No	> 60%	Not curated or externally validated
2	Data requirements met with substantial limitations	Yes	> 50–60%	Not curated or externally validated
3	Some data requirements are met/requirements are partially met	Yes	> 40–50%	Not curated or externally validated
4	Data requirements nearly all met	Yes	> 30–40%	Not curated or externally validated
5	All or nearly all data requirements are met/data requirement met	Yes	> 30%	Curated and/or externally validated

No human experimentation was conducted for this study, which relies on information from published clinical trials.

RESULTS

Selection of candidate oncology RCTs for emulation

We identified 23 RCTs conducted from 2015 to 2020 that included patients with a common oncology indication (breast, colorectal, non‐small‐cell lung, pancreatic, prostate, renal, or urothelial cancer; acute myeloid or chronic lymphocytic leukemia; follicular lymphoma), used an active comparator, and led to an oncology therapy approval by the FDA (Figure 1 ). Three RCTs that supported very recent drug approvals or that involved assessment of a novel biomarker were deemed infeasible because few, if any, patients of interest would be expected to be identified in any RWD source.

Figure 1.

Overview of the selection process for CARE initiative oncology trial emulations.

Feasibility assessment of candidate RCT emulations in available RWD sources

Based on data partner information about the broad indications captured in the six available RWD sources, 54 potential RCT‐RWD source combinations were identified for data feasibility assessment. Forty‐four of these (including all potential studies evaluated in Data Sources 5 and 6) did not progress to detailed assessment because the specific RCT indication and/or outcome was not available in the dataset or too few patients with the indication of interest initiated the RCT treatment or comparator therapy (< 1.5‐times the RCT sample size for each arm). Prior to full feasibility assessment, one RCT‐RWD source combination (KEYNOTE‐189, Data Source 1) was selected for a pilot emulation based on adequate outcome capture, potential sample size, and complexity of RCT to pilot methodologic and analytic approaches to refine the overall process. ²⁹

Table 2 presents the detailed feasibility assessments for potential emulations of the KEYNOTE‐189 trial in the three RWD sources remaining after screening, referred to here as Data Sources 2–4. The primary objective of the KEYNOTE‐189 trial was to compare overall survival and progression‐free survival among adult patients with metastatic non‐squamous non‐small cell lung cancer without EGFR or ALK mutations who were treated with first‐line pembrolizumab and chemotherapy vs. chemotherapy alone. ³⁰ As described above, definitions of the treatment regimens, inclusion and exclusion criteria, and outcomes were identified from the KEYNOTE‐189 trial publication and key potential confounders were determined based on patient characteristics presented in the publication and subject matter knowledge.

Table 2.

Data feasibility assessments for emulation of the KEYNOTE‐189 trial in three real‐world data sources

KEYNOTE‐189 Trial publication
Step 1: State research aim, question, and objectives
Step 1a: Overarching research aim
To emulate the KEYNOTE‐189 randomized clinical trial of pembrolizumab + chemotherapy for the first‐line treatment of metastatic nonsquamous non‐small cell lung cancer without EGFR or ALK mutations using real‐world data.
Step 1b: Trial research question
Among adult patients with metastatic nonsquamous non‐small cell lung cancer without EGFR or ALK mutations, does initial treatment with pembrolizumab + chemotherapy compared with treatment with chemotherapy alone result in longer overall and progression‐free survival?
Step 1c: Trial primary objective(s)
Among adult patients with metastatic nonsquamous non‐small cell lung cancer without EGFR or ALK mutations, compare overall survival and progression‐free survival for patients treated with pembrolizumab + chemotherapy and patients treated with chemotherapy alone.
Design elements	Step 2: Describe original clinical trial	Step 3: Describe real‐world data study emulation of original clinical trial		Step 4: Real‐world data source feasibility assessment
		3a. Minimal criteria for valid operationalization in real‐world data source	3b. Criteria ranking with regard to uniqueness and importance	Data source 2		Data source 3		Data source 4
Overall rating				3		5		4
General
Sample size	Trial sample size	1.5x trial sample size	Ranking	Sample size among adult patients with metastatic non‐small cell lung cancer who received first‐line pembrolizumab + pemetrexed + (carbo/cis)platin or pemetrexed + (carbo/cis)platin
Treated	410	615	Essential	1,736	5	2,605	5	576	4
Comparator	206	309		948	5	1,925	5	600	5
Length and frequency of follow‐up 1	Median reported follow‐up: 10.5 months (range: 0.2 to 20.4 months)	Sufficient time coverage in dataset to identify outcome after receipt of treatment		Earliest metastatic diagnosis date: Q2 2009 End of datacut: Q1 2023	N/A 2	Earliest metastatic diagnosis date: Q3 2004 End of datacut: Q3 2023	N/A 2	Earliest metastatic diagnosis date: Q3 2019 End of data cut: Q1 2023	N/A 2
Variable‐related
Variable	Original clinical trial definition (per publication)	Minimal criteria for valid operationalization in any real‐world data source based on routine clinical care	Criteria ranking with regard to uniqueness and importance	Operationalization and coverage in data source	Rating	Operationalization and coverage in data source	Rating	Operationalization and coverage in data source	Rating
Treatment	200 mg intravenous (IV) pembrolizumab + four cycles of the investigator's choice of IV cisplatin (75 mg/m^2) or IV carboplatin (area under the concentration‐ time curve, 5 mg per milliliter per minute) + pemetrexed (500 mg/m^2), every 3 weeks, followed by pemetrexed (500 mg per square meter) every 3 weeks	Date of pembrolizumab, pemetrexed and carboplatin or cisplatin treatment	Essential	Date of pembrolizumab, pemetrexed, and carboplatin or cisplatin treatment is available. Exploratory, curated lines of therapy are available.	5	Date of pembrolizumab, pemetrexed, and carboplatin or cisplatin treatment is available. Curated lines of therapy are available.	5	Curated date of pembrolizumab, pemetrexed, and carboplatin or cisplatin regimens are available.	5
Comparator	200 mg IV saline placebo + four cycles of the investigator's choice of IV cisplatin (75 mg/m^2) or IV carboplatin (area under the concentration–time curve, 5 mg per milliliter per minute) + pemetrexed (500 mg/m^2), every 3 weeks, followed by pemetrexed (500 mg per square meter) every 3 weeks	Date of pemetrexed and carboplatin or cisplatin treatment	Essential	Date of pemetrexed and carboplatin or cisplatin treatment is available. Exploratory, curated lines of therapy are available.	5	Date of pemetrexed and carboplatin or cisplatin treatment is available. Curated lines of therapy are available.	5	Curated date of pemetrexed and carboplatin or cisplatin regimens are available.	5
Inclusion Criterion 1	18 years of age or older	Year of birth	Essential	Year of birth is available.	5	Year of birth is available.	5	Year of birth can be determined using age at diagnosis and date of diagnosis information.	5
Inclusion Criterion 2	Pathologically confirmed metastatic nonsquamous non‐small cell lung cancer	Diagnosis of non‐small cell lung cancer with histological and/or pathological confirmation of subtype; date of metastatic diagnosis	Essential	Curated non‐small cell lung cancer diagnosis and histology and/or pathology are available. Curated date of metastatic diagnosis is available.	5	Curated non‐small cell lung cancer diagnosis and histology and/or pathology are available. Curated date of metastatic diagnosis is available.	5	Curated non‐small cell lung cancer diagnosis and histology and/or pathology are available. Date of metastatic diagnosis can be determined from staging and Tumor, Node, Metastasis (TNM) information.	5
Inclusion Criterion 3	No sensitizing epidermal growth factor receptor (EGFR) or anaplastic lymphoma kinase (ALK) mutations	Dates and result of biomarker tests	Essential	Curated biomarker test results and dates are available and are populated for 50‐60% of patients.	3	Curated biomarker test results and dates are available.	5	Curated biomarker test results and dates are available.	5
Inclusion Criterion 4	Received no previous systemic therapy for metastatic disease	Names/types and dates of antineoplastic treatment; date of metastatic diagnosis	Essential	Curated dates of antineoplastic treatment are available. Exploratory, curated lines of therapy are available.	5	Curated lines of therapy are available.	5	Curated date of metastatic treatment regimens is available.	5
Inclusion Criterion 5	Eastern Cooperative Oncology Group (ECOG) performance status score of 0 or 1	ECOG performance status result	Essential	Curated ECOG or Karnofsky performance status is available and is populated for 60‐70% of patients.	4	Curated ECOG or Karnofsky performance status information is available.	5	Curated ECOG or Karnofsky performance status information is available.	5
Inclusion Criterion 6	Has at least one measurable lesion according to the Response Evaluation Criteria in Solid Tumors (RECIST) v1.1	RECIST is not used to assess progression or response in a real‐world setting	Not Applicable	RECIST is not used to assess progression or response in a real‐world setting. Progression will be assessed with available real‐world information (see below).	N/A	RECIST is not used to assess progression or response in a real‐world setting. Progression will be assessed with available real‐world information (see below).	N/A	RECIST is not used to assess progression or response in a real‐world setting. Progression will be assessed with available real‐world information (see below).	N/A
Inclusion Criterion 7	Provided a tumor sample for determination of programmed death‐ligand 1 (PD‐L1) status	Patient agreement to provide a tumor tissue sample is not captured outside of a clinical trial setting and therefore is not relevant to a real‐world emulation	Not Applicable	This criterion will not be operationalized.	N/A	This criterion will not be operationalized.	N/A	This criterion will not be operationalized.	N/A
Exclusion Criterion 1	Evidence of symptomatic central nervous system metastases	Dates and locations of distant metastases	Supplemental	Curated date and site of metastases are available.	5	Curated date and site of metastases are available.	5	Curated date and site of metastases are available.	5
Exclusion Criterion 2	History of noninfectious pneumonitis that required the use of glucocorticoids	Date of noninfectious pneumonitis diagnosis; date of glucocorticoid treatment	Supplemental	Curated date of noninfectious pneumonitis diagnosis is available, and is populated for 50‐60% of patients., and completeness could not be ascertained. Date of glucocorticoids treatment is available.	3	Curated date of noninfectious pneumonitis diagnosis is available and is populated for 60‐70% of patients and completeness could not be ascertained. Date of glucocorticoids treatment is available.	4	Non‐cancer diagnoses and treatments are not available and are populated for 40‐50% of patients. Non‐infectious pneumonitis can be proxied using a curated Charlson Comorbidity variable indicating presence of chronic pulmonary disease, but completeness could not be ascertained.	2
Exclusion Criterion 3	Active autoimmune disease or systemic immunosuppressive treatment	Date of autoimmune disease diagnosis; dates of immunosuppressant treatment	Supplemental	Curated date of autoimmune disease diagnosis is available, but completeness could not be ascertained. Date of immunosuppressive treatment is available.	3	Curated date of autoimmune disease diagnosis is available and is populated for 60‐70% of patients and completeness could not be ascertained. Date of immunosuppressive treatment is available.	4	Non‐cancer diagnoses and treatments are not available and are populated for 40‐50% of patients. Active autoimmune disease can be proxied using a curated Charlson Comorbidity variable indicating presence of rheumatic disease, but completeness could not be ascertained.	2
Exclusion Criterion 4	Received > 30 Gray of radiation therapy to the lung in the 6 months prior to the first dose of study medication	Date, location, and dose of radiation therapy	Supplemental	Curated dates, doses, and sites of radiation therapy are available and populated for 40‐< 50% of patients.	2	Curated dates, doses, and sites of radiation therapy are available and populated for 60‐< 70% of patients.	4	Curated dates, doses, and sites of radiation therapy are available and populated for > = 70% of patients.	5
Primary Outcome 1 (Definition & Ascertainment)	Overall survival	Date of death; dates of healthcare interactions	Essential	Date of death is available, but vendor reports low ascertainment. Date of last activity is available.	2	Date of death is available. Date of last activity is available.	4	Date of death is available and validated. Date of last activity is available.	5
Primary Outcome 2 (Definition & Ascertainment)	Progression‐free survival	Date of death; curated progression variable; imaging results; dates of healthcare interactions	Essential	Curated progression information is available. Date of death is available, but vendor reports low ascertainment. Date of last activity is available.	2	Curated progression information is available. Date of death is available. Date of last activity is available.	4	Curated progression information is available. Date of death is available and validated. Date of last activity is available.	5
Confounding Variable 1	Not applicable in a randomized setting	Age	Essential	Year of birth is available.	5	Year of birth is available.	5	Year of birth can be determined using age at diagnosis and date of diagnosis information.	5
Confounding Variable 2	Not applicable in a randomized setting	Sex	Essential	Sex is available.	5	Sex is available.	5	Sex is available.	5
Confounding Variable 3	Not applicable in a randomized setting	Race/ethnicity	Essential	Race/ethnicity is available.	5	Race/ethnicity is available.	5	Race/ethnicity is available.	5
Confounding Variable 4	Not applicable in a randomized setting	Performance status	Essential	Curated ECOG or Karnofsky performance status is available but is populated for 60‐70% of patients.	4	Curated ECOG or Karnofsky performance status information is available.	5	Curated ECOG or Karnofsky performance status information is available.	5
Confounding Variable 5	Not applicable in a randomized setting	Smoking status	Essential	Curated smoking status is available.	5	Curated smoking status is available.	5	Curated smoking status is available.	5
Confounding Variable 6	Not applicable in a randomized setting	Progression/disease free interval (Time from initial diagnosis to metastatic diagnosis)	Essential	Curated date of initial diagnosis is available. Curated date of metastatic diagnosis is available.	5	Curated date of initial diagnosis is available. Curated date of metastatic diagnosis is available.	5	Date of initial diagnosis is available. Metastatic diagnosis date can be determined from staging and TNM information.	5
Confounding Variable 7	Not applicable in a randomized setting	Year of study treatment initiation	Essential	Year of study treatment initiation is available.	5	Year of study treatment initiation is available.	5	Year of study treatment initiation is available.	5
Confounding Variable 8	Not applicable in a randomized setting	Number and/or location(s) of metastatic sites	Essential	Curated location of metastatic sites is available. Number of metastatic sites can be determined from this information.	5	Curated location of metastatic sites is available. Number of metastatic sites can be determined from this information.	5	Curated location of metastatic sites is available. Number of metastatic sites can be determined from this information.	5
Confounding Variable 9	Not applicable in a randomized setting	PD‐L1 tumor proportion score status	Essential	Curated biomarker test results and dates are available and are populated for 50‐60% of patients.	3	Curated biomarker test results and dates are available.	5	Curated biomarker test results and dates are available.	5

¹ Follow‐up time is stated as reported in the trial publication. Maximum available observation time is reported for the real‐world data source. These are not directly comparable.

² The final study period would be defined in the study protocol based on the date of treatment approval and relevant updates to treatment guidelines.

As for any real‐world study, sufficient sample size and the ability to validly identify the indicated population, treatments of interest, and outcomes in a dataset were considered essential elements. These study design features were well‐captured in all three RWD sources. Data elements required to operationalize all inclusion criteria mentioned in the KEYNOTE‐189 publication were also classified as essential for accurately emulating the study population, apart from two criteria determined to be not applicable in a real‐world setting: the requirement to have at least one measurable lesion according to RECIST and patient willingness to provide a tumor sample for determination of programmed death‐ligand 1 (PD‐L1) status. Data elements required to operationalize trial exclusion criteria were classified as supplemental. All hypothesized confounders were classified as essential elements to generate valid estimates.

Essential elements were available in all three RWD sources assessed for emulation of the KEYNOTE‐189 RCT, but they varied primarily with respect to completeness and validity. For example, Data Source 2 received lower ratings than Data Sources 3 and 4 on most elements due to larger amounts of missing data and less reliable outcome information (e.g., low ascertainment of date of death due to limited external sourcing). Data Sources 3 and 4 both contained reliable and complete information for most essential design elements. However, death data were not formally validated in Data Source 3, resulting in a lower rating for that element and Data Source 4 had lower estimated sample size. These differences informed the overall ratings assigned to the three data sources.

Key findings from all nine detailed feasibility assessments are summarized in Table 3 . The three assessments conducted in Data Source 1 received the lowest overall ratings (“2”), driven by absence of performance status and disease progression measures and the need to rely on International Classification of Disease (ICD) codes rather than curated variables to determine the fact and date of metastatic disease diagnosis. Data sources with higher overall ratings tended to include more curated data elements—variables constructed from abstraction and review of both structured and unstructured (e.g., provider notes, pathology, and imaging reports) EHR data elements. The feasibility of using a particular RWD source for emulation also varied across different RCTs. For example, Data Source 4 received the highest possible rating (“5”) for emulation of the ASCEND trial ³¹ because of the availability of high‐quality, oncology‐specific measures in these data. However, this same dataset received a lower feasibility rating (“4”) for emulating the KEYNOTE‐189 RCT ³⁰ due to relatively few patients with metastatic non‐small cell lung cancer and a lack of information on non‐cancer diagnoses and treatments required for evaluating eligibility criteria.

Table 3.

Results of phase two detailed feasibility assessments

Trial	Indication	Primary outcome(s)	Data source	Overall rating (1–5)	Key rating determinants
KEYNOTE‐045	Advanced urothelial cancer	Overall survival, Progression‐free survival	DS1	2	Date of metastatic diagnosis primarily determined using ICD codes
					Performance status not available
					Progression not available
KEYNOTE‐426	Advanced renal cancer	Overall survival, Progression‐free survival	DS1	2	Date of metastatic diagnosis primarily determined using ICD codes
					Performance status not available
					Progression not available
NAPOLI‐1	Metastatic pancreatic cancer	Overall survival	DS1	2	Date of metastatic diagnosis primarily determined using ICD codes
					Performance status not available
PALOMA‐2	Metastatic breast cancer	Progression‐free survival	DS2	4	Key inclusion/exclusion criteria can be operationalized with good coverage
					Progression available
					Low quality death data, but long survival in trial and in ER(+)/HER2(‐) breast cancer general population
			DS3	5	Key inclusion/exclusion criteria can be operationalized with good coverage
					Progression and reliable death data available
KEYNOTE‐189	Metastatic non‐small cell lung cancer	Overall survival, Progression‐free survival	DS2	3	Key inclusion/exclusion criteria can be operationalized with good coverage
					Progression available
					Low quality death data
			DS3	5	Key inclusion/exclusion criteria can be operationalized with good coverage
					Progression and reliable death data available
			DS4	4	Key inclusion/exclusion criteria can be operationalized with good coverage
					No information on non‐cancer treatments
					Low sample size
ASCEND	Relapsed/ refractory chronic lymphocytic leukemia	Progression‐free survival	DS4	5	Key inclusion/exclusion criteria can be operationalized with good coverage
					Progression available

Based on these findings, and further refined sample size estimates indicating inadequate sample size to proceed with the ASCEND emulation, three real‐world emulations—KEYNOTE‐189 ³⁰ in Data Sources 3 and 4, and PALOMA‐2 ³² in Data Source 3—were ultimately selected by consensus for protocol development.

DISCUSSION

Identification of a fit‐for‐purpose data source is critical to the success of any study that uses RWD. ⁹ , ³³ This process is particularly challenging when seeking data for RCT emulations in oncology because oncology‐specific eligibility and outcome measures, such as ECOG status and disease progression, are collected in trials according to standardized protocol‐specified guidelines that dictate which assessments can inform these measures, when to assess them, and how to assign response categories. In contrast, real‐world clinical practice is less standardized, and equivalent real‐world measures may be based on variable clinical information and assessment timing and may consider symptoms and other aspects of a patient's overall clinical status. Furthermore, documentation of these assessments in the patient record varies by clinician and practice. In data feasibility assessments conducted for the CARE Initiative, we found that the best‐performing EHR‐based data sources were highly curated to create standardized oncology‐specific measures using both structured and unstructured data elements. However, given the relatively smaller patient counts in curated data sources and the specificity of many oncology RCT indications and study populations, sources with the highest‐quality oncology data could not be used for all potential RCT emulations. We also observed that RWD sources with curated, oncology‐specific measures were less likely to include information about non‐cancer diagnoses and treatments, though these data are likely routinely captured in other EHR and insurance claims data. This made it difficult to identify off‐the‐shelf RWD sources with both high‐quality, oncology‐related measures needed for internal validity and sufficient information to fully operationalize RCT inclusion and exclusion criteria needed for external validity. This suggests that data fitness for oncology emulation might be improved by augmenting existing, oncology‐specific sources through linkage to broader EHR and claims data sources or through customized data abstraction.

Of 54 candidate RCT‐RWD combinations evaluated in phase one of the CARE Initiative feasibility assessments, 10 potential emulations were identified as potentially feasible based on the presence of the RCT indication, treatments, and outcomes in the data source and sufficient sample size based on preliminary estimates. One RCT–dataset combination was selected for the pilot emulation ²⁹ and nine proceeded to detailed feasibility assessment. Ultimately, only three of these nine emulation studies advanced to protocol development. This attrition emphasizes the importance of conducting feasibility assessments specifically geared toward RCT emulation when selecting RWD sources for this purpose.

We used a modified version of the SPIFD2 framework to guide the detailed feasibility assessments. For most non‐interventional RWD studies, the SPIFD2 framework begins by specifying the hypothetical target trial that would be conducted to answer the research question if it were feasible and ethical. Data feasibility assessments then primarily focus on how well the indicated population, exposures, outcomes, and confounders can be operationalized in a real‐world data source to ensure internal validity. While these same considerations are also important for RCT emulation studies, success is additionally highly dependent on operationalizing the specific features and design elements of the RCT being emulated. RCTs often employ stringent inclusion and exclusion criteria that shape the demographic and clinical characteristics of the study population and therefore dictate the target population to which the RCT results can be generalized. As a result, a fit‐for‐emulation RWD source must achieve internal validity and enable identification of a study population similar to that of the RCT. This highlights an important distinction: because the goal of RCT emulation studies is to mirror the RCT as closely as possible, design choices may not be optimal for producing real‐world evidence of comparative effectiveness. As a result, the same RWD source may not be fit‐for‐emulation of an RCT but could be fit‐for‐purpose for other types of RWD studies.

A primary limitation of this work is that we did not consider the entire universe of available RWD sources and only assessed emulation feasibility in six, US‐based partner datasets. The results of our data feasibility assessments and the patterns of RWD source characteristics may have differed if additional sources of RWD had been included. Further work is needed to understand the strengths and limitations of other US and global RWD sources for oncology RCT emulation. Additionally, ratings of how well individual data elements could be operationalized involved a degree of subjectivity, particularly in cases where completeness could not be ascertained, and we did not evaluate agreement metrics, which may have influenced overall feasibility ratings. We attempted to mitigate this by having multiple study team members review each feasibility assessment, seeking subject matter expertise from Steering Committee members, and requiring study team consensus on final decisions. However, small differences in element and overall ratings between data sources should not be overinterpreted as indicating superiority of a given data source. Additionally, variables required to address complex real‐world analytic considerations, such as substantial treatment arm crossover and time‐varying confounding, were outside the scope of the present data feasibility assessment and therefore would remain a potential explanation for observed differences between RCT and emulation findings.

Despite these limitations, we believe that the results of this systematic assessment of fit‐for‐emulation data sources can aid other researchers and improve transparency around how and why a particular RWD source was selected for emulation of an RCT. This work also highlights the unique challenges of conducting RWD studies and emulations in oncology. Our finding that RWD sources with high‐quality, oncology‐specific data lacked information on non‐cancer diagnoses and treatments suggests that there is an opportunity for data vendors to improve their capture of other data generated during routine clinical practice through linkage and additional abstraction. Such efforts would enhance existing RWD sources and increase opportunities for its use in studying the real‐world effectiveness and safety of oncology treatments.

Funding

This study was funded by grants from Amgen, AbbVie, Bayer, AstraZeneca, Gilead Sciences, Pfizer, and Johnson & Johnson. Aetion received funding from the multistakeholder sponsors to cover the costs to conduct and manage the CARE Initiative.

Conflicts of interest

Levy, Sheridan, Campbell, Lenis, O'Dougherty, Estrin, Gautam, McDonald and Madsen were employed by Aetion during the conduct of the study. Schneeweiss has received grants from UCB Pharma, Takeda, and Boehringer Ingelheim outside of the submitted work. Dr S. Wang reported receiving personal fees from Veracity Healthcare Analytics, Exponent and MITRE. Dr Carrigan reported being employed by and owning stock in Amgen outside the submitted work. Dr Chia reported being employed by and owning stock in Amgen outside the submitted work. Dr Pinheiro reported being employed by and owning stock in AbbVie during the conduct of the study. Dr Khosla reported being employed by AstraZeneca during the conduct of the study. Dr Rimawi reported receiving personal fees from Pfizer, AstraZeneca, Novartis and Sermonix and grants from Greenwich LifeSciences (paid to institution) outside the submitted work. Dr A. Taylor reported being employed by and owning stock in Gilead during the conduct of the study. Dr Jiao reported being employed by Bayer during the conduct of the study and having previously been employed by Pfizer outside the submitted work. Dr McRoy reported being employed by Pfizer during the conduct of the study. Dr Lunacsek reported being employed by Bayer during the conduct of the study. Dr Sarsour reported being employed by Johnson & Johnson during the conduct of the study. Dr Belli reported being employed by and owning stock in COTA during the conduct of the study. Dr C.K. Wang reported being employed by COTA during the conduct of the study. Dr Chen reported receiving personal fees from Tempus AI during the conduct of the study. Dr Sangli reported being employed by Tempus AI during the conduct of the study. Dr Natanzon reported being employed by ConcertAI during the conduct of the study. Dr Chan reported being employed by TriNetX during the conduct of the study and receiving grants from Amgen and Boehringer Ingelheim outside the submitted work. Dr Dhopeshwarkar reported being employed by and owning stock in TriNetX during the conduct of the study. Dr Rider reported being employed by Aetion and Concert AI during the conduct of the study and receiving personal fees from Bayer and Monsanto outside the submitted work. No other disclosures were reported.

Author contributions

N.L, P.S., and A.M. wrote the manuscript. N.L., P.S., U.C., D.L., I.O., A.E., N.G., M.I., S.M., A.B., G.C., K.A.C., J.C., V.C., N.D., J.E., L.F., J.G., R.H.S., J.H., X.J., S.K., O.L., L.M., Y.N., O.O., N.P., S.P., M.R., J.R., M.F.R., T.R., C.R.‐W., C.S., K.S., S.S., M.Sh., M.St., A.T., C.K.W., S.W., Y.Z., and A.M. designed the research. N.L., P.S., A.E., N.G., M.I., and S.M., performed the research; N.L., P.S., A.E., N.G., M.I., and S.M., analyzed the data.