Treatment effectiveness in a rare oncology indication: Lessons from an external control cohort study

Abstract

Real‐world data (RWD) reflecting patient treatment in routine clinical practice can be used to develop external control groups for single‐arm trials. External controls can provide valuable benchmark results on potential comparator drug effectiveness, particularly in rare indications when randomized controlled trials are either infeasible or unethical. This paper describes lessons learned from a descriptive real‐world external control cohort study conducted to provide benchmark data for a single‐arm clinical trial in a rare oncology biomarker driven disease. Conducting external control cohort studies to evaluate treatment effectiveness in rare indications likely will present data and analysis challenges as seen in the example study. However, there are mitigating measures that can be applied in the study design, identification of RWD sources, and data analysis. The lessons learned and reported here with a proposal of an external control study framework can provide guidance for future research in this area, and may be applicable as well in other rare indications. Taking these learnings into consideration, the use of real‐world external controls to contextualize treatment effectiveness in rare indications is a valuable approach and warrants further application in the future.

Study Highlights

• WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?

Real‐world data (RWD) can provide useful insights into patient treatment and outcomes in routine clinical practice, and can be used to develop external controls for single‐arm clinical trials.

• WHAT QUESTION DID THIS STUDY ADDRESS?

The applied methods, challenges, and lessons learned from an external control study were reported.

• WHAT DOES THIS STUDY ADD TO OUR KNOWLEDGE?

Whereas there are inherent challenges associated with conducting a real‐world external control cohort study design, in the oncology disease setting, critical data are furthermore often missing in RWD sources.

• HOW MIGHT THIS CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE?

Lessons learned and reported here provide important considerations for the conduct of external control studies in rare oncology indications, and may be applicable to research in other rare conditions.

INTRODUCTION

Treatments targeting rare diseases with high morbidity and mortality are more often investigated in single‐arm trials than treatments targeting more common indications. ¹ Historical data from earlier clinical trials, or from similar patient cohorts to single‐arm trial cohorts, have been used to report comparative effectiveness outcomes among a “real‐world” control. ² , ³ , ⁴ External controls, an umbrella term, that includes “historical controls” and “synthetic controls,” describes a sample of patients from another source, which is comparable to a clinical trial population treatment arm. ⁵ For example, the term has been used to describe the analysis of data from single‐arm, single‐institution interventional oncology trials, where efficacy or safety is compared with outcomes in historical patient cohorts previously treated with a comparator drug, for the same indication at the same institution. ⁶ , ⁷ , ⁸ , ⁹ , ¹⁰ , ¹¹ , ¹² , ¹³ Data from randomized controlled trials’ treatment arms could potentially be used as external controls for a single‐arm trial. However, real‐world data (RWD) from sources reflecting patient treatment in routine clinical practice, including registry chart data or electronic medical records (EMRs) from existing databases, can be used to identify controls when clinical trial control data are not available. ¹⁴ , ¹⁵ , ¹⁶

Use of real‐world evidence (RWE) derived from RWD to evaluate the effectiveness and safety of available therapies or standard of care is increasingly considered by national regulatory agencies as complementary to clinical trial data for regulatory decisions. ¹⁷ , ¹⁸ , ¹⁹ , ²⁰ , ²¹ , ²² , ²³ RWE can provide valuable data on the effectiveness of therapies administered in routine care to use as a benchmark for the target population of a single‐arm trial. This is particularly useful for single‐arm trials in rare and novel indications with little historical or published evidence available. According to guidance from regulators to industry on external controls used in regulatory submissions, RWD should be of high quality and the comparability of real‐world patients to clinical trial patients assured, minimizing the potential for selection bias and confounding factors as much as possible. ²⁴ Examples include the selection of real‐world patients according to the same or similar criteria as in the corresponding clinical trial, and application of analyses that account for all factors with potential to impact effectiveness results. ²⁰ , ²¹ , ²⁵ Guidance from regulators to industry further details that the use of RWE should be planned a priori, transparently protocolized, and possibly considered prior to or in conjunction with the planning of the single‐arm clinical trial to generate evidence on the rare condition to guide clinical decision making. ²⁰ , ²¹

Electronic medical records are a promising source of patient‐level information, but data elements may be missing from typical patient EMRs recorded in routine clinical practice that are particularly valuable in oncology disease assessments. One example of missing information is on novel genetic markers, which are not routinely tested outside a clinical trial setting. ¹² , ¹⁵ , ¹⁹ , ²⁵ Missing data can complicate appropriate patient identification for analysis when using RWD sources. Using RWD‐derived external control data as comparators for single‐arm studies is associated with potential bias related to patient selection, and confounding arising from the lack of randomization. ¹⁵ , ²⁵ , ²⁶ However, carefully considered external control study designs may address this potential bias in the selection criteria for patients, and statistical methods can be applied to ensure comparability of the external controls to the single‐arm patients. For example, propensity score (PS) methodologies in an external control study could be used to address the inherent lack of randomization in single‐arm trials. ¹⁵ , ²⁷ In addition to the industry guidance on external controls, ²⁰ a white paper released by the Friends of Cancer working group in 2019, described the use and benefit of external controls for augmenting randomized control arms. ⁵ Other groups have provided guidance and suggestions for improving the methods applied in order to address the challenges in selecting external controls. ²⁸ , ²⁹ , ³⁰ , ³¹

Here, we use as an example an external control cohort study for a single‐arm trial, VISION, an open‐label, phase II trial (NCT02864992) that assessed the effectiveness of tepotinib on tumor objective response in patients with advanced (stage IIIB–IV) non‐small cell lung cancer (NSCLC) harboring mesenchymal epithelial transition exon 14 (METex14) skipping alterations, ³² an oncogenic driver occurring in 3–4% of patients with NSCLC. ³³ , ³⁴ Tepotinib recently received approval in Japan, the United States, Canada, Switzerland, Brazil, Great Britain, Taiwan, Singapore, and the Republic of Korea, for treatment of patients with either advanced or metastatic METex14 NSCLC, irrespective of prior anticancer therapies. ³⁵ , ³⁶ , ³⁷ , ³⁸ , ³⁹ , ⁴⁰ The low prevalence of METex14 skipping in patients with NSCLC precluded a randomized controlled clinical trial for this indication. ²⁰ , ⁴¹ , ⁴² Using RWD, the external control study aimed to provide contextualization data for the single‐arm trial, according to guidance from health authorities on use of RWE in the regulatory context available at the time, and applying the strictest possible criteria to both data and methodology, to enhance comparability of the external control cohort to the clinical trial cohort.

This paper describes and highlights the key challenges and associated lessons learned, as well as design and analytical considerations for future use of external real‐world controls to provide evidence on treatment effectiveness.

METHODS

The methodology of the example study for these lessons learned in this paper is summarized briefly below.

Study design and data sources

Given that the specific subtype of NSCLC is rare and, at the time of the study, only recently discovered as a treatment target for patients with NSCLC, historical data were limited for the example study. Three different EMR databases were pooled to create a larger sample: EMR data from the US ConcertAI EMR database (42 patients), the US COTA Healthcare EMR database (89 patients), and EMR data from a multi‐country chart review study (70 patients). ⁴³ These three fit‐for‐purpose databases were selected after a feasibility assessment. All had genetic data to confirm METex14 status together with EMR data, and variables for selected patients could be extracted as close as possible to the variables in the VISION trial. Additional datasets were not found or did not have the required data at the time of the study. The time period of the pooled dataset spanned from January 2004 to February 2020. International review board approvals, waivers, or patient consent forms had been obtained for use of data, which permitted the analysis. As the intent was to identify an external control cohort to descriptively contextualize the results from the clinical trial, no direct comparison was planned a priori.

Identification of external controls

External controls were selected from the pooled dataset. Adult patients with advanced (stage IIIB–IV) NSCLC harboring METex14 skipping alterations who were treated with systemic therapies were included.

To identify external controls comparable to the VISION tepotinib intention‐to‐treat patient population, VISION inclusion and exclusion criteria were applied, unless the criterion was not applicable in the routine real‐world practice (Table S1). Potential confounders, such as disease characteristics and severity, were considered to further refine the selection of patients. ¹² , ²⁰ Then, PS‐based methods were applied to the external controls to increase comparability to the VISION trial patients. This approach considered first‐, second‐, and third‐line patients separately, so that patient characteristics were balanced within each treatment line of therapy (LOT) group compared to the corresponding treatment line of tepotinib used in the trial.

An iterative, contingent analysis approach was planned a priori, and applied to optimize covariate balance between the trial population and the external control cohort, which prioritized the retention of all trial patients. The steps of the approach are further described below and summarized in Table 1. Comparability between the clinical trial patient population and the external control cohort was assessed using absolute standardized mean differences before and after each applied PS method.

TABLE 1.

Pre‐planned stepwise assessment of the method to achieve balance between clinical trial population and external control cohort

Step	Description	Criteria necessary to negate the subsequent step
1. Unadjusted	Covariate balance assessment via standardized mean differences between the VISION patient population and the RWC (selected according to VISION inclusion and exclusion criteria)	Covariate balance
2. PS matching	Nearest neighbor 1:1 PS matching. Covariate balance between the VISION population and external control cohort using a caliper of 0.01, and then increasing by increments of 0.01, as necessary, until a maximum caliper of 0.05 is reached	Retain all trial patients (all trial patients are able to be matched) Covariate balance
3. PS weighting	Standardized mortality weighting to estimate the average treatment effect in the treated and evaluation of covariate balance between the VISION population and external control cohort (weights ≤10)	Retain all trial patients (no patients excluded due to weights >10) Covariate balance
4. Partially matched	Populations identified in step 2 re‐evaluated by partial matching	Retain 90% of trial patients (90% of trial patients are able to be matched) Covariate balance

Abbreviations: PS, propensity score; RWC, real‐world cohort.

Statistical analysis

Patients were followed from the date of initiation of an LOT until death, disease progression, change of LOT, or end of the study period. The aim was to describe real‐world effectiveness outcomes including: objective response (OR); duration of response (DoR); progression‐free survival (PFS); and overall survival (OS). Effectiveness outcomes (OR, DoR, and PFS) that rely on tumor response information were described in patients with responses that could be categorized as best possible mimicking Response Evaluation Criteria in Solid Tumors (RECIST) version 1.1. Any tumor response recorded in the patient record was assumed to count, even if unconfirmed by other assessments, as the concept of confirmed or unconfirmed responses is not applicable in the real‐world clinical care settings. Tumor response was assessed based on either availability of tumor scan reports and/or physician notes in the patient record. If there were conflicts between records, the physician’s tumor response assessment was prioritized. All patient records were reviewed by at least two independent clinical reviewers to assess tumor response.

Descriptive statistics were reported for all study variables, and Kaplan–Meier analyses were conducted for each time‐to‐event outcome. Missing patient Eastern Cooperative Oncology Group performance status (ECOG) score information was supplemented by Karnofsky scores, where available, as a proxy. ⁴⁴

Lessons learned

For this paper, lessons learned were captured throughout all stages of the process from design and data acquisition, to analysis implementation and interpretation of findings. Categories were created according to key external control study principles and the respective challenges. Information was summarized and used to compose design and analytical considerations for future use of external real‐world controls to provide evidence to support treatment effectiveness in rare oncology indications.

RESULTS

The following categories were created according to the key external control study principles that the lessons learned related to:

• Anticipate external control data challenges

• Consider the time‐window for external control inclusion and study index date

• Optimize confounding control considering the limitations of the data

• Prepare your analysis plan for the unknown

Anticipate external control data challenges

In the example external control study, datasets were selected where variables similar to the trial could be retrieved. The real‐world datasets had to be standardized to a common data model to enable pooling with the clinical trial data which applied Clinical Data Interchange Standards Consortium (CDISC) standards. The RWD datasets used for this study all used different data standards than CDISC, which is common in secondary use datasets. However, the data transformation into CDISC‐like standards presented a unique challenge, requiring data analysis experts from the clinical trial setting and the epidemiology/real‐world setting to collaborate on developing a common data model.

Pooling datasets resulted in a total of 201 real‐world patients (Figure S1) and 99 clinical trial patients. However, after applying the inclusion/exclusion criteria from the trial, the number of external control patients was smaller than in the corresponding trial for all analysis groups (across lines of therapy). Some missingness was anticipated for clinical parameters, such as ECOG score, which is used consistently in the trial setting, but not necessarily used or reliably recorded in the routine clinical practice setting. Finally, missing clinical data parameters, such as ECOG or Karnofsky score, prior to treatment initiation was the most common reason for patient exclusion, and was the main cause of the small number of external controls for the study apart from the rareness of the studied indication. Thus, even though the challenge of the rare indication was anticipated by pooling of different sources, strictness of patient selection, and data missingness in combination resulted in the small final sample.

Consider the time‐window for external control inclusion and study index date

For the example study, strictness of applied trial criteria was high with most criteria applied (Table S1) to ensure that patients were comparable at the exact time of treatment initiation. Most covariates were evaluated upon pre‐screening for entry into the clinical trial, whereas for the external control, most covariates were captured at the time of first‐line treatment initiation. Aligning the time‐window for external control inclusion and assignment of index dates, relative to advanced diagnosis and treatment initiation is important to avoid immortal time bias. This can arise if the external control population is treated at a later timepoint than the trial treatment group with respect to the initial diagnosis, or when baseline data were collected. Nevertheless, as ECOG score was often not available prior to their initiation of a treatment in either first or later lines of therapy, as indicated above, this meant that the majority of patients had to be excluded. Some missingness of data prior to initiation of a treatment line was expected and pooling of multiple datasets attempted to address this issue.

Optimize confounding control considering the limitations of the data

Confounding control should be applied to ensure that the selected real‐world external control cohort has optimal comparability to present unbiased treatment effectiveness estimates. Challenges arise when there is data missingness or validity concerns for any of the key covariates. Additional challenges arise when selected covariates are further limited, due to violations of the positivity assumption (i.e., present in one treatment arm but either very rare or not occurring in the other), which is necessary to perform valid PS estimations.

The small external control analysis groups in the example study limited the ability to apply robust models, by minimizing the number of covariates that could reasonably be controlled, largely due to lack of positivity for certain covariates. In addition, limited data for some categories of variables required collapsed categories, which may have had heterogeneous impact on the PS model, thereby reducing the number of parameters included (Table S3).

Prepare your analysis plan for the unknown

As noted previously, the iterative analysis approach steps could be applied to the first‐line therapy real‐world cohort (RWC) group only. Although the full model proposed a priori could be fitted, due to concerns of overfitting with a small sample size, a reduced model was used that prioritized the most imbalanced variables (i.e., sex, age, cancer stage, and ECOG score). In this final reduced model, there was sufficient overlap of the PS distributions (Figure S2), and no PS weight exceeded 10. This experience in the example study reinforces the decision to apply an iterative approach with contingent assessment of methods, given its support of the prioritization of retention of trial patients. This iterative analysis approach was developed a priori, and included in the protocol to consider the best method to achieve comparability between clinical trial patients and external controls (Table 1).

Moreover, despite rigorous assessment of LOT and tumor response assessments for patients by multiple clinical reviewers, the level of missingness was high, and analysis for certain outcomes, such as PFS, was not possible. Although data in the real‐world datasets used for the study had been extracted to mimic the clinical trial variables, the level of missingness in the pooled dataset was overall high. Last, as the treatment landscape in NSCLC during the study period changed significantly over time, some heterogeneity was expected, and descriptive outcome analysis by drug class had been anticipated already in the protocol. Even though treatment drug class categories were defined broadly, there were only a few patients seen in each category (Table S1). Within each drug class category, heterogeneity was seen in terms of mono‐ and combination therapy, necessitating additional stratification; However, this was infeasible due to the small sample size, and further outcome analyses could not be performed. Thus, data challenges were anticipated in the analysis plan of the example external control study, but could still not be sufficiently addressed with the datasets at hand.

DISCUSSION

This study highlighted key learnings for consideration when developing a real‐world external control cohort, which may be applicable to future external control studies in oncology and, potentially, to rare diseases in general. Based on the lessons learned from the example study in this paper, a potential framework for conducting external control studies is provided in Figure 1, and the steps discussed below.

FIGURE 1.

Illustration of framework for conducting external control studies. ECA, external control arm; EMRs, electronic medical records; RWD, real‐world data; SAT, single‐arm trial; SAP, statistical analysis plan

In the proposed framework, after the determination of a specific research question comes the development of a transparent protocol and statistical analysis plan (Figure 1). This step should consider the definition and documentation of outcomes, start of follow‐up, and any key variables necessary to apply the trial inclusion and exclusion criteria. An assessment of how best to achieve comparability between the clinical trial patients and the external control patients should be carefully described. The external control example study aimed to provide analyses on treatment patterns and outcomes for real‐world patients comparable to the trial dataset. It was possible to achieve an external control cohort with similar baseline characteristics to the clinical trial and known potential confounders balanced. Although it was not possible to match every clinical trial inclusion and exclusion criterion exactly using RWD, certain inclusion or exclusion criteria could be broadened to the real‐world context, as other researchers have also proposed ²⁷ and/or ignored if these were not perceived to affect comparability (Table S2). For example, trial selection criteria that aim to exclude patients due to a contraindication for the study treatment, may not be applicable for the comparator drug. Another selection criteria that could have been considered for omission is ECOG score if other key confounding patient characteristics were already well‐balanced. The external controls will have received the comparator drug, and thus are eligible for systemic therapy. An unknown distribution of ECOG score may be acceptable if other variables that indicate performance status can be balanced for. Sensitivity analyses could be performed in an external control study to investigate whether ECOG score significantly impacts overall results.

The low number of external controls was due to the rarity of the population, relative recent identification of the genetic biomarker studied, challenge of missingness of clinical variables in RWD, and strictness of applied clinical trial criteria. These issues likely extend to other rare patient populations, where an external control study could be considered to complement the clinical trial data. ³⁰ It is important to consider this in the step of identifying fit‐for‐purpose data (Figure 1) for the external control where data may have to be pooled from different sources. Furthermore, if pooling is necessary for a rare disease external control study, data standardization will be the first step to anticipate. Data extraction according to particular standards or data transformation may have to be implemented to enable pooling of the real‐world datasets and pooling of these with trial data. The ability to combine data sources is relevant for the rare disease setting, and when combining data with a common data model (CDM), the CDM needs to allow pooling while not losing important data for each data source. Data standardization is also important considering recent US Food and Drug Administration (FDA) guidance on submission of RWE to complement new drug applications which mentions submission of individual level data. ⁴⁵ RWD sources in general have different data formats and standards than clinical trials. National health authorities (HAs) typically receive clinical trial data in CDISC format. It follows that standardization of the external control dataset could potentially be needed in the future to enable pooling, but also to meet requirements of HAs for submission of the underlying data for external control studies. For prospective real‐world studies running in parallel with the trial, by aligning the electronic case report form to the clinical trial could be one way of anticipating this issue.

Although there are limitations associated with all RWD sources, ¹² , ²⁵ observations from the example external control study further highlight that secondary RWD challenges can potentially be anticipated to enable use for the development of external control populations in rare oncology patient populations. ²⁹ , ⁴⁶ The absence of certain clinical parameters commonly used in oncology clinical trials may reflect either a lack of routine use of these parameters in clinical practice, or a case‐ or center‐specific lack of recording. The example of the external control study applied rigorous assessment of key variables, such as LOT and tumor response with multiple clinical reviewers of individual level patient data, which still resulted in a high level of missingness. Nevertheless, for future studies, it could be possible to enrich RWD sources. For example, with additional human review of the original patient’s EMR and other available records at the site that provided the RWD, it may be possible to obtain additional data that include the missing information (i.e., imaging data, radiology reports, or missing calendar dates) which could be retrieved through third‐party providers and merged with the patient’s EMR. The anticipation of missingness can help to plan these activities well in advance of the external control study. This requires close collaboration with data providers to fully understand the completeness of data elements needed for the external control study.

Conducting the external control study prospectively and concurrently with the single‐arm trial (Figure 1), in order to plan and capture all necessary non‐interventional RWD, could also be an option to improve RWD quality by reducing missing variables and supporting more sophisticated matched comparisons of patient groups (Figure 1). ⁴⁷ Applying clinical trial inclusion and exclusion criteria to the real‐world patient group, and early consideration of clinical trial outcomes to accurately capture appropriate tumor response assessment data, could be more feasible in the primary data collection setting, compared with the secondary setting. ¹² , ³⁰ However, prospective patient enrollment in the trial and a parallel external control study could be challenging, when the condition is very rare. Additionally, if patients are screened for a biomarker in the trial but not routinely screened for this biomarker in the routine clinical setting, identifying sites for enrollment for these parallel studies may conflict. Hybrid designs, including historical data and prospective follow‐up of patients in care, could perhaps address this issue. As suggested above, there may be ways to anticipate missingness in the selection, collection, and/or curation of RWD in close collaboration with data providers which can enable a combination of different approaches.

For the final step of the framework of analysis of end points (Figure 1), there are particular considerations for studies in oncology. The primary end point of interest for the single‐arm trial in the setting of a rare biomarker identified oncology patient population is often OR, assessed according to RECIST version 1.1 with regulatory approval relying on this end point. Real‐world RECIST‐like tumor responses extracted from EMRs are likely not directly comparable to clinical trial RECIST version 1.1 assessments of tumor responses. In addition, assessment time bias (i.e., differences in tumor assessment schedules in the clinical trial and the real‐world practice), hinders direct comparison of outcomes in the study design of an external control study. ²⁰ , ⁴⁸ Although tumor response assessments in the real world are conducted at regularly scheduled intervals to monitor treatment effect and patient status, the frequency may differ from that of a clinical trial. This challenge may also present for time‐dependent end points, like DoR and PFS. When using RWD for an external control study in oncology assessment, time bias is likely to happen when compared with the regular schedule of a clinical trial. However, it may be feasible to assess differences in assessment protocols between the clinical trial and the external controls, and subsequently adjust for them in the analysis. ⁴⁸ Similarly, to check the robustness of outcomes definition, such as PFS or DoR when using RWD, a sensitivity analysis should be planned, notably in the context of the estimand strategy. This should be considered already at the step of defining and documenting outcomes carefully in the protocol and analysis plan within the proposed framework (Figure 1).

In conclusion, conducting external control cohort studies to evaluate treatment effectiveness in rare biomarker‐driven patient populations likely will present data and analysis challenges, but there are measures that can be applied to try to anticipate these. The lessons learned and reported here can provide guidance for future research in this area, and may be applicable as well in other rare indications. Taking these learnings into consideration, the use of real‐world external controls to contextualize treatment effectiveness in rare indications is a valuable approach and warrants further application in the future.

AUTHOR CONTRIBUTIONS

D.O. and E.M.G. wrote the manuscript. All authors designed and performed the research and analyzed the data.

CONFLICT OF INTEREST

P.P., E.M.G., A.E., and N.M.G. are full‐time employees of Aetion, Inc. a software and data analytics company, of which they hold stock options or equity. D.O., E.B., B.‐E.L., A.J., and P.V. are full‐time employees of the healthcare business of Merck KGaA, Darmstadt, Germany.

Supporting information

Figure S1

Figure S2

Table S1

Oksen D, Prince P, Boutmy E, et al. Treatment effectiveness in a rare oncology indication: Lessons from an external control cohort study. Clin Transl Sci. 2022;15:1990‐1998. doi: 10.1111/cts.13315