Systematic analysis of off-label and off-guideline cancer therapy usage in a real-world cohort of 165,912 US patients

Summary

Patients with cancer may be given treatments that are not officially approved (off-label) or recommended by guidelines (off-guideline). Here we present a data science framework to systematically characterize off-label and off-guideline usages using real-world data from de-identified electronic health records (EHR). We analyze treatment patterns in 165,912 US patients with 14 common cancer types. We find that 18.6% and 4.4% of patients have received at least one line of off-label and off-guideline cancer drugs, respectively. Patients with worse performance status, in later lines, or treated at academic hospitals are significantly more likely to receive off-label and off-guideline drugs. To quantify how predictable off-guideline usage is, we developed machine learning models to predict which drug a patient is likely to receive based on their clinical characteristics and previous treatments. Finally, we demonstrate that our systematic analyses generate hypotheses about patients’ response to treatments.

Graphical abstract

Highlights

• •Analyses of cancer treatments in 165,912 US patients with 14 common cancer types
• •Prevalent off-label and off-guideline usage of cancer drugs
• •Statistical model identifies which patients are more likely to receive off-label drugs
• •Quantifies adoption patterns of immunotherapies

Cancer patients may receive treatments that are not officially approved. Liu et al. systematically characterize off-label and off-guideline drug usage in 165,912 US patients with 14 cancer types. They model which patients are more likely to receive these off-label and off-guideline treatments and the adoption patterns of immunotherapies.

Introduction

In modern oncology care, anticancer drugs are frequently used beyond official regulatory approval (off-label) or outside of curated evidence-based guidelines (off-guideline).¹ Off-label drug use in the United States is defined as prescribing medicines inconsistent with the label approved by the US Food and Drug Administration (FDA). In addition to FDA labels, the US National Comprehensive Cancer Network has developed the Clinical Practice Guidelines in Oncology (NCCN Guidelines) that describe recommendations for oncology care and are updated regularly. The guidelines often include, in addition to approved drugs, off-label drug usage that is supported by clinical evidence. This allows patients to benefit from the therapies months or years before FDA approval.² We refer to drug uses outside NCCN guideline recommendations as “off-guideline.”

Off-label use in cancer care is widespread and acknowledged by the National Cancer Institute (NCI) and the FDA.^3,4 A literature review of patients from different countries revealed that, as of 2016, 13%–71% of adult patients with cancer received at least one off-label chemotherapy, and that off-label drug use outside standard treatment guidelines was between 7% and 31%.^1,5 Unfortunately, despite the widespread practice of treating patients with unapproved drugs, the clinical outcomes and adverse events associated with off-label or off-guideline therapy usage are not systematically collected or shared.⁶ Moreover, there has been little systematic analysis of off-guideline usage in the United States.

The use of drugs for an unlicensed indication does not necessarily imply a safety hazard. Approved labels are often limited in scope⁵ due to restrictive clinical trial populations.⁷ There are many instances where use outside of the label is uncontroversial and known to be advantageous to the patients. For example, oxaliplatin, which is approved in colorectal cancer, is often used off-label and off-guideline in triple-negative breast cancer patients^8,9,10 and carboplatin, which is not labeled for children, is routinely used for the treatment of solid tumors in pediatric patients.¹¹ Off-label usage enables treating patients diagnosed with specific tumor types where treatment options are limited and allows physicians to use their clinical judgment to determine suitable treatment for an individual patient. For example, in bladder cancer, carboplatin can be more conveniently administered than cisplatin.⁵ Furthermore, once sufficient scientific evidence (e.g., publications showing treatment efficacy and patient safety) is given, approved drugs for one indication may be included in the guidelines for other indications.

Despite the widespread practice of off-label use, the lack of systematic assessment of usage patterns and clinical benefits remains a major concern.¹² Real-world data (RWD) are starting to be used to address the lack of high-quality data needed to characterize the prevalence and clinical outcomes of off-label and off-guideline use. Recent studies have illustrated the power of RWD in assessing the prevalence, effectiveness, and/or safety of off-label drugs in primary central nervous system lymphoma,¹³ breast cancer,^14,15 and melanoma.¹⁶ The FDA has also increasingly adopted RWD to support FDA approval for oncology drugs and label expansions.¹⁷ For example, the CDK4/6 inhibitor palbociclib was first approved in 2015 to treat women with estrogen receptor–positive, HER2-negative breast cancer. In 2019, the FDA expanded the palbociclib indications to include men on the basis of retrospective outcomes analysis of RWD derived from EHRs.¹⁸

However, existing studies leveraging RWD to investigate non-standard cancer treatment usage have several shortcomings. They have only been performed on small sample sizes, a single cancer, or a single drug, and they do not distinguish between off-label and off-guideline usage.^14,19,20 With the advancement of anticancer therapies and the increase of approved drugs, we can expect a continuous increase in the use of off-label and off-guidance drugs over time, particularly in novel combinations.²¹ Hence, a better understanding of off-label and off-guidance usage is critical. The present study is a large-scale RWD analysis to fill this gap and address previous studies’ limitations. Specifically, we systematically analyze the prevalence of off-label and off-guidance drugs across 14 cancer types in 165,912 patients from 280 cancer clinics across the United States. We further perform a temporal analysis to highlight trends of off-guidance usage over time as well as an assessment of a patient’s characteristics that are associated with off-guidance usage. Finally, we provide three use cases to showcase the impactful benefit of large-scale systematic analyses of RWD in the assessment of off-label and off-guidance drug usage.

Results

Summary of data and methodology

We present a framework for analyzing off-label and off-guideline usage of cancer drugs using RWD, and we apply it to study treatment patterns in 165,912 patients with cancer in the United States (Figure 1). This study used the nationwide Flatiron Health (FH) electronic health record (EHR)-derived de-identified database. The FH database is a longitudinal database, comprising de-identified patient-level structured and unstructured data, curated via technology-enabled abstraction^22,23 (see Table S1 for the summary statistics of patient characteristics). The de-identified data originated from approximately 280 cancer clinics (∼800 sites of care) and have been shown to be representative of patients with cancer in the United States²² (Figure S1). The study included patients diagnosed with exactly one of the following 14 cancers: advanced non-small cell lung cancer (aNSCLC) (n = 48,853), metastatic colorectal cancer (mCRC) (n = 22,838), metastatic breast cancer (mBC) (n = 23,477), multiple myeloma (MM) (n = 10,910), chronic lymphocytic leukemia (CLL) (n = 9,389), metastatic pancreatic cancer (mPCa) (n = 8,414), ovarian cancer (OC) (n = 5,220), adult acute myeloid leukemia (AML) (n = 5,876), small cell lung cancer (SCLC) (n = 6,550), advanced bladder cancer (aBCa) (n = 5,774), diffuse large B-cell lymphoma (DLBCL) (n = 6,308), advanced melanoma (aMel) (n = 5,763), follicular lymphoma (FL) (n = 3,318), and hepatocellular carcinoma (HCC) (n = 3,222). We used the cancer classifications established by the FDA and NCCN as a basis to define the cancer types analyzed. The selection of cancer types is limited by the available cohorts curated by FH and made available for researchers. We removed patients diagnosed with multiple cancers (14.1% of the original cohort) from all of the downstream analyses. No other filters were used to include or exclude patients in the de-identified FH database. The data included treatments with 241 antineoplastic drugs (Table S2). For the analysis of predictive biomarkers, we used the nationwide (US-based) de-identified FH-Foundation Medicine clinico-genomic database (FH-FMI CGDB)

Figure 1.

Framework for analyzing off-label and off-guideline treatments using real-world data

(A) For each cancer, we classify drug usage outside of FDA approval as “off-label” and drug usage not in accordance with NCCN Guidelines as “off-guideline.” The analysis framework consists of three components.

(B) First, we systematically identify patients who received off-label and off-guideline treatments based on their EHR. The total number of patients and the proportion of patients with off-label usage and off-guideline usage for 14 cancers are in Flatiron Health EHR-derived database.

(C) We then characterize how a patient’s characteristics are associated with off-guideline usage. Adjusted odds ratio analysis was used to estimate the extent to which patient characteristics were associated with off-guideline usage. We use machine learning to predict the treatment trajectory of patients and off-guideline usage.

(D) Next, we evaluate the effectiveness of off-guideline drugs using survival analysis with propensity score weighting.

We retrieved the FDA-approved drug list for each cancer type from the records of the NCI at the National Institutes of Health (NIH) up to June 20, 2022 (Table S3).²⁴ We used a conservative definition of off-label and off-guideline usage. For each cancer, a drug that is not approved under any condition (e.g., not approved in any dosage) is considered off-label. For example, a drug’s approved indication may be quite specific with regard to the patient subgroup, prior therapies received, or if the drug is given as monotherapy or in combination with another drug. For our analyses, treatments received not meeting these detailed conditions were not counted as off-label usage. The main factor defining off-label use in our analysis was the tumor type. In addition, we retrieved the recommended drug list of NCCN Guidelines for each cancer type from the NCCN Compendia (Table S4). The NCCN Guidelines are recognized as the standard of care for oncology treatment. For each cancer, a drug that is not recommended under any condition by NCCN Guidelines is considered off-guideline.

Overview of off-label and off-guideline usage patterns

Treatment in accordance with NCCN Guidelines is generally regarded as “appropriate” and usually covered by health insurance, even when it is outside of FDA labeling. Thus, we differentiate between off-label use that is and is not included in the NCCN Guidelines. For example, carboplatin does not have official FDA approval for mBC, SCLC, aBCa, DLBCL, aMel, and FL, but is recommended by NCCN Guidelines for those malignancies and commonly administered in clinical practice. In the FH database, 18.6% of patients were treated with at least one off-label drug and 4.4% of patients were treated with at least one off-guideline drug; see Tables S5 and S6 for the statistics of off-label and off-guideline usage in different lines of treatments for each cancer type. From an additional study on patients who were excluded from the primary analysis, we found that patients with two cancers (Table S7) are more likely to receive off-label (19.9%) and off-guideline (4.6%) treatments compared with those with a single cancer (Tables S8 and S9).

Figure 2 summarizes the cross-use of FDA-approved and NCCN-recommended drugs between different cancer types in the FH database. The most commonly administered off-label treatments were carboplatin, etoposide, and bendamustine. The most commonly used off-guideline therapies include leuprolide, pemetrexed, and bevacizumab. We found that the most prevalent off-label usage is in SCLC, with borrowed FDA-approved drugs from aNSCLC and OC—77.6%, 23.2%, and 8.2% of patients with SCLC received carboplatin, cisplatin, and paclitaxel, respectively. The most prevalent off-guideline usages are in mBC and aBCa, with borrowed NCCN-recommended drugs from aNSCLC and OC. Among patients with mBC, 3.8% of them received leuprolide (approved for OC) and 1.9% of them received atezolizumab (approved for aNSCLC). Among patients with aBCa, 4.1% and 1.5% of them received pemetrexed and paclitaxel protein-bound (approved for both aNSCLC and OC), respectively.

Figure 2.

Overview of off-label and off-guideline usage patterns and off-guideline temporal trends

(A and B) The cross-use of (A) approved and (B) recommended drugs for 14 cancer types. The (i, j) grid shows the proportion of patients with cancer i (row i) receiving drugs that are (A) off-label or (B) off-guideline, but (A) approved or (B) recommended by NCCN Guidelines in the cancer j (column j).

(C) Temporal trend of off-guideline usage. For each cancer, we plot the proportion of patients receiving any off-guideline treatment (solid line) and the commonly administered off-guideline drug (dashed line) for each year. Error bars correspond to standard errors of the off-guideline proportions.

Temporal trends

We analyzed the temporal trends of patients receiving off-guideline treatments (Figure 2) and off-label treatments (Figure S2) in each year in the FH database. We focused on the off-guideline analysis in the main text because off-guideline drugs recommended by NCCN Guidelines are generally also regarded as appropriate and are covered by health insurance. Off-guideline usage has been increasing in recent years for mCRC, AML, DLBCL, FL, and HCC, due to the increasing use of off-guideline targeted therapies such as binimetinib and vemurafenib for mCRC and ruxolitinib for AML. Binimetinib was approved by the FDA in 2018 to treat BRAF mutation-positive melanoma. Consistent with this, 94% of the patients with mCRC who received binimetinib and tested for BRAF status in FH tested positive for BRAF mutations. While ruxolitinib is not in the treatment guideline for AML, it has shown promising results in patients with AML in early clinical trials.²⁵ In contrast, the off-guideline usage has decreased for CLL, SCLC, aBCa, and aMel. For example, the off-guideline usage in aBCa dropped from 10% in 2015 to 2% in 2020, mainly due to the approval of the immunotherapy pembrolizumab. After its initial FDA approval in 2014 (for melanoma patients), pembrolizumab was rapidly used for patients with aBCa (Figure 2). Off-guideline drug use for aBCa (e.g., pemetrexed) especially declined after pembrolizumab was approved for aBCa by the FDA in 2017.

Early adoption of immunotherapy

In our dataset, we find that 7.6% of all immunotherapy usages across patients are off-label usages. There is growing interest in characterizing the adoption patterns of new immunotherapies, but relatively little is known about the frequency of immunotherapy usage before FDA approval.²⁶ As a case study, we report the FDA approval history of nivolumab, one of the most commonly used immunotherapies, for different cancer types in Table 1. Additional analysis on other recent immunotherapies is provided in Table S10 and the Supplementary Note. Nivolumab was first approved in aMel in 2014, and its off-label use in other cancer types has been observed since then. For example, nivolumab was not approved for SCLC until August 2018, but 8.3% of patients with SCLC already received nivolumab before this date. As of the writing of this paper, nivolumab is not yet approved for mPCa and we evaluate its off-label use in 2+ lines in the FH database in Table S11. Across different cancer types, we find that a non-negligible fraction of patients received nivolumab in non-approved tumor types before official FDA approval, suggesting enthusiasm for new immunotherapy adoption. We further analyze the early adoption of nivolumab before the publication dates of the key clinical trial results in each of the cancer types (Table S12) and find that there is notable early adoption of nivolumab in mCRC, HCC, aBCa, and mPCa even before the publication of the key trials. The early adoption patients are not concentrated at academic medical centers (Tables 1 and S12) and are treated in many states (Tables S13 and S14), suggesting broad enthusiasm for new immunotherapies. After FDA approval, the adoption of nivolumab increased as expected (Table 1). Some eligible patients may not receive nivolumab due to concerns about adverse reactions to immunotherapies or patient preferences.

Table 1.

FDA approval history for nivolumab

Cancer	Approval date	Before approval			After approval
		Total number of patients	Patients on nivolumab		Total number of patients	Patients on nivolumab
			n (%)	% at academic		n (%)	% at academic
aMel	12/22/2014	1,114	1 (0.1)	0.0	4,866	2911 (59.8)	15.6
aNSCLC	03/04/2015	14,449	3 (0.0)	0.0	36,938	7734 (20.9)	5.7
mCRC	08/01/2017	10,836	53 (0.5)	1.9	14,541	287 (2.0)	8.3
HCC	09/22/2017	1,545	58 (3.8)	9.4	1,802	603 (33.5)	16.2
SCLC	08/16/2018	4,128	342 (8.3)	11.3	2,751	308 (11.2)	7.7
aBCa	08/19/2021	5,456	354 (6.5)	2.6	554	47 (8.5)	10.6
mPCa	Unapproved	8,414	71 (0.8)	3.9	–	–	–

For each cancer type, we report the date when nivolumab is first approved by the FDA, and the number of patients that received nivolumab before and after the approval date in the FH database. We also report the percentage of patients that are treated at academic medical centers.

Characteristics of patients who received off-guideline treatments

We next investigate which patient characteristics are associated with increased usage of off-guideline treatments in the FH database. For each cancer, we fit a logistic regression model with patients’ characteristics to predict whether or not they received off-guideline drugs (STAR Methods). We summarized the odds ratio (OR) of different patients’ characteristics associated with off-guideline usage in Figure 3 and off-label usage in Figure S3. Several clinical characteristics of patients are significantly associated with increased off-guideline usage. Across the 14 cancer types, patients in later lines of treatment and with worse performance status (higher Eastern Cooperative Oncology Group [ECOG] values) are more likely to receive off-guideline treatments. This suggests that patients who are more advanced in their cancer progression, and thus have fewer options, are more likely to be given off-guideline drugs as treating physicians try to identify effective treatment options. Patients treated at academic hospitals are also more likely to receive off-guideline drugs compared with patients treated at community clinics (median adjusted OR = 2.03 across 14 cancer types). This may reflect the fact that academic hospitals are more likely to use experimental treatments. Patients with tumors that are positive for specific biomarkers such as EGFR and KRAS in the FH database tended to have less off-guideline usage, perhaps due to the availability of FDA-approved targeted therapies for patients whose tumors harbor these mutations.

Figure 3.

Characteristics of patients who received off-guideline treatments

(A–H) Adjusted odds ratio with two-sided Wald test p value for off-guideline drug usage for (A) line of treatment, (B) practice type (academic or community center), (C) ECOG value, (D) biomarker status, (E) age in increments of 10 years, (F) gender (male or not), (G) race (non-White or not), (H) insurance category (commercial or not, on the patient’s first insurance record). Here the odds ratios are adjusted for age, gender, race, ECOG, staging, practice type, insurance type, year of receiving treatment, histology (for patients with aNSCLC and OC), smoking status (for patients with aNSCLC and SCLC) and biomarkers status. Error bars correspond to 95% binomial confidence intervals.

Certain patient demographics were also associated with off-guideline usage. Younger patients receive more off-guideline usage. One possible explanation is that clinicians may feel that younger patients can better tolerate off-guideline treatment, which may have more uncertain side effects. Self-reported underrepresented minorities (non-White) are less likely to receive off-guideline treatments in several cancer types (aMel, aNSCLC, CLL, DLBCL, and AML), potentially reflecting historically well-documented racial disparities in cancer care (Figure 3). Such disparities impact the minority patient care experience in myriad ways that may affect off-label and off-guideline use, including reduced access to novel or experimental therapies,²⁷ reduced access to specialist care and resources,²⁸ and lack of information and communication about treatment options.^29,30,31 There is no broad trend between off-guideline usage and insurance type (e.g., Medicare, Medicaid, and commercial insurance), which is also considered a proxy for socioeconomic status. There is no consistent association between gender and off-guideline usage except for mBC. For mBC, around 1% of patients are male and they are significantly more likely to receive off-guideline treatments.³² Because male breast cancer patients are rare, few are included in clinical trials, and recommendations on the clinical management of male breast cancer patients are extrapolated from the population of female breast cancer patients in trials. Generally, although males are treated similarly to females, there are some differences. For example, aromatase inhibitors for mBC are given concurrently with gonadotropin-releasing hormone analogs (e.g., leuprolide) for males, but without those analogs for most women, especially post-menopausal women, according to the NCCN Guidelines for breast cancer.

We further grouped CLL, FL, and DLBCL as non-Hodgkin lymphoma and aNSCLC and SCLC as lung cancer for additional analysis to account for overlap in treatment at specialized care facilities and within related tumor subtypes. The primary off-guideline and off-label usage analysis and the analysis of factors associated with off-guideline usage were repeated for these grouped cancers, and the results were consistent with those obtained for individual cancers (Figures S4 and S5).

It is interesting to quantify how well factors captured in the EHR can holistically predict when a patient might receive off-guideline treatments. When off-guideline usage is predictable, models could potentially alert clinicians for additional treatment options to consider. It is also useful to know when off-guideline usage is difficult to predict, as this quantifies the extent to which factors or randomness beyond the guidelines and standard clinical features influence treatment recommendations. To quantify off-guideline predictability, we developed several machine learning algorithms to model the trajectory of patient treatments. Overall, a transformer neural network achieved the best performance in predicting which patient would receive off-guideline treatments in a particular line. The transformer is first trained with self-supervised learning to predict which drug a patient receives in each line based on the patient’s baseline characteristics (e.g., age, ECOG, staging) and previous lines of treatments (Table S15). The patient embedding from the transformer is then trained to predict off-guideline usage, where the model achieves area under the precision recall curve (AUPRC) of 0.12, 0.20, and 0.28 for first, second, and third lines of therapy, respectively (Table S16). One reason that the AUPRC is relatively low is because the prediction task is heavily imbalanced, with only 4.4% of patients receiving off-guideline treatment (Table S6). The relative proportion of off-guideline usage increases for later lines, consistent with the increase in AUPRC. The modest prediction performance of state-of-the-art machine learning models also suggests that off-guideline usage decisions may be influenced by factors outside of what is typically captured in the EHR or the guidelines.

Systematic RWD analysis of off-label and off-guideline usage can generate valuable insights into how patients respond to these treatments. We next conduct case studies to illustrate these applications.

RWD support the efficacy of hormone therapy on subsets of patients with OC

The role of hormone therapy in treating OC has been proposed in the NCCN Guidelines. Recommended uses have generally been relegated to the less common epithelial OCs such as grade 1 endometrioid and low-grade serous carcinomas for which hormonal therapy is a recommended therapy in the adjuvant and/or maintenance setting. Furthermore, the efficacy of letrozole as an adjuvant monotherapy in comparison to paclitaxel/carboplatin plus letrozole is currently under investigation.³³ Patients deemed to be too frail or ill for cytotoxic chemotherapies are sometimes given hormonal therapies, which also has the benefit of not inducing chemo-resistance.

We evaluated the off-label uses of hormone therapy in OC using the FH database. Hormone therapies are mostly used in second and later lines of treatment for ovarian cancer, where they were associated with better survival compared with non-hormone therapies in the FH database (hormone drugs are listed in Table S17), with adjusted hazard ratio (HR) of 0.56 (95% confidence interval [CI] 0.44–0.72), 0.63 (95% CI 0.50–0.79), and 0.65 (95% CI 0.51–0.83) for letrozole, tamoxifen, and anastrozole, respectively (Table 2). Similar results were also observed in additional analyses of patients who received these same three hormonal therapies as single agents (i.e., monotherapy as opposed to combination therapy) in second and later lines (Table S18). Overall, 15.4% of OC patients receive hormone therapy in second or later lines. After the adjustments of propensity weighting, the two comparative cohorts (patients receiving hormone therapy vs. patients receiving non-hormone therapy) were balanced based on all patient characteristics including demographics, disease conditions, and treatment dates (Figure S6). Additionally, we identified TP53, PRKCI, BCL6, CCND2, RAD52, CDKN1B, FGF6, and FGF23 as potential predictive biomarkers for hormone therapy in OC using the FH-FMI CGDB database (Table S19). For example, patients with TP53 mutations were observed to have worse survival when treated with hormone therapy (adjusted interaction HR 1.63, 95% CI 1.14–2.32) compared with TP53 wild-type patients receiving the same treatment.

Table 2.

Evaluation of hormone therapy in ovarian cancer for 2L+

Drug	Number of patients	Control	n control	HR (95% conf)
Letrozole	160	Non-hormone treatments	2,438	0.56 (0.44–0.72)
		Standard of care - Doxorubicin Pegylated Liposomal	522	0.46 (0.35–0.60)
Tamoxifen	143	Non-hormone treatments	2,438	0.63 (0.50–0.79)
		standard of care - Doxorubicin Pegylated Liposomal	522	0.50 (0.39–0.64)
Anastrozole	120	Non-hormone treatments	2,438	0.65 (0.51–0.83)
		Standard of care - Doxorubicin Pegylated Liposomal	522	0.54 (0.41–0.71)

Here the experiment arm consists of patients who received treatments containing the hormone therapies (e.g., letrozole alone or in combination with other drugs). The control arm includes patients (1) who received treatments not containing any hormone agents; (2) standard of care, defined as the most commonly used treatment. Hazard ratios are adjusted for age, gender, race, ECOG, stage, histology and molecular alterations, practice type, insurance type, year of receiving treatment.

RWD provides evidence suggesting ineffective immunotherapy combination, consistent with clinical trials

Ipilimumab and nivolumab are both immune checkpoint inhibitors that have complementary mechanisms of action. Nivolumab was approved for SCLC in 2018; mounting evidence illustrated that the addition of ipilimumab to nivolumab was more effective than nivolumab monotherapy for NSCLC,³⁴ but until recently the efficacy of the combination for SCLC was unclear. In our study of the FH database, we find that patients with SCLC who received nivolumab and ipilimumab (n = 214) did not show improved overall survival compared with patients who received only nivolumab (n = 381) in 2+ lines with adjusted HR of 1.03 (95% CI 0.85–1.25). Consistent with our findings on real-world patients, recent clinical trials for SCLC reported that overall survival was similar between the group treated with nivolumab alone and the group treated with nivolumab plus ipilimumab,^35,36 and the nivolumab plus ipilimumab combination was removed from the SCLC guideline in 2021. This shows how RWD analysis could generate hypotheses for clinical trials and potential guideline changes.

Discussion

We present a data science framework to leverage EHR-derived RWD to characterize off-label and off-guideline treatment patterns. Application of this framework to 14 common cancer types in 165,912 US patients demonstrates that prescribing a drug outside its official labeling and practice guidelines is relatively widespread in oncology. To the best of our knowledge, this is the largest study investigating off-label and off-guideline usage of anticancer drugs using real-world data. As more and higher-quality RWD become available, our computational framework could be expanded beyond the FH data.

We identified that certain patient characteristics were associated with increased likelihood of receiving off-guideline treatments: younger age, academic settings, and the receipt of later lines of therapy. Additionally, our results suggest that underrepresented minorities are less likely to receive off-guideline treatments. Few studies have explored racial disparities in the context of off-label usage, in particular in the context of cancer; however, one psychiatry study found that minority patients were less likely to fill off-label prescriptions than White patients when restricted to private insurance.³⁷ Still, further research and more high-quality data are needed to fully explore the dynamics at play and whether these disparities in off-label and off-guideline treatment imply racial disparities in quality of patient care.

Multiple factors can contribute to off-label or off-guideline usage of certain cancer treatments. For example, many chemotherapies are older drugs that received approval in one cancer indication many years ago, but manufacturers of those drugs may lack incentives to conduct the additional clinical trials needed to pursue official FDA approval for other tumor types.²⁴ Moreover, cytotoxic chemotherapies have the same overall mechanism of stopping fast-reproducing cancer cells, so subclasses of these chemotherapies are sometimes used interchangeably by oncologists even if only one drug received regulatory approval (e.g., cisplatin and carboplatin are both platinum-based therapies). In addition, cancer treatments are often given in combination (e.g., combination of chemotherapies such as FOLFOX and FOLFIRI for mCRC). Since it is challenging for the FDA to try to approve all chemotherapy combinations, a combination chemotherapy regimen could contain one or more chemotherapies that are not approved for the tumor type being treated.³ Despite its prevalence, the safety and efficacy of off-label and off-guideline cancer treatments have not been clearly established in many settings. Additional post-market analysis and monitoring are critical. In addition, our machine learning analysis shows that off-guideline usage has low predictability, suggesting that complex factors beyond treatment guidelines and clinical information commonly captured in the medical records may influence decisions to use off-guideline therapies. This finding also highlights the research benefit of collecting additional features (e.g., patients’ history of side effects) that can contribute to the selection of non-standard treatments.

Limitations of the study

There are several limitations with RWD to be further explored in future works. First, we emphasize that the goal of this study is to characterize usage patterns of cancer treatments; our findings should not be used to suggest treatments. Although we used state-of-the-art propensity adjustment methods to mitigate selection bias, there could be unmeasured confounders and residual confounding in RWD. For example, our findings of the benefits of hormone therapy may be confounded because tumor grade information was not collected in the FH database, and low-grade OC tends to associate with estrogen receptor/progesterone receptor expression and better prognosis than high-grade OC. In addition, detailed safety analyses were challenging due to the sparsity of related endpoints in the FH database. Moreover, we focus our analysis on drugs captured in the FH database. While we excluded patients who were known to have more than one malignancy, it is possible that some patients who were included in the analysis had multiple malignancies but with the diagnosis code for the other malignancy not available in the electronic medical record. To mitigate some of these missing data challenges in this work, we used a strict definition of off-label or off-guideline drug—a drug that is not approved or recommended under any condition for a specific cancer type. Moreover, the FH data used here mostly captures patients with advanced or metastatic cancers, and therefore our findings reflect treatment patterns in those cancers and should be interpreted as such. Further analysis of treatments in early-stage cancers would be an interesting complement of this work. The official labels and practice guidelines may also include specific requirements and conditions, such as usage in a special subpopulation, in a specified dosage, and through a specific administration route. Analysis of drug usage not in accordance with detailed approval and recommendation conditions is an important future direction.

This paper demonstrates that large-scale analysis of high-quality RWD can help to characterize the landscape of off-label and off-guideline treatment usage. This kind of analysis has the potential to provide quantitative insights into the usage patterns of off-label and off-guideline treatments that can then be used to inform research and development efforts.

STAR★Methods

Key resources table

REAGENT or RESOURCE	SOURCE	IDENTIFIER
Software and algorithms
We newly generated the code for our proposed computational analysis framework.	The open source Python code for our algorithm is available at https://github.com/RuishanLiu/off-label.	N/A

Resource availability

Lead contact

Further information and requests for resources and reagents should be directed to and will be fulfilled by the lead contact, James Zou (jamesz@stanford.edu).

Materials availability

This study did not generate new unique materials.

Data and code availability

The data used in this study were licensed from Flatiron Health (https://flatiron.com/real-world-evidence/) and Foundation Medicine. These de-identified data can be made available upon request by contacting DataAccess@flatiron.com and cgdb-fmi@flatiron.com. The open source Python code for the computational analysis framework is available at https://github.com/RuishanLiu/off-label. Any additional information required to reanalyze the data reported in this work paper is available from the lead contact upon request.

Method details

Flatiron Health database

The data that support the findings of this study come from the nationwide EHR-derived de-identified Flatiron Health database (March 2022 datacut). The Flatiron Health database is a longitudinal database, comprising de-identified patient-level structured and unstructured data, curated via technology-enabled abstraction.^22,23 The Flatiron Health data is considered one of the industry’s leading research databases in oncology due to the rigorous data curation and abstraction processes as well as publications demonstrating their efforts to validate outcomes. In previous validation studies comparing the Flatiron Health mortality data to the gold standard National Death Index, the sensitivity of mortality capture in an aNSCLC population was shown to be 91%, and the impact of remaining missing deaths on survival analyses was minimal.^38,39,40 In addition to curation accuracy, the Flatiron Health data is harmonized and aggregated across approximately 280 cancer clinics across the country, which enables its data to be more representative than single healthcare EHR. The majority of patients in the database originate from community oncology settings; relative community/academic proportions may vary depending on study cohort. Data provided to investigators was de-identified and subject to obligations to prevent re-identification and protect patients’ confidentiality. The data that support the findings of this study originated by Flatiron Health, Inc. and Foundation Medicine, Inc. Requests for data sharing of these de-identified data by license/by permission for the specific purpose of replicating results in this manuscript can be submitted to dataaccess@flatiron.com. Portions of research conducted by Flatiron Health were approved with waiver of informed consent by the WCG Institutional Review Board prior to study conduct.

FH-FMI CGDB data

The Flatiron Health database is linked with the comprehensive genomic profiling results from Foundation Medicine (FMI) as the de-identified Flatiron Health-Foundation Medicine clinico-genomic database (FH-FMI CGDB, datacut December 31, 2022). We used the FH-FMI CGDB database to identify potential genetic biomarkers for hormone therapy in OC. Retrospective longitudinal clinical data were derived from electronic health record (EHR) data, comprising patient-level structured and unstructured data, curated via technology-enabled abstraction, and were linked to genomic data derived from FMI comprehensive genomic profiling (CGP) tests in the FH-FMI CGDB by de-identified, deterministic matching.⁴¹ The FH and FMI data match was conducted by an independent third party.⁴¹ Data provided to investigators were de-identified and subject to obligations to prevent re-identification and protect patients’ confidentiality. Portions of research conducted by Flatiron Health were approved with waiver of informed consent by the WCG Institutional Review Board prior to study conduct.

Genomic alterations were identified via comprehensive genomic profiling (CGP) of more than 300 cancer-related genes on FMI’s next-generation sequencing (NGS) tests.^42,43,44 Eligible patients must receive the Foundation Medicine test on a tumor sample with pathologist-confirmed histology that is consistent with Flatiron Health abstracted tumor type, and their Foundation Medicine specimen collection date must be no earlier than 30 days after the initial diagnosis date in Flatiron Health. All the patients have at least two documented clinical visits in the Flatiron network on or after January 1, 2011. This study included 3,576 patients diagnosed with OC (removed patients diagnosed with multiple cancers) who underwent FMI CGP testing.

Identification of off-label drugs

The data included treatments with 241 antineoplastic drugs (Table S2). The drug biosimilars are grouped together (Table S20). The FDA approved drug list for each cancer type was retrieved from the records of the National Cancer Institute (NCI) at the National Institutes of Health (NIH) (data up to June 20, 2022) (Table S3).²⁴ We use a conservative definition of off-label usage. For each cancer, a drug that is not approved under any condition is considered off-label.

Identification of off-guideline drugs

The recommended drug list of NCCN Guidelines for each cancer type was retrieved from the NCCN Compendia (Table S4). The NCCN Guidelines are recognized as the standard for clinical direction. Similar to the conservative definition of off-label usage, a drug that is not recommended under any condition for a given cancer type by NCCN Guidelines (October 2022 version) is considered off-guideline.

Removal of patients with multiple cancers

For rigorous off-label usage analysis, we focused on patients with only one cancer. We filtered patients in two steps. First, we reviewed the diagnosis records for all patients and removed those with ICD diagnosis codes in multiple cancers. Apart from the 14 cancers in our analysis, we also checked for and removed patients with diagnosis codes for chronic myelogenous leukemia (CML), acute lymphocytic leukemia (ALL), cutaneous T cell lymphomas (CTCLs), myelofibrosis (MF), renal cell carcinoma (RCC), head and neck cancer, gastric cancer, malignant neoplasm of brain, prostate cancer, and endometrial cancer. Second, we removed all the patients who qualified as patients in multiple Flatiron Health datasets for different cancer types. Altogether, 14.1% of patients with multiple cancers were excluded.

Second cancer analysis

We conducted a second cancer analysis on patients who were excluded from the primary analysis. The second cancers are identified from the presence of ICD diagnosis codes for a tumor type that differed from the patient’s tumor-specific de-identified Flatiron Health database. For example, if a patient was enrolled in the de-identified Flatiron Health aNSCLC database but had a diagnosis code for OC, we considered this as a case of second cancer. Table S7 summarizes the number of patients diagnosed with precisely 1 s cancer in each tumor-specific Flatiron Health dataset. This method which does not involve manual chart review may result in some misclassification. Overall, patients with two cancers are more likely to receive off-label and off-guideline treatments than patients with a single cancer — 19.9% patients were treated with at least one off-label drug and 4.6% of patients were treated with at least one off-guideline drug; see Tables S8 and S9 for the statistics of patients with different second cancer types.

Cancer grouping analysis

The FDA and NCCN considers aNSCLC and SCLC as distinct cancer types for treatment purposes, and our primary analyses followed this classification. In the grouped analysis (Figures S4 and S5), we consider a drug to be off-label (off-guideline) for lung and NHL if it is not approved (in the guideline) for any of the cancers under lung or NHL. As expected, the proportion of off-label and off-guideline usages is lower for these two groups due to the more restrictive requirement.

Odds ratio evaluating associations between patient characteristics and off-label usage

The adjusted odds ratio (aOR) between each covariate and off-label usage is defined as the exponential of the logistic regression coefficient adjusted for baseline confounders. The categorical variables are encoded as dummy variables and continuous variables are standardized. We control for the following baseline confounders: age, gender, race, ECOG, staging, year of receiving treatment, practice type, insurance type in the patient’s first insurance record, histology (aNSCLC and OC), smoking status (aNSCLC and SCLC) and biomarkers status (Table S21). When different lines of treatments are analyzed together, we also include line number as a confounder. The ECOG value and biomarker status are selected as the closest value assessed within a −180 to +7 days window around the start date of the line. All the baseline confounders are one-hot encoded for categories with at least 5% of patients. To study the association between one covariate and off-label (off-guideline) usage for each cancer, we fit a multivariate logistic regression model with the covariate of interest $X_1$ and baseline confounders $X_j$ ($j=2,...m$) to predict whether or not a patient received off-label (off-guideline) drugs.

$$y\left(X\right)=\frac{1}{\left(1+\mathit{exp}\left(-{\alpha }_0-{\alpha }_1{X}_1-\sum _{j=1}^m{\alpha }_j{X}_j\right)\right)}\text{,}$$ (Equation 1)

Here $y$ indicates whether the patient is off-label (off-guideline) or not. The antilog of the regression coefficient $\mathit{exp}\left({\alpha }_1\right)$ produces the odds ratio for the corresponding variable. Patients with missing values specifically in the covariate of interest were removed from the analysis, and missing values in other baseline confounders were treated as a separate category labeled as "missing".

Quantifying the predictability of off-guideline drug usage

In order to quantify the predictability of off-guideline drug usage, we trained several machine learning models to predict which patients would receive off-guideline treatment in each line. Here we focus on off-guideline rather than off-label, because off-guideline treatments are clinically more interesting and are also more challenging to predict. We interpret the best performance of the machine learning models as approximating the predictability of off-guideline usage. The best performing model overall is a transformer-based language model that we developed to model the trajectory of patient treatments.⁴⁵ This model considers drugs and patient characteristics as individual tokens. For example, the drug “carboplatin” and gender “male” are fed to the language model as tokens. We define the end of each line, death and censoring as special tokens. The task of the language model is to predict which drug is likely to follow a sequence of tokens; that is, to predict the probability of a patient receiving a given drug considering their characteristics and all of the prior treatments. For each line of treatment, the drugs are sequenced alphabetically. The sequence of tokens are passed to an embedding layer with dimension of 200 and a positional encoding layer with dropout rate of 0.2. A square attention mask is used to mask out all the tokens in the future positions. We use two layers of transformer encoder with two attention heads and feedforward network with dimension of 200. The output of the transformer (learned embeddings) is passed to a linear layer with log softmax. We train the model for 100 epochs and use a batch size of 20 and stochastic gradient descent with a learning rate starting from 5 and decaying 5% every epoch.

To evaluate the performance of the transformer in predicting which drugs a patient takes in each line, we input the patient characteristics and sequence of treatments prior to a given line to the model. The drug within the given line is predicted iteratively until the prediction of end of the line, death or censoring. The accuracy of predicting the drug combinations in each line is defined as

$$accuracy=2\times |S\cap S_{true}|/(|S|+|S_{true}|)\text{,}$$ (Equation 2)

where $S$ ($S_{true}$) is the learned (true) set of drugs in the given line of treatment.

To predict the off-guideline usage in each line, we input the patient embedding learned by the transformer to a logistic regression model. We compare the transformer’s performance with a logistic regression model trained on the original patient characteristics. Other machine learning models, such as random forest with 100 trees, trained on patient characteristics have similar performance as logistic regression. In the experiments, we randomly split the patients into training, validation and test set, with ratio 70%, 15% and 15%, respectively. We fit the model on the training set, use the validation set for early stopping, and report the model’s performance on the test set. The result is averaged over three independent runs.

Performance of off-label drugs

To evaluate the performance of one off-label drug for one cancer type, we compare the survival behavior of two groups — the experiment group with all the patients on treatments containing that drug (i.e., drug given alone or in combination with another drug treatment) and the control group with all the patients not receiving that drug. We used Cox proportional-hazards model to assess hazard ratio and its 95% confidence interval. To perform survival analysis, we set the index date (time zero) as the start date of the corresponding line of treatment. Patients were followed until death for real world overall survival (OS) analysis. The censoring dates are chosen as their last visit dates to the clinics.

To obtain unbiased estimates, we used inverse probability of treatment weighting (IPTW) to adjust for baseline confounding factors. In the survival analysis, patient $i$ is weighted by in Equation 3, where $e^{\left(i\right)}$ is the propensity score for patient $i$ and $Z^{\left(i\right)}$ is an indicator to represent whether patient $i$ is in the experiment group or not. For example, when analyzing the prognostic effect of gene mutation on overall survival, patients with a mutated gene of interest compose the experiment group ($Z^{\left(i\right)}=1$) and patients without mutations in that gene are in the control group ($Z^{\left(i\right)}=0$). The propensity score $e^{\left(i\right)}$ is estimated by a logistic regression model in Equation 4, where $X_j^{\left(i\right)}$ indicates the $j$-th baseline covariate for patient $i$. The $n_X$ confounders are effectively balanced between the experiment and control groups after adjusting by propensity score.

$$=Z^{\left(i\right)}/e^{\left(i\right)}+{(1-Z}^{\left(i\right)})/\left(1-e^{\left(i\right)}\right)$$ (Equation 3)

$$e^{\left(i\right)}=\mathit{Pr}\left(Z^{\left(i\right)}=1|X_1^{\left(i\right)},X_2^{\left(i\right)},...,X_{n_X}^{\left(i\right)}\right)$$ (Equation 4)

The baseline confounders $X$ is the same as used in odds ratio analysis.

Ovarian cancer analysis

For ovarian cancer patients in the Flatiron Health database, 2L therapy represents the 2nd regimen received after the patient’s initial ovarian cancer diagnosis. This differs from the traditional definition of line of therapy, where LoT is specifically 1st treatment regimen, 2nd treatment regimen, etc that a patient receives after their advanced or metastatic diagnosis.

Predictive biomarker identification

The interactive Cox proportional hazards model between one gene mutation and one treatment includes gene mutation status $G$, treatment status $T$, a gene-by-treatment interaction term $G\times T$ and the baseline confounders $X_j$, as defined in Equation 5.

$$h\left(t\right)=h_0\left(t\right)\mathit{exp}\left({\beta }_{G}G+{\beta }_{T}T+\beta G\times T+\sum _{j=1}^{n_X}{\beta }_{X_j}{X}_j\right)$$ (Equation 5)

Here $h\left(t\right)$ is the hazard function at time $t$, $h_0\left(t\right)$ is the baseline hazard. The antilog of the regression coefficient $\mathit{exp}\left({\beta }_G\right)$, $\mathit{exp}\left({\beta }_T\right)$, $\mathit{exp}\left(\beta \right)$ and $\mathit{exp}\left({\beta }_{X_j}\right)$ produce hazard ratios for the corresponding variables. A gene-by-treatment interaction HR $\mathit{exp}\left(\beta \right)$ larger than 1 indicates that the treatment effect is worse in patients whose tumors harbor a mutation in that gene, compared to patients without that tumor genetic mutation. Here we use the same set of confounders $X$ as in the odds ratio analysis. To avoid the immortal time bias that may arise from death before getting the chance to take genomic profiling tests,⁴⁴ we penalized the Cox proportional hazards model with left-truncation method with the dates when patients receive the FMI tests.⁴⁶,⁴⁷

Quantification and statistical analysis

The statistical analysis use Python packages Numpy (version 1.20.0), Pandas (version: 1.2.1), scikit-learn (version 0.24.1), lifelines (version 0.26.0), statsmodels (version 0.12.2), and custom Python scripts. All the statistical details of experiments can be found in the figure legends, the results section of the main text and STAR Methods. We use Cox proportional-hazards model to assess hazard ratio. The 95% confidence intervals for the hazard ratios are calculated using the variance matrix of the coefficients obtained from the Cox proportional-hazards model fitting. The model controls for the following baseline confounders: age, gender, race, ECOG, staging, year of receiving treatment, practice type, insurance type in the patient’s first insurance record, histology (aNSCLC and OC), smoking status (aNSCLC and SCLC) and biomarkers status. Our balance assessment shows that after controlling for these confounders, the patients cohorts are effectively balanced across these attributes. For odds ratio analyses, we use the two-sided Wald test to compute p values. The analyses were conducted on 165,912 U.S. patients with cancer. All patients must receive the Foundation Medicine test on a tumor sample with pathologist-confirmed histology that is consistent with Flatiron Health abstracted tumor type, and their Foundation Medicine specimen collection date must be no earlier than 30 days after the initial diagnosis date in Flatiron Health. All the patients have at least two documented clinical visits in the Flatiron network on or after January 1, 2011.

Published: February 29, 2024