Comparative role of real-world study and traditional randomized controlled trials in head and neck cancer: a literature-based analysis

To the Editor: In the era of evidence-based medicine, the traditional randomized controlled trials (RCT) is on the top of the pyramid of evidence hierarchy. With well-controlled inclusion and exclusion criteria, randomization, and strict intervention protocol, traditional RCT construct the ideal medical circumstance, under which the causal relationship between the outcome and intervention can be better interpreted. However, it will also make the external validity and generalizability of traditional RCT limited.^[1,2] Real-world study (RWS) refers to research that analyzes real-world data, including registries, databases, electronic health records, claims or insurance databases, and other health data outside the traditional controlled trial settings.^[3] Comparative effectiveness research (CER), the counterpart of traditional RCT in RWS, refers to the studies comparing the effectiveness and safety of an intervention in real-world clinical settings. CER, composed of observational studies and pragmatic RCT, has become a promising source of evidence that complements traditional RCT.^[4] Unlike traditional RCT, pragmatic RCT attempts to perform trials in real-world clinical settings to extend the generalizability of RCT.^[5]

The traditional RCT in head and neck cancer try to answer the questions of great concern, but with limited range of coverage.^[6] Answers to more questions are expected to be mined from real-world data. However, to what extent can RWS in head and neck cancer complement traditional RCT and extend the range of coverage of clinical questions still remains unknown. Moreover, CER and traditional RCT testing the same clinical question might yield inconsistent statistical significance and estimates of effect. To better interpret the results, whether the type of clinical research has an impact on evidence generation should be further explored.

This article aimed at providing a comprehensive analysis of the role of RWS in clinical evidence generation in head and neck cancer compared with traditional RCT, and identifying potential factors associated with evidence generation by reviewing RWS and traditional RCT publications in head and neck cancer from PubMed between 2010 and 2020. Awareness of the role of current RWS in head and neck cancer will be essential for creating, interpreting, and applying real-world evidence.

We collected publications between 2010 and 2020 from PubMed. To search for head and neck cancer, the MeSH term “Head and Neck Neoplasms” and the terms “neoplasm”, “tumor”, “cancer”, or “carcinoma” plus “larynx”, “glottic”, “pharynx”, “hypopharynx”, “oropharynx”, “nasopharynx”, “lip”, “oral”, “paranasal”, “nasal”, “sinus”, “salivary”, “parotid”, and their derivatives were used.^[7] We searched for RWS using the terms “database”, “registry”, “real-world”, “claim”, “electronic health records”, “medical records”, “pragmatic”, and their derivatives, and used “phase”, “randomized”, “randomization”, and “random” to search for traditional RCT. The most recent search was in February 2020. After carefully reviewing the titles and abstracts, we excluded publications other than traditional RCT or RWS in head and neck cancer. A total of 1979 RWS, including 256 CER, and 164 traditional RCT in head and neck cancer were included for analysis.

Related information was extracted from the abstracts and titles by two oncologists (Guang-Li Zhu and Cheng Xu); discrepancies were solved by consensus or by referring to the third oncologist (Jun Ma).

Descriptive analysis was used to summarize the characteristics of traditional RCT and RWS. Logistic regression analysis was performed to identify factors associated with CER and traditional RCT evidence generation. Evidence generation is defined as the report of at least one statistically significant result in a study. The logistic regression model included the study type (traditional RCT or CER), sample size, follow-up time of the endpoints, number of endpoints, and whether the study included overall survival (OS) as an endpoint. The statistical analysis was performed using SPSS (version 24.0, SPSS Inc., Chicago, IL, USA).

As a promising complement to traditional RCT, CER extend the generalizability of traditional RCT by designing the study in the real-world clinical settings. In recent years, an increasing number of regulatory agencies have taken CER into account in decision-making.^[8,9] An obviously increasing trend was observed in the number of traditional RCT and CER publications in head and neck cancer. There was a surge in CER publications after 2016 when the 21st Century Cures Act was initiated in the United States. On the contrary, the number of traditional RCT publications only increased between 2010 and 2014, and then remained relatively stable.

CER and traditional RCT differed in the distribution of the cancer type and treatment modalities. Unspecified head and neck cancer ranked first in both CER (18.8%) and traditional RCT (43.3%), followed by thyroid cancer (18.4%) and laryngeal cancer (13.7%) in CER, and head and neck squamous cell cancer (20.1%) and nasopharyngeal cancer (18.3%) in traditional RCT. As for treatment modalities, although radiotherapy (35.1% vs. 30.3%) and chemotherapy (19.7% vs. 36.8%) comprised a large proportion of the treatment techniques evaluated in CER and traditional RCT, the remaining treatment techniques varied greatly. Surgery and radioactive iodine were mainly evaluated in CER, while adenovirus gene therapy was only evaluated in traditional RCT. In addition, targeted therapy (9.0%) and immunotherapy (12.4%) were also important parts in traditional RCT.

The disparities observed in the treatment modalities and types of head and neck cancer limited the function of CER in complementing traditional RCT. Moreover, it was reported that only 15.0% of the clinical trials could be replicated using observational data.^[10] Thus, pragmatic RCT is the more ideal alternative to traditional RCT than observational CER, especially for experimental treatments, such as targeted therapy and immunotherapy. But pragmatic RCT is still rare, as only three pragmatic RCT were identified in the present study, which all compared non-anti-cancer treatments. Although PRECIS-2 (PRagmatic Explanatory Continuum Indicator Summary 2) has been developed for guiding pragmatic RCT design and evaluation,^[11] it can only be used at the design stage of the protocol, as using PRECIS-2 involves comparing the clinical settings in the trials and that of routine care, which change over time and place, and it is difficult for readers to evaluate or compare the pragmatism of RCT.

In addition, the data source of RWS of head and neck cancer exhibited an unequal geographical distribution. Over half of the data were from the United States (56.6%), followed by China (11.3%) and Denmark (3.1%). Data from Africa (0.5%) and South America (0.7%) were rare; data from more than one country (2.0%) were also limited. The geographical distribution of the data source was apparently discordant from the geographical distribution of the head and neck cancer. It seems reasonable that the results of these RWS are applied to wider populations of the United States. However, if these results are applied to the other countries, the applicability would be questionable. Routine care varies between countries. Sociodemographic and socioeconomic characteristics are also important factors influencing the intervention's efficacy and safety.

In the evaluation of treatment or intervention, the inconsistent results between traditional RCT and CER testing the same clinical question are common, making it hard to interpret the efficacy or safety of interventions. In head and neck cancer, CER (78.2%) had a higher evidence generation rate than traditional RCT (54.1%).

CER and traditional RCT differ greatly in study design. CER (median: 1241, IQR: 255–4068) had much larger sample sizes than traditional RCT (median: 176, IQR: 84–345). Regarding endpoints, 32.0% of CER analyzes more than one endpoint, while the same was true for 68.3% of traditional RCT. OS ranked first in both study types. OS was included in the analysis of 67% of CER, as compared with traditional RCT (53.0%). In addition to OS, traditional RCT preferred progression-free survival (21.3%), complication or toxicity of treatment (17.7%), locoregional control rate (9.0%), and disease-free survival (6.9%), while RWS preferred disease-specific survival (9.0%) and mortality (6.6%). The follow-up time of endpoint was also higher in CER (4.9 ± 2.3 years) compared with traditional RCT (4.0 ± 2.7 years) (P = 0.01).

Table 1 shows the results of logistic regression analysis of the impact of study type, sample size, follow-up time of endpoints, number of endpoints, and whether the study included OS as an endpoint on evidence generation. Compared with traditional RCT, CER was more likely to generate evidence (adjusted odds ratio [OR] = 7.088, 95% confidence interval [CI]: 2.511–20.009, P < 0.001). The number of endpoints (adjusted OR = 1.724, 95% CI: 1.047–2.83, P = 0.032) and the inclusion of OS as a study endpoint (adjusted OR = 0.317, 95% CI: 0.117–0.860, P = 0.024) were also independent factors that influenced evidence generation. There was no statistically significant impact of sample size (adjusted odds ratio [OR] = 0.975, 95% confidence interval [CI]: 0.914–1.026, P = 0.155) and follow-up time of endpoints (adjusted odds ratio [OR] = 0.928, 95% confidence interval [CI]: 0.799–1.079, P = 0.331) on evidence generation rate.

Table 1.

Logistic regression analysis of the generation of evidence of real-world study and traditional randomized controlled trials in head and neck cancer.

	Unadjusted		Adjusted
Variable	OR	P	OR	P
Type
RCT	1.000		1.000
CER	3.053 (1.970–4.730)	<0.001	7.088 (2.511–20.009)	<0.001
Sample size	1.012 (0.922–1.087)	0.611	0.975 (0.914–1.026)	0.155
Number of endpoints	0.882 (0.400–1.111)	0.286	1.724 (1.047–2.830)	0.032
Follow–up time of endpoints	1.036 (0.908–1.181)	0.601	0.928 (0.799–1.079)	0.331
Inclusion of OS as an endpoint in analysis
No	1.000		1.000
Yes	1.287 (0.931–1.992)	0.258	0.317 (0.117–0.860)	0.024

RCT: Randomized controlled trials; CER: Comparative effectiveness research; OR: Odd ration; OS: Overall survival.

In this study, we found that proportion of studies reporting at least one statistically significant results was higher in CER compared with traditional RCT. For CER, the accuracy, completeness, reliability, and transparency of real-world data are usually questioned.^[12] Moreover, most CER are retrospective, and therefore mainly utilize post hoc analysis and lack randomization, which would lead to bias and compromise its internal validity inevitably.^[13] Thus, when results are inconsistent between traditional RCT and CER testing a same clinical question, people usually tends to believe RCT. However, we found that study type is an independent factor associated with the difference in evidence generation rate, after controlling the sample size, follow-up time of endpoints, number of endpoints, and whether the study included OS as an endpoint. Although traditional RCT are believed to have better reliability, inconsistent results from CER cannot be proven wrong, but should be interpreted more carefully.

Apart from complementing traditional RCT which evaluates the intervention's efficacy and safety only, RWS generates evidence with a wider range of coverage. Most RWS aimed at evaluating distribution or characteristic of disease (29.8%), prognostic factors of survival event (26.8%), effectiveness of treatment or intervention (15.6%), and risk factors or causes of non-survival events (14.0%). In the RWS with only descriptive analysis, most of the focus was on incidence or prevalence or mortality or survival (36.1%), sociodemographic characteristics (20.7%), clinicopathological characteristics (15.9%), and utility of treatment (9.6%). In RWS evaluating the prognostic factors of survival events, the most prevalent prognostic factors for analysis included the clinicopathological characteristics (17.6%), biomarkers (11.3%), and sociodemographic characteristics (8.1%). Of these studies, 40.3% conducted extensive exploration with multiple indicators. These evidences contribute to our understanding of disease and healthcare, a large proportion of which are beyond the reach of traditional RCT.

Funding

This study was supported by grants from the National Natural Science Foundation of China (No. 81930072), the Key-Area Research and Development Program of Guangdong Province (No. 2019B020230002), the Natural Science Foundation of Guangdong Province (No. 2017A030312003), the Health & Medical Collaborative Innovation Project of Guangzhou City, China (No. 201803040003), the Innovation Team Development Plan of the Ministry of Education (No. IRT_17R110), and the Overseas Expertise Introduction Project for Discipline Innovation (111 Project, No. B14035)

Conflicts of interest

None.