Fabio Conforti and colleagues did a meta-analysis1 to assess heterogeneity in efficacy of immune checkpoint inhibitors between men and women. The primary outcome was overall survival. The authors included 20 randomised controlled trials in their meta-analysis, in which the hazard ratio (HR) was used to quantify the treatment effect for each study and sex. A random-effects model was used to obtain a pooled HR for the overall treatment effect for each sex. Therefore, the 20 individual HRs were assumed to be a random sample from a hypothetical, super-population (ie, HRs from all current and future comparative trials of this type of intervention in similar patient populations) with a log-normal distribution. The pooled HR was 0·72 (95% CI 0·65–0·79) for men and 0·86 (0·79–0·93) for women, in favour of immune checkpoint inhibitors. The overall survival benefit from immunotherapy was significantly higher for men than women. However, it is not clear how to interpret this difference clinically.
The use of HRs to assess overall treatment effect has several limitations. First, this type of immunotherapy often has a delayed treatment effect, indicating that a HR is not an adequate summary measure of treatment benefit and is difficult to interpret clinically.2–4 Second, if the assumption of log-normally distributed study-level HRs is not valid, the random-effects estimate might not be valid. Third, the assumption that the selected studies represent a random sample from a super-population of studies is not well-defined or easily understood.5 Even if each study-specific HR is an appropriate summary measure, the pooled HR should not be interpreted as the HR for any specific patient population. Moreover, it is not obvious how to justify whether this super-population of HRs would be applicable to patient populations that are included in future studies. In a future study of a similar population, the true study-level treatment effect might differ substantially from the pooled estimate since the pooled estimate is an average across all theoretically possible study-level treatment effects.
The first fundamental principle in a comparative clinical study is to define the patient population of interest, then to identify a population-level summary measure for quantifying the treatment effect, and finally to collect data that enables inferences to be made about the treatment effect in that population. It is not appropriate to rely on a single, clinically uninterpretable, averaged treatment effect, estimated via meta-analysis, for decision making in any specific population, especially when the treatment effects might vary substantially across different target populations. An approach that combines treatment effects across heterogenous study populations, which does not require strong modelling assumptions, might be more appropriate.5
Footnotes
We declare no competing interests.
Contributor Information
Brian Claggett, Division of Cardiovascular Medicine, Brigham and Women’s Hospital, Boston, MA, USA.
Lu Tian, Department of Biomedical Data Science, Stanford University School of Medicine, Palo Alto, CA, USA.
Zachary R McCaw, Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA 02115, USA.
Masahiro Takeuchi, Department of Clinical Medicine (Biostatistics), Kitasato University School of Pharmacy, Tokyo, Japan.
Lee-Jen Wei, Department of Biostatistics, Harvard T.H. Chan School of Public Health, Boston, MA 02115, USA.
References
- 1.Conforti F, Pala L, Bagnardi V, et al. Cancer immunotherapy efficacy and patients’ sex: a systematic review and meta-analysis. Lancet Oncol 2018; 19: 737–46. [DOI] [PubMed] [Google Scholar]
- 2.Uno H, Claggett B, Tian L, et al. Moving beyond the hazard ratio in quantifying the between-group difference in survival analysis. J Clin Oncol 2014; 32: 2380–85. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Pak K, Uno H, Kim DH, et al. Interpretability of cancer clinical trial results using restricted mean survival time as an alternative to the hazard ratio. JAMA Oncol 2017; 3: 1692–96. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Alexander BM, Schoenfeld JD, Trippa L. Hazards of hazard ratios-deviations from model assumptions in immunotherapy. N Engl J Med 2018; 378: 1158–59. [DOI] [PubMed] [Google Scholar]
- 5.Hasegaw T, Claggett B, Tian L, Solomon SD, Pfeffer MA, Wei LJ. The myth of making inferences for an overall treatment efficacy with data from multiple comparative studies via meta-analysis. Stat Biosci 2017; 9: 284–97. [Google Scholar]