Caution is advised when interpreting subgroup analyses

In their randomized trial of augmentation therapy with minocycline for depression in patients considered treatment-resistant and with levels of serum C-reactive protein (CRP) ≥ 1 mg/L, Nettis and colleagues found no effect of the treatment on their primary outcome, the change in Hamilton Depression Rating Scale (HAMD-17) score from baseline to end-of-treatment, or on any of five other clinical outcome measures [1]. Based on a subgroup analysis, however, they concluded that their data showed some evidence of efficacy of minocycline treatment but only in those with levels of CRP ≥ 3 mg/L and stated that their findings provided “robust evidence” in favor of using CRP = ~3 mg/L as a threshold to identify response to minocycline.

Subgroup analyses can potentially lead to better informed treatment decisions when they are credible but can be misleading when that is not the case. The likelihood that an observed subgroup effect represents a real effect, depends on a number of factors and is best viewed in terms of a continuum, ranging from “extremely unlikely” to “highly plausible” [2]. Criteria have been outlined to assess the credibility of subgroup claims [3]. We would like to direct attention to several of these criteria that should be considered when assessing the credibility of the subgroup claim in the study by Nettis et al. [1].

First, their finding of a statistical effect in those participants with levels of CRP ≥ 3 mg/L does not provide evidence of a subgroup effect. The appropriate statistical approach for establishing a subgroup effect is a test of subgroup-treatment effect interaction, not an analysis of statistical significance of the treatment in one subgroup or the other [4]. Thus, the authors’ analysis did not address the question whether chance alone was likely to explain the difference in the apparent effect between the CRP groups and their claim of a subgroup effect was therefore not supported by their data. Even if done, however, the power of an interaction analysis would likely be low, given their small sample size [5].

Second, while it was not clear from the published report, their subgroup analysis and the anticipated direction of effect was not prespecified in the protocol and their subgroup analysis should therefore be considered exploratory.

Third, randomization was not stratified according to the subgroups. This, especially given the small sample size, may have resulted in imbalances in confounders between the subgroups. It is therefore possible, that the apparent subgroup effect was due to confounding from a different factor. A presentation of potential prognostic variables by subgroup could have allowed for evaluation of some potential imbalances between the groups; residual confounding, however, would still be possible.

Fourth, the authors did not present subgroup analyses for the other clinical outcomes. If their subgroup effect manifested itself across all closely related outcomes, this would increase the likelihood that the observed subgroup effect is real [2].

Based on these issues, we suggest it would be difficult to consider it likely that the subgroup effect reported by Nettis et al. [1] represents a real effect. While an appropriate interpretation of their results may instead be that a difference in effect between subgroups is uncertain [6], their findings could be considered potentially hypothesis generating, warranting further investigation in adequately powered, well-conducted trials with appropriate methodology for analyzing pre-specified subgroup hypotheses.

Funding and disclosure

Cochrane Denmark is funded by the Danish Government. The authors declare no competing interests.

Author contributions

KM: Conceptualization, Methodology, Writing – Original Draft. ASPM: Methodology, Writing – Review & Editing.