Authors' reply to “Comments on Identifying inconsistency in network meta‐analysis: Is the net heat plot a reliable method?”

We thank Krahn et al for their letter¹ and welcome the opportunity to discuss this further.

When assessing inconsistency, interest lies in the difference between the direct evidence and the network evidence. We believe the difference between our paper and the correction that Krahn et al¹ have suggested arises due to the possibility of interpreting “detaching a design” in one of two ways.

In their original paper, when describing the approach for detaching a single design, Krahn et al state “This procedure is equivalent to a ‘leave one out’ approach: Once per fit, studies with one design are left out of the network estimate to obtain an independent estimate of the treatment effect in design d and to obtain a network model fit independent of studies with design d”.² Our interpretation of this statement³ is that, when assessing the impact of detaching design d on the network estimate for design d, the network estimate is calculated from the evidence that remains when trials of design d are left out. If design d is left out then the network evidence can only be informed by what remains that is, the indirect evidence. This leads to our original result.

However, Krahn et al² also describe their approach in a subtly different way: as a design‐by‐treatment interaction. Following the design‐by‐treatment interaction approach, instead of leaving out design d, a new parameter is assigned to design d so that the model now includes two parameters for design d representing the direct and the indirect evidence. Krahn et al have now clarified (in their second paragraph) that they consider the former (ie, the direct evidence) to be the network evidence. Following this approach, we agree with the correction proposed by Krahn et al. However, we also incline to the view that the interpretation of the original paper² outlined in the preceding paragraph is more natural.

Nevertheless, accepting the correction proposed by Krahn et al¹ simplifies the argument in our paper³ because with $Q_{\mathrm{AC}(c)}^{\mathrm{inc}}=0$, our equation (7) is equal to our equation (6). Further, because Q ^diff is still a squared and scaled version of the inconsistency parameter, our observation that this is correlated with the formal inconsistency test statistic, while awkward to interpret, remains valid. The question remains, why use a scaled version when the unscaled version has a known distribution?

While acknowledging the authors' important contributions in this area, we note that they have not disputed our key finding which is that the net heat plot displays a somewhat arbitrary weighting of the loop inconsistency statistics, which does not lend itself to statistical testing, and which should therefore be interpreted cautiously.

DATA AVAILABILITY STATEMENT

Data sharing is not applicable to this article as no new data were created or analyzed in this study.