Estimating the contribution of studies in network meta-analysis: paths, flows and streams

Version Changes

Revised. Amendments from Version 1

This revised version of this manuscript includes suggestions and additions suggested by the first reviewer, Dr. Jochem König. Our improvements include the following:

1. Percentage contributions are now distributed to single trials according to their weights from direct meta-analyses. Thus, study percentage contributions can now be derived. Figure 3 and Table 4 have been updated accordingly.

2. We have now added a paragraph in the Discussion to highlight the limitations of the presented method and to clarify that the scope of this paper is not to quantify the impact of within-study bias on the NMA estimates.

3. We re-arranged Figures 1 and 2 such that source is on the left and sink on the right.

4. Supplementary file 3 has been modified.

Abstract

In network meta-analysis, it is important to assess the influence of the limitations or other characteristics of individual studies on the estimates obtained from the network. The percentage contribution matrix, which shows how much each direct treatment effect contributes to each treatment effect estimate from network meta-analysis, is crucial in this context. We use ideas from graph theory to derive the percentage that is contributed by each direct treatment effect. We start with the ‘projection’ matrix in a two-step network meta-analysis model, called the H matrix, which is analogous to the hat matrix in a linear regression model. We develop a method to translate H entries to percentage contributions based on the observation that the rows of H can be interpreted as flow networks, where a stream is defined as the composition of a path and its associated flow. We present an algorithm that identifies the flow of evidence in each path and decomposes it into direct comparisons. To illustrate the methodology, we use two published networks of interventions. The first compares no treatment, quinolone antibiotics, non-quinolone antibiotics and antiseptics for underlying eardrum perforations and the second compares 14 antimanic drugs. We believe that this approach is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the percentage contributions of direct evidence from individual studies to network treatment effects.

Introduction

Decision making around multiple alternative healthcare interventions is increasingly based on meta-analyses of a network of relevant studies, which contribute direct and indirect evidence to different treatment comparisons ^{1, 2}. Limitations in the design and flaws in the conduct of studies synthesized in network meta-analysis (NMA) reduce the confidence in the results: a treatment comparison in the network may be directly or indirectly informed by studies at high risk of bias. A relative treatment effect from NMA (hereafter the NMA effect estimate) is estimated as a linear combination of the available direct estimates of the treatment effect (i.e. the results from pairwise meta-analyses) and the indirect evidence on the treatment effect.

Salanti et al. suggested that in order to assess the impact of study deficiencies on an NMA effect estimate, the limitations of studies contributing to direct estimates should be considered jointly, taking into account their relative contribution to the overall NMA effect estimate ³. The percentage contribution matrix plays a key role in this approach: a matrix that shows how much each direct effect contributes to the estimation of the NMA effect.

The percentage contribution matrix is derived from the absolute contribution matrix. The absolute contributions of direct effects to an NMA effect is the projection matrix from a two-step NMA model ^{4, 5}. In the first stage, all direct effects are derived from pairwise meta-analyses. In the second stage, the NMA effect estimates are produced as a linear combination of the derived direct effects. The respective projection matrix is called the H matrix and it is analogous to the hat matrix in a linear regression model. The elements in the H matrix can be viewed as generalized weights from pairwise meta-analysis, but they do not add up to 1 and depend on the precision of the available studies, the degree of between-study heterogeneity and the network structure.

To translate the entries of the H matrix into percentage contributions, Salanti et al. suggested normalizing the absolute entries of each row of H and interpret them as percentages ³. However, H represents the flow of evidence in different paths; the weight of each path is assigned to each direct effect involved. Thus, ignoring the multiple occurrences of the same values by taking standardized absolute values is incorrect. In particular, such a process overestimates the contribution of comparisons involved in long paths and underestimates the weights of the shortest paths. In this paper, we address this issue and present a method that properly translates the entries of the H matrix into percentages. The methodology is based on the observation that the rows of the H matrix can be interpreted as flow networks ^{4, 6}.

Motivating example

To illustrate the ideas presented in this paper, we will use a network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations ⁷. This network was used in Salanti et al. ³ and compares no treatment ( x), quinolone antibiotic treatment ( y), non-quinolone antibiotic treatment ( u) and antiseptic treatment ( v) ⁷. The study outcome was the proportion of patients with persistent discharge from the ear after 1 week, measured using the odds ratio (OR). The network plot shown in Figure 1a shows that direct evidence exists for all comparisons except u versus x (non-quinolone antibiotic versus no treatment).

Figure 1. Network plot for the network of topical antibiotics without steroids for chronically discharging ears (a), comparison graph corresponding to the h ^xy row of H matrix (b), flows f _uv with respect to the ‘ x versus y’ network meta-analysis treatment effect are indicated along the edges), streams (c) and percentage contributions of each direct comparison (d).

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

In order to assess the confidence that should be placed in an NMA effect estimate, Salanti et al. suggested considering the quality of all pieces of evidence that contributed to it ³. For example, the studies directly comparing ‘ u versus v’ were judged to be at high risk of bias; however, in order to judge the quality of the NMA effect estimate of ‘ u versus v’, we need to consider the amount of data that these studies contributed to its estimation.

Methods

We first present the random-effects two-stage NMA model first described by Lu et al. ⁵. We will employ a simplified version of the H matrix described by König et al. ⁴ that does not take into account the correlation induced by multi-arm trials. We ignore this correlation for the sake of ease of interpretation of the entries in the H matrix; we discuss implications of multi-arm trials at the end of the Methods section. Taking advantage of previous findings on how the flow of evidence can be considered in NMA ^{4, 6}, we present an algorithm to decompose the flow in a network and subsequently approximate the percentage contributions of direct effect estimates for each NMA effect estimate.

Two-stage network meta-analysis model

Consider a network of T competing treatments. The set of treatments is denoted by V = { x, y, u, v, ...} and let x denote the reference treatment. The number of NMA effects to be estimated is $\left(\begin{array}{l}T\\ 2\end{array}\right)$ but the estimation of T – 1 effects allows the derivation of the remaining effects via linear combination. We collect the T – 1 effects against the reference treatment x in a vector of basic parameters θ = ( θ _xy,θ _xu,θ _xv, ...)′. In the case of a dichotomous outcome, θ is the parameter vector of all log-ORs compared to the common reference treatment x.

We assume that the distribution of effect modifiers is similar across comparisons and thus the transitivity assumption is plausible. The consistency assumption refers to the statistical manifestation of transitivity and implies that all sources of evidence are in agreement; this is expressed via the consistency equations

                θ _uv = θ _xv – θ _xu, for all u, v ∈ V

Let us denote the number of comparisons with direct data (that is, at least one direct study) with D. For simplification, consider that there are no multi-arm studies. At the first stage of the NMA model, direct effects are estimated, using random-effects pairwise meta-analyses. The estimates of the direct effects are collected in a column vector ${\hat{\mathit{\theta }}}^{\mathit{D}}$ of length D; their estimated variances are collected in a diagonal D × D matrix V ^D. At the second stage, the NMA effects are estimated as

$${\hat{\mathit{\theta }}}^{\mathit{N}}\mathit{=}\mathit{H}{\hat{\mathit{\theta }}}^{\mathit{D}}\mathbf{Equation}\mathbf{1}$$

where H is

                            H = Y( X′( V ^D) ^–1X) ^–1X′( V ^D) ^–1                                                                                                    Equation   2

Matrix X is a D × ( T – 1) design matrix expressing the linear relationships between the available direct effects and the basic parameters and Y is a $\left(\begin{array}{l}T\\ 2\end{array}\right)$ × ( T – 1) design matrix that links the NMA estimates with the basic parameters. Note that X is identical to Y only when there are direct studies for all treatment comparisons in the network.

Matrix H is of dimensions $\left(\begin{array}{l}T\\ 2\end{array}\right)$ × D and describes the influence of each direct effect (specified in the column) to an NMA effect (specified in the row). As Equation 2 implies, H is derived as a function of the variances of the direct effects ${\hat{\mathit{\theta }}}^{\mathit{D}}$ and the network structure; therefore, the exact (absolute) contribution of each direct comparison depends on the precision of the available direct data and the comparison’s connectivity to the rest of the network. Note that it resembles the hat matrix in a linear regression model.

Let us focus on a single row of the H matrix which, say, corresponds to the NMA effect estimate of ‘ x versus y’ and is denoted by h ^xy. Elements of h ^xy are denoted by $h_{uv}^{xy}$ and show the absolute contribution of the direct effect ${\hat{\theta }}_{uv}^D$ indicated in the subscript (‘ u versus v’) to the ‘ x versus y’ NMA effect ${\hat{\theta }}_{xy}^N$. Consider our motivating example which examines the set of treatments V = { x, y, u, v}. Equation 1 implies that the NMA treatment effect for the ‘ x versus y’ comparison is derived as a linear combination of the direct meta-analyses

$${\hat{\theta }}_{xy}^N=h_{xy}^{xy}{\hat{\theta }}_{xy}^D+h_{yv}^{xy}{\hat{\theta }}_{yv}^D+h_{xv}^{xy}{\hat{\theta }}_{xv}^D+h_{yu}^{xy}{\hat{\theta }}_{yu}^D+h_{uv}^{xy}{\hat{\theta }}_{uv}^D\mathbf{Equation}\mathbf{3}$$

The element $h_{xy}^{xy}$ represents the absolute but also the percentage contribution $p_{xy}^{xy}$ of the direct evidence for the particular NMA effect. Assuming that the comparison ‘ x versus y’ is not part of any multi-arm study, the evidence to derive the NMA effect estimate can be portioned into direct and indirect estimates

$${\hat{\theta }}_{xy}^N=h_{xy}^{xy}{\hat{\theta }}_{xy}^D+(1-h_{xy}^{xy}){\hat{\theta }}_{xy}^I\mathbf{Equation}\mathbf{4}$$

with ${\hat{\theta }}_{xy}^I$ denoting the indirect effect for the ‘ x versus y’ comparison; hence $p_{xy}^{xy}=h_{xy}^{xy}$. While the percentage contribution of each direct effect to its NMA effect can be obtained as the diagonal of the H matrix, the percentage contributions of other direct relative effects via indirect evidence (e.g. the $p_{uv}^{xy}$ percentage contribution of ${\hat{\theta }}_{uv}^D$ to ${\hat{\theta }}_{xy}^N$) cannot be easily derived from the absolute contributions (that is, from $h_{uv}^{xy}$). In the next section we will present how the absolute contributions $h_{uv}^{xy}$ could be translated to percentage contributions $p_{uv}^{xy}$.

To explain the method, we will continue focusing on one row of the H matrix, say h ^xy, corresponding to the ‘ x versus y’ comparison. Box 1 includes the definitions, along with the notation, of some of the notions used in this paper.

Box 1. Definitions.

Set of vertices

The set of vertices is defined as the set of treatments examined in the network, V = { x, y, u, v ...}.

Set of directed edges

The set of directed edges E is defined as the set of direct comparisons respecting the signs of the entries of h ^xy, the row of the H matrix corresponding to the ‘ x versus y’ comparison. Edges are given a direction upon the definition of flows; the network itself corresponds to an undirected graph.

Set of flows

The set of flows is defined as F = { f _uv, ∀ uv ∈ E} where f _uv is equal to | h _uv|.

Comparison graph

A comparison graph is defined as a graph G _xy = ( V, E, F) constructed from a row of the H matrix, h ^xy; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F.

Source

Source is defined as a vertex with no incoming edges.

Sink

Source is defined as a vertex with no outgoing edges.

Path

A path π _i is defined as a sequence of connected directed edges belonging to E.

Stream

A stream S _i is defined as the composition of a path and its associated flow, S _i = ( φ _i, π _i). with i = 1, ... , I where I is the total number of streams.

Comparison graph

König et al. showed that every row of the H matrix, h ^xy, can be interpreted as a flow network with source x and sink y, and visualised in a directed acyclic graph (DAG) ⁴. Thus, we create a graph G _xy = ( V, E, F) from h ^xy; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F. The set of vertices is defined as the set of treatments examined in the network, V = { x, y, u, v ...}. Set E is defined as a set of directed edges that correspond to observed direct comparisons respecting the signs of the entries of h ^xy. To simplify the notation, we drop from now on all superscripts assuming they all refer to xy. Then, the set E contains uv if h _uv > 0 or contains vu if h _uv < 0. The set of flows is defined as F = { f _uv, ∀ uv ∈ E} where f _uv is equal to | h _uv|.

The following conditions hold for the elements of set F (see Supplementary File 1 for proof):

      a. The sum of outflows of node x (source) is 1

$${\displaystyle \sum _{xu\in E}{f}_{xu}=1}$$

      b. The sum of inflows of node y (sink) is 1

$${\displaystyle \sum _{uy\in E}{f}_{uy}=1}$$

      c. The flow passing through each internal node (any node except x or y) is conserved

$$\forall z\in V\backslash \{x,y\},{\displaystyle \sum _{v\in V}{f}_{vz}}={\displaystyle \sum _{u\in V}{f}_{zu}}$$

      d. G _xy is acyclic; there is no path (sequence of edges) that visits the same vertex twice.

Consider, for example, the graph G _xy in Figure 1b, which corresponds to the xy comparison of the network of four treatments of Figure 1a. The set of vertices is V = { x, y, u, v} and the set of directed edges is E = { xy, xv, vy, vu, uy}. Flows f _uv are given along the edges; their numerical values are equal to the respective absolute entries of h ^xy and the direction of their corresponding edge is indicated in the subscript. As properties (a) to (d) imply, the arrows in Figure 1b indicate that the outflows of x, as well as the inflows of y, equal 1, and that the inflows equal the outflows in the intermediate nodes u and v.

Streams

In Figure 1b there are three different paths from x to y, one based on direct evidence, { xy}, and two based on indirect evidence, { xv, vy} and { xv, vu, uy}. A path is a sequence of connected directed edges belonging to E, and we denote it as π _i. As property (d) implies, each node occurs at most once in π _i. Then, given the above properties of f _uv, we can assign a flow φ _i to each path π _i. Flow φ _i is equal to the smallest f _uv in the path π _i. Figure 1c shows the three paths from x to y; π ₁, π ₂ and π ₃, and their corresponding flows. Path π ₁ corresponds to xy and its flow, φ ₁, equals the flow of the single edge in path, f _xy = 0.635. Path π ₂ is constituted from two edges, xv and vy; thus, flow φ ₂ = min( f _xv, f _vy) = 0.251. The flow corresponding to the third path π ₃ is φ ₃ = min( f _xv, f _vu, f _uy) = 0.114.

We define a stream, S _i, as the composition of a path and its associated flow, S _i = ( φ _i, π _i) with i = 1, ..., I where I is the total number of streams; here I = 3. Note that it holds ${\displaystyle {\sum }_{i=1}^I{\phi }_i=1}$.

Percentage contributions of direct comparisons

In order to assign percentage contributions to each direct comparison, we need to split each stream’s flow to the involved edges in the stream’s path. It can be shown that an NMA effect is a linear combination of the direct effects combined within paths. More specifically, Equation 3 can be re-written as

$${\hat{\theta }}_{xy}^N={\phi }_1{\hat{\theta }}_{xy}^D+{\phi }_2({\hat{\theta }}_{xv}^D-{\hat{\theta }}_{yv}^D)+{\phi }_3({\hat{\theta }}_{xv}^D-{\hat{\theta }}_{uv}^D-{\hat{\theta }}_{yu}^D)\mathbf{Equation}\mathbf{5}$$

with ${\displaystyle {\sum }_{i=1}^3{\phi }_i=1}$. Equation 5 represents ${\hat{\theta }}_{xy}^N$ as a weighted sum of direct and two indirect effect estimates; the effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.

To approximate the percentage contributions per comparison, we suggest dividing φ _i by the length of the respective path π _i, # π _i. This will leave the percentage contribution of the direct evidence of the same treatment comparison equal to the diagonal of the H matrix and assign to each comparison involved in an indirect route a portion of the respective stream’s flow. Note that directed edges might be involved in more than one path; we thus define the percentage contribution of an edge uv as

$$p_{uv}={\displaystyle \sum _{\forall iwhereuv\in {\pi }_i}{\phi }_i/\#{\pi }_i}\mathbf{Equation}\mathbf{6}$$

Figure 1d shows the derivation of the percentage contributions of each direct comparison in the network of topical antibiotics. Hence, from the row of the H matrix (0.635, 0.365, –0.114, –0.251, –0.114), which shows the absolute contributions of the direct effects ${\hat{\theta }}_{xy}^D,{\hat{\theta }}_{xv}^D,{\hat{\theta }}_{yv}^D,{\hat{\theta }}_{yu}^D,{\hat{\theta }}_{uv}^D$ to ${\hat{\theta }}_{xy}^N$, we approximated their percentage contribution as 63.5%, 16.4%, 3.8%, 12.6% and 3.8%, respectively.

Algorithm to decompose flows into percentage contributions

In this section, we present an iterative algorithm that generalizes the process outlined above to derive percentage contributions of each direct effect to the estimation of a ‘ x versus y’ NMA effect. We start by defining a graph G _xy from h ^xy.

The algorithm is described as follows:

• 0. Set initial graph G ₀ = ( V, E ₀, F ₀) = G _xy. E ₀ contains uv if h _uv > 0 or contains vu if h _uv < 0. The set of flows is F ₀ = { f _{0, uv} | uv ∈ E ₀}; numerical values of f _{0, uv} are equal to h _uv.

Then, repeat the process below I times, equal to the number of streams in G _i, until E _i = {Ø}.

• 1. In G _i–1, find the shortest path from x to y, π _i, and define its flow as φ _i = min{ f _{i–1, uv}, uv ∈ π _i}. Then, use π _i and φ _i to define the stream S _i = ( φ _i, π _i).
• 2. Recalculate the flow of edges uv ∈ π _i by subtracting φ _i from the flow of the edges of the stream found: f _i,uv = f _{i–1, uv} – φ _i ∀ uv ∈ π _i. The flow of the rest of the edges that do not belong to π _i remain unchanged: f _i,uv = f _{i–1, uv} ∀ uv ∉ π _i.
• 3. Define E _i as the set of edges uv for which f _i,uv > 0; this is E _i–1 after removing the edges with zero flow, E _i = E _i–1\{ uv | f _i,uv = 0}. Collect f _i,uv to form the set F _i = { f _i,uv | uv ∈ E _i}.
• 4. If E _i ≠ {Ø} define G _i = ( V, E _i, F _i) and go to step 1.

When the algorithm terminates, all streams S _i = ( φ _i, π _i) have been identified and Equation 6 is used to derive the percentage contributions p _uv.

Repeating the same process for all NMA effects, we derive all $p_{uv}^{xy}$ and collect them in a matrix P of the same dimensions as H. The presented algorithm could be described as a reverse maximum flow Edmonds Karp algorithm ⁸, but instead of adding we remove augmenting paths.

It is possible that multiple shortest paths exist; in this case, the order in which one chooses such a path could in principle result in different percentage contributions per comparison. We can, thus, use the following modification in the algorithm to impose consistency. Instead of selecting the shortest path, we assign cost values c _i,uv to each edge uv as follows: c _i,uv = 2 – f _{i–1, uv}. Then, we select the path from x to y with the minimum cost across comparisons included in π _i. The definition of the cost values c _i,uv assures that paths are selected from shortest to longest and removes any ambiguity regarding the selection of paths.

The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the the H matrix as described by König et al. ⁴ that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm.

Calculations in this paper were performed using R.

Application

We apply the algorithm described above to the network of topical antibiotics ⁷.

Percentage contributions of direct relative treatment effects to the estimation of the NMA effect between non-quinolone antibiotic and no treatment

Direct effects are obtained using the random effects model and the H matrix of dimension 6 ×5 is calculated using Equation 2. The H matrix, along with NMA effects, is given in Table 1.

Table 1. H matrix in the network of topical antibiotics without steroids for chronically discharging ears.

Columns correspond to direct comparisons and rows correspond to network meta-analysis (NMA) effects. Direct effects along with their variances and NMA effects with 95% confidence intervals (CIs) are given in the last column. Direct and NMA effects are measured as log odds ratios. Positive values favour the first treatment.

	xy	xv	yu	yv	uv	Direct effect	Variance of direct effect	NMA effect (95% CIs)
xy	0.635	0.365	–0.114	–0.251	–0.114	–2.29	0.42	–1.86 (–3.05,–0.67)
xu	0.603	0.397	0.632	–0.029	–0.368	–	–	–1.32 (–2.61,–0.02)
xv	0.545	0.455	0.170	0.375	0.170	0.35	0.63	–0.72 (–1.95,0.52)
yu	–0.032	0.032	0.745	0.223	–0.255	0.39	0.12	0.54 (–0.07,1.16)
yv	–0.090	0.090	0.284	0.627	0.284	1.24	0.15	1.14 (0.47,1.81)
uv	–0.058	0.058	–0.462	0.404	0.538	0.53	0.22	0.60 (–0.12,1.33)

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

We begin by applying step 0 of the algorithm. We construct the network G ₀ = G _xy = ( V, E ₀, F ₀) with source x and sink y corresponding to row h ^xy. The set of vertices is V = { x, y, u, v} and the set of directed edges, taking into account the signs of the elements of h ^xy, is E ₀ = { xy, xv, vy, vu, uy}. The set of flows is F ₀ = { f _{0, uv} | uv ∈ E ₀}, where f _{0, uv} equal the respective absolute values of Table 1 and are given along the edges of Figure 1b.

Then, we apply the developed iterative algorithm until E _i = {Ø}. The iterations of the algorithm equal the number of existing streams from x to y and are illustrated in Figure 2.

Figure 2. Illustration of the steps of the algorithm for approximating percentage contributions per comparison in the network of topical antibiotics without steroids for chronically discharging ears focusing on the comparison ‘ x versus y’.

Treatment labels: x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

First iteration

• 1. In G ₀, find the shortest path from x to y, π ₁ = { xy}. Define its flow as φ ₁ = min{ f _{0, uv}, uv ∈ π ₁} = f _{0, xy} = 0.635. Define stream S ₁ = ( φ ₁, π ₁) ( Figure 2a).
• 2. Recalculate the flow of edge xy ∈ π ₁ as f _{1, xy} = f _{0, xy} – φ ₁ = 0.635 – 0.635 = 0. The flow of the rest of the comparisons remains unchanged: f _{1, uv} = f _{0, uv} ∀ uv ∉ π ₁ ( Figure 2b).
• 3. Define E ₁ as the set of edges uv for which f _{1, uv} > 0; edge xy is removed since its flow is zero, E ₁ = E ₀\{ xy}. Collect f _{1, uv} to form the set F ₁ = { f _{1, uv} | uv ∈ E ₁} ( Figure 2c).
• 4. Define G ₁ = ( V, E ₁, F ₁). As E ₁ = { xv, vy, vu, uy} ≠ {Ø}, go to step 1 ( Figure 2c).

Second iteration

• 1. In G ₁, find the shortest path from x to y, π ₂ = { xv,vy}. Define its flow as φ ₂ = min{ f _{1, uv}, uv ∈ π ₂} = f _{1, vy} = 0.251. Define stream S ₂ = ( φ ₂, π ₂) ( Figure 2d).
• 2. Recalculate the flow of edges xv and vy as f _{2, xv} = f _{1, xv} – φ ₂ = 0.365 – 0.251 = 0.114 and f _{2, vy} = f _{1, vy} – φ ₂ = 0.251 – 0.251 = 0. The flow of the rest of the comparisons remains unchanged: f _{2, uv} = f _{1, uv} ∀ uv ∉ π ₂ ( Figure 2e).
• 3. Define E ₂ as the set of edges uv for which f _{2, uv} > 0; edge vy is removed since its flow is zero and thus E ₂ = E ₁\{ vy}. Collect f _{2, uv} to form the set F ₂ = { f _{2, uv} | uv ∈ E ₂} ( Figure 2f).
• 4. Define G ₂ = ( V, E ₂, F ₂). As E ₂ = { xv, vu, uy} ≠ {Ø}, go to step 1 ( Figure 2f).

Third iteration

• 1. In G ₂, find the shortest path from x to y, π ₃ = { xv, vu, uy}. Define its flow as φ ₃ = min{ f _{2, uv}, uv ∈ π ₃} = 0.114. Define stream S ₃ = ( φ ₃, π ₃) ( Figure 2g).
• 2. Recalculate the flow of edges xv, vu and uy as f _{3, xv} = f _{3, vu} = f _{3, uy} = f _{2, xv} – φ ₃ = 0.114 – 0.114 = 0 ( Figure 2h).
• 3. Define E ₃ as the set of edges uv for which f _{3, uv} > 0; edges xv, vu and uy are removed since their flow is zero, E ₃ = E ₂\{ xv, vu, uy}. Collect f _{3, uv} to form the set F ₃ = { f _{3, uv} | uv ∈ E ₃} ( Figure 2i).
• 4. Define G ₃ = ( V, E ₃, F ₃). The set of direct edges is E ₃ = {Ø} and the algorithm is terminated at this point ( Figure 2i).

Figure 1c shows the flows of the three streams identified when applying the above algorithm. We then calculate the percentage contributions of each comparison to the ‘ x versus y’ NMA treatment effect estimate using Equation 6 ( Figure 1d). For instance, to calculate p _xv we first have to identify the relevant paths; these were π ₂ and π ₃. Consequently,

$$p_{xv}=\frac{{\phi }_2}{\left|{\pi }_2\right|}+\frac{{\phi }_3}{\left|{\pi }_3\right|}=\frac{0.251}{2}+\frac{0.114}{3}=0.164=16.4\%$$

The calculations for deriving the percentage contributions of the other comparisons are shown in Table 2. Applying the algorithm to all NMA treatment effect estimates we get the entire percentage contribution matrix P.

Table 2. Percentage contributions of direct comparisons to the ‘ x versus y’ network meta-analysis treatment effect in the network of topical antibiotics without steroids for chronically discharging ears.

	xy	xv	yu	yv	uv
xy	$p_{xy}=\frac{{\phi }_1}{\left|{\pi }_1\right|}$ = 63.5%	$p_{xv}=\frac{{\phi }_2}{\left|{\pi }_2\right|}+\frac{{\phi }_3}{\left|{\pi }_3\right|}$ = 16.4%	$p_{yu}=\frac{{\phi }_3}{\left|{\pi }_3\right|}$ = 3.8%	$p_{yv}=\frac{{\phi }_2}{\left|{\pi }_2\right|}$ = 12.6%	$p_{uv}=\frac{{\phi }_3}{\left|{\pi }_3\right|}$ = 3.8%

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Percentage study contributions

Matrix P ( Table 3) shows the percentage contributions of each direct comparison to each NMA treatment effect estimate. These percentages can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p _xy = 63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting in study percentage contributions of $\frac{0.69}{0.69+1.54}63.5\%=19.6\%$ and $\frac{1.54}{0.69+1.54}63.5\%=43.8\%$ to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer ⁹ are used for multi-arm studies.

Table 3. Percentage contribution matrix P for the network of topical antibiotics without steroids for chronically discharging ears.

Cells show the percentage contribution of direct comparisons indicated in the column to the network meta-analysis treatment effects indicated in the rows.

	xy	xv	yu	yv	uv
xy	63.5%	16.4%	3.8%	12.6%	3.8%
xu	30.1%	19.4%	31.1%	1%	18.4%
xv	24.4%	45.5%	5.7%	18.8%	5.7%
yu	1.1%	1.1%	74.5%	11.1%	12.2%
yv	4.5%	4.5%	14.2%	62.7%	14.2%
uv	1.9%	1.9%	22.1%	20.2%	53.8%

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Table 4. Study percentage contribution matrix P* for the network of topical antibiotics without steroids for chronically discharging ears.

Cells show the percentage contribution of individual studies indicated in the column to the network meta-analysis treatment effects indicated in the rows.

	Study 1	Study 2	Study 3	Study 4	Study 5	Study 6	Study 7	Study 8	Study 9	Study 10	Study 11	Study 12	Study 13
xy	19.7	63.4	2.8	0.5	0.6	0.4	0.6	0.7	2.7	1.3	0.8	2.3	4.2
xu	9.3	40.4	8.8	4.4	5.3	3.5	5.1	5.4	6.9	6.5	3.9	0.2	0.3
xv	7.6	67.1	4.2	0.8	1	0.6	0.9	1	4.1	2	1.2	3.4	6.2
yu	0.3	4.63	13.6	10.5	12.7	8.4	12.2	12.9	12.2	4.3	2.6	2	3.7
yv	1.4	23.5	12.1	2	2.4	1.6	2.3	2.4	12.2	5	3	11.3	20.7
uv	0.6	8.4	19	3.1	3.8	2.5	3.6	3.8	14.4	19.1	11.5	3.6	6.7

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Using percentage study contributions to quantify the impact of a characteristic in a direct comparison

The algorithm translating the H matrix into study percentage contributions can be applied to quantify the influence that a study-level characteristic has in the estimation of the NMA effects. For instance, if risk of bias judgements for individual studies are available, we can obtain an approximation of the percentage of each NMA treatment effect estimate that is coming from studies with a ‘high’, ‘moderate’, or ‘low’ risk of bias. Salanti et al. suggested the visualisation of this information using a bar plot, in which direct comparisons of the same risk of bias level have been grouped ³. Figure 3 shows such a bar plot using the algorithm described in this paper and distributing comparison percentage contributions to study percentage contributions; inspecting Figure 3 can support judgements regarding the importance of study limitations for different NMA treatment effect estimates. For instance, studies with high risk of bias contribute more than 50% in the estimation of the ‘ u versus v’ comparison, potentially reducing the confidence that we can place in this particular NMA treatment effect estimate.

Figure 3. Bar plot showing the study percentage contributions of direct comparisons with low (green), moderate (yellow) and high (red) risk of bias.

The bar plot has been produced in CINeMA (Confidence In Network Meta-Analysis) software ¹². Studies are synthesized using the random effects model. x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Percentage contributions of direct comparisons in a large complex network of interventions

So far, we have illustrated how to derive percentage contributions for a network with four treatments. However, the algorithm can be straightforwardly applied to large networks of any structure, as soon as the involved treatments are connected. Consider for example a large network examining antimanic drugs ( Figure 4) ¹¹. Let us concentrate on the comparison PLA versus OLA (‘placebo versus olanzapine’); the algorithm starts by applying step 0 and constructing network G ₀. Then, we continue by finding the shortest path in the first iteration, which corresponds to the direct comparison, and define its flow and stream S ₁ = ( φ ₁, π ₁). The number of algorithm’s iterations is equal to the number of streams from placebo to olanzapine, which turns out to be 16. The resulting entire percentage matrix is given in Supplementary File 2.

Figure 4. Network plot for the network of antimanic drugs.

ASE, asenapine; ARI, aripiprazole; PLA, placebo; HAL, haloperidol; QUE, quetiapine; LITH, lithium; ZIP, ziprasidone; OLA, olanzapine; DIV, divalproex; RIS, risperidone; CARB, carbamazepine; LAM, lamotrigine; PAL, paliperidone; TOP, topiramate; ASE, asenapine.

Outcome data from the example network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations ⁷

Data labels: study, name of individual studies; id, id of the individual studies; t; treatment; r, number of events; n, sample size; rob, risk of bias per study.

Copyright: © 2018 Papakonstantinou T et al.

Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Discussion

In this paper, we present a new approach to derive percentage contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects.

Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates. Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the percentage contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its percentage contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.

Alternative methods to derive the relative contribution of all sources of evidence have been developed ^{4, 12, 13, 14}. An alternative approach has been proposed to derive percentage study weights in a variety of meta-analysis models, including meta-regression, network meta-analysis and individual patient data meta-analysis ¹⁰. This approach is based on the decomposition of Fisher’s information matrix and thus the derived weights are not influenced by the network structure. Further investigation of the degree of agreement between our algorithm and that of Riley et al. ¹⁰ would be of interest.

In the example implemented in the Application, there is no other possible set of paths, and associated streams, that could be selected from x to y in order to partition the inflow of x: π ₁, π ₂ and π ₃ is the only possible set of streams ( Figure 1c). Thus, even if we were taking paths using different criteria, i.e. from longest to shortest, according to values from the H matrix or even randomly, the percentage contributions given in Table 2 would be identical. However, cases exist where the selection of paths does influence the derivation of the P matrix. In Supplementary File 3, we elaborate on the selection of direct paths in the algorithm and discuss some alternative modifications of the algorithm. We are planning to examine the properties of the different approaches in greater detail in a follow up project.

We offer an R package ¹¹, which we also use in the software application CINeMA (Confidence In Network Meta-Analysis) ¹², that aims to simplify the evaluation of confidence in the findings from NMA. While CINeMA largely follows the framework previously developed by Salanti et al. ³, the refinement of several methodological aspects is currently under development. Core aspects of the approach include the consideration of the relative contributions of each direct comparison to each NMA treatment effect estimate. To this end, CINeMA uses the percentage contribution matrix as described in this paper. The command netweight in Stata has also been updated to use the described approach.

We believe that the approach described in this paper is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the percentage contributions of direct evidence from individual studies to network treatment effects.

Data availability

The data referenced by this article are under copyright with the following copyright statement: Copyright: © 2018 Papakonstantinou T et al.

Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication). http://creativecommons.org/publicdomain/zero/1.0/

Dataset 1: Outcome data from the example network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations ⁷. Data labels: study, name of individual studies; id, id of the individual studies; t; treatment; r, number of events; n, sample size; rob, risk of bias per study. DOI: 10.5256/f1000research.14770.d203174 ¹³.

Funding Statement

TP received funding from a Campbell Collaboration Method Grant [grant no. CMG2.04].

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Supplementary material

Supplementary File 1. Proof that the sum of all outflows from the source is equal to the sum of all inflows to the sink and both are 1.

Supplementary File 2. Percentage contribution matrix P for the network of antimanic drugs. Cells show the percentage contribution of direct comparisons indicated in the column to the network meta-analysis treatment effects indicated in the rows. ASE, asenapine; ARI, aripiprazole; PLA, placebo; HAL, haloperidol; QUE, quetiapine; LITH, lithium; ZIP, ziprasidone; OLA, olanzapine; DIV, divalproex; RIS, risperidone; CARB, carbamazepine; LAM, lamotrigine; PAL, paliperidone; TOP, topiramate; ASE, asenapine.

Supplementary File 3

Considerations on the selection of streams