Estimating the contribution of studies in network meta-analysis: paths, flows and streams

Version Changes

Revised. Amendments from Version 2

The revised version of this manuscript includes suggestions by Drs. Annette M. O’Connor and John R. Thomson. In particular:

1. We changed ‘percentages’ to ‘proportions’ throughout. Thus, Figure 1 has also been revised.

2. We clarify in which situations our algorithm can be used in sections 'Proportion contributions of direct comparisons' and 'Algorithm to decompose flows into proportion contributions'.

3. We have revised the Discussion according to reviewers' comments

Abstract

In network meta-analysis, it is important to assess the influence of the limitations or other characteristics of individual studies on the estimates obtained from the network. The proportion contribution matrix, which shows how much each direct treatment effect contributes to each treatment effect estimate from network meta-analysis, is crucial in this context. We use ideas from graph theory to derive the proportion that is contributed by each direct treatment effect. We start with the ‘projection’ matrix in a two-step network meta-analysis model, called the H matrix, which is analogous to the hat matrix in a linear regression model. We develop a method to translate H entries to proportion contributions based on the observation that the rows of H can be interpreted as flow networks, where a stream is defined as the composition of a path and its associated flow. We present an algorithm that identifies the flow of evidence in each path and decomposes it into direct comparisons. To illustrate the methodology, we use two published networks of interventions. The first compares no treatment, quinolone antibiotics, non-quinolone antibiotics and antiseptics for underlying eardrum perforations and the second compares 14 antimanic drugs. We believe that this approach is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the proportion contributions of direct evidence from individual studies to network treatment effects.

Introduction

Decision making around multiple alternative healthcare interventions is increasingly based on meta-analyses of a network of relevant studies, which contribute direct and indirect evidence to different treatment comparisons ^{1, 2}. Limitations in the design and flaws in the conduct of studies synthesized in network meta-analysis (NMA) reduce the confidence in the results: a treatment comparison in the network may be directly or indirectly informed by studies at high risk of bias. A relative treatment effect from NMA (hereafter the NMA effect estimate) is estimated as a linear combination of the available direct estimates of the treatment effect (i.e. the results from pairwise meta-analyses) and the indirect evidence on the treatment effect.

Salanti et al. suggested that in order to assess the impact of study deficiencies on an NMA effect estimate, the limitations of studies contributing to direct estimates should be considered jointly, taking into account their relative contribution to the overall NMA effect estimate ³. The proportion contribution matrix plays a key role in this approach: a matrix that shows how much each direct effect contributes to the estimation of the NMA effect.

The proportion contribution matrix is derived from the absolute contribution matrix. The absolute contributions of direct effects to an NMA effect is the projection matrix from a two-step NMA model ^{4, 5}. In the first stage, all direct effects are derived from pairwise meta-analyses. In the second stage, the NMA effect estimates are produced as a linear combination of the derived direct effects. The respective projection matrix is called the H matrix and it is analogous to the hat matrix in a linear regression model. The elements in the H matrix can be viewed as generalized weights from pairwise meta-analysis, but they do not add up to 1 and depend on the precision of the available studies, the degree of between-study heterogeneity and the network structure.

To translate the entries of the H matrix into proportion contributions, Salanti et al. suggested normalizing the absolute entries of each row of H and interpret them as proportions ³. However, H represents the flow of evidence in different paths; the weight of each path is assigned to each direct effect involved. Thus, ignoring the multiple occurrences of the same values by taking standardized absolute values is incorrect. In particular, such a process overestimates the contribution of comparisons involved in long paths and underestimates the weights of the shortest paths. In this paper, we address this issue and present a method that properly translates the entries of the H matrix into proportions. The methodology is based on the observation that the rows of the H matrix can be interpreted as flow networks ^{4, 6}.

Motivating example

To illustrate the ideas presented in this paper, we will use a network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations ⁷. This network was used in Salanti et al. ³ and compares no treatment ( x), quinolone antibiotic treatment ( y), non-quinolone antibiotic treatment ( u) and antiseptic treatment ( v) ⁷. The study outcome was the proportion of patients with persistent discharge from the ear after 1 week, measured using the odds ratio (OR). The network plot shown in Figure 1a shows that direct evidence exists for all comparisons except u versus x (non-quinolone antibiotic versus no treatment).

Figure 1. Network plot for the network of topical antibiotics without steroids for chronically discharging ears (a), comparison graph corresponding to the h ^xy row of H matrix (b), flows f _uv with respect to the ‘ x versus y’ network meta-analysis treatment effect are indicated along the edges), streams (c) and proportion contributions of each direct comparison (d).

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

In order to assess the confidence that should be placed in an NMA effect estimate, Salanti et al. suggested considering the quality of all pieces of evidence that contributed to it ³. For example, the studies directly comparing ‘ u versus v’ were judged to be at high risk of bias; however, in order to judge the quality of the NMA effect estimate of ‘ u versus v’, we need to consider the amount of data that these studies contributed to its estimation.

Methods

We first present the random-effects two-stage NMA model first described by Lu et al. ⁵. We will employ a simplified version of the H matrix described by König et al. ⁴ that does not take into account the correlation induced by multi-arm trials. We ignore this correlation for the sake of ease of interpretation of the entries in the H matrix; we discuss implications of multi-arm trials at the end of the Methods section. Taking advantage of previous findings on how the flow of evidence can be considered in NMA ^{4, 6}, we present an algorithm to decompose the flow in a network and subsequently approximate the proportion contributions of direct effect estimates for each NMA effect estimate.

Two-stage network meta-analysis model

Consider a network of T competing treatments. The set of treatments is denoted by V = { x, y, u, v, ...} and let x denote the reference treatment. The number of NMA effects to be estimated is $\left(\begin{array}{l}T\\ 2\end{array}\right)$ but the estimation of T – 1 effects allows the derivation of the remaining effects via linear combination. We collect the T – 1 effects against the reference treatment x in a vector of basic parameters θ = ( θ _xy,θ _xu,θ _xv, ...)′. In the case of a dichotomous outcome, θ is the parameter vector of all log-ORs compared to the common reference treatment x.

We assume that the distribution of effect modifiers is similar across comparisons and thus the transitivity assumption is plausible. The consistency assumption refers to the statistical manifestation of transitivity and implies that all sources of evidence are in agreement; this is expressed via the consistency equations

                θ _uv = θ _xv – θ _xu, for all u, v ∈ V

Let us denote the number of comparisons with direct data (that is, at least one direct study) with D. For simplification, consider that there are no multi-arm studies. At the first stage of the NMA model, direct effects are estimated, using random-effects pairwise meta-analyses. The estimates of the direct effects are collected in a column vector ${\hat{\mathit{\theta }}}^{\mathit{D}}$ of length D; their estimated variances are collected in a diagonal D × D matrix V ^D. At the second stage, the NMA effects are estimated as

$${\hat{\mathit{\theta }}}^{\mathit{N}}\mathit{=}\mathit{H}{\hat{\mathit{\theta }}}^{\mathit{D}}\mathbf{Equation}\mathbf{1}$$

where H is

                            H = Y( X′( V ^D) ^–1X) ^–1X′( V ^D) ^–1                                                                                                    Equation   2

Matrix X is a D × ( T – 1) design matrix expressing the linear relationships between the available direct effects and the basic parameters and Y is a $\left(\begin{array}{l}T\\ 2\end{array}\right)$ × ( T – 1) design matrix that links the NMA estimates with the basic parameters. Note that X is identical to Y only when there are direct studies for all treatment comparisons in the network.

Matrix H is of dimensions $\left(\begin{array}{l}T\\ 2\end{array}\right)$ × D and describes the influence of each direct effect (specified in the column) to an NMA effect (specified in the row). As Equation 2 implies, H is derived as a function of the variances of the direct effects ${\hat{\mathit{\theta }}}^{\mathit{D}}$ and the network structure; therefore, the exact (absolute) contribution of each direct comparison depends on the precision of the available direct data and the comparison’s connectivity to the rest of the network. Note that it resembles the hat matrix in a linear regression model.

Let us focus on a single row of the H matrix which, say, corresponds to the NMA effect estimate of ‘ x versus y’ and is denoted by h ^xy. Elements of h ^xy are denoted by $h_{uv}^{xy}$ and show the absolute contribution of the direct effect ${\hat{\theta }}_{uv}^D$ indicated in the subscript (‘ u versus v’) to the ‘ x versus y’ NMA effect ${\hat{\theta }}_{xy}^N$. Consider our motivating example which examines the set of treatments V = { x, y, u, v}. Equation 1 implies that the NMA treatment effect for the ‘ x versus y’ comparison is derived as a linear combination of the direct meta-analyses

$${\hat{\theta }}_{xy}^N=h_{xy}^{xy}{\hat{\theta }}_{xy}^D+h_{yv}^{xy}{\hat{\theta }}_{yv}^D+h_{xv}^{xy}{\hat{\theta }}_{xv}^D+h_{yu}^{xy}{\hat{\theta }}_{yu}^D+h_{uv}^{xy}{\hat{\theta }}_{uv}^D\mathbf{Equation}\mathbf{3}$$

The element $h_{xy}^{xy}$ represents the absolute but also the proportion contribution $p_{xy}^{xy}$ of the direct evidence for the particular NMA effect. Assuming that the comparison ‘ x versus y’ is not part of any multi-arm study, the evidence to derive the NMA effect estimate can be portioned into direct and indirect estimates

$${\hat{\theta }}_{xy}^N=h_{xy}^{xy}{\hat{\theta }}_{xy}^D+(1-h_{xy}^{xy}){\hat{\theta }}_{xy}^I\mathbf{Equation}\mathbf{4}$$

with ${\hat{\theta }}_{xy}^I$ denoting the indirect effect for the ‘ x versus y’ comparison; hence $p_{xy}^{xy}=h_{xy}^{xy}$. While the proportion contribution of each direct effect to its NMA effect can be obtained as the diagonal of the H matrix, the proportion contributions of other direct relative effects via indirect evidence (e.g. the $p_{uv}^{xy}$ proportion contribution of ${\hat{\theta }}_{uv}^D$ to ${\hat{\theta }}_{xy}^N$) cannot be easily derived from the absolute contributions (that is, from $h_{uv}^{xy}$). In the next section we will present how the absolute contributions $h_{uv}^{xy}$ could be translated to proportion contributions $p_{uv}^{xy}$.

To explain the method, we will continue focusing on one row of the H matrix, say h ^xy, corresponding to the ‘ x versus y’ comparison. Box 1 includes the definitions, along with the notation, of some of the notions used in this paper.

Box 1. Definitions.

Set of vertices

The set of vertices is defined as the set of treatments examined in the network, V = { x, y, u, v ...}.

Set of directed edges

The set of directed edges E is defined as the set of direct comparisons respecting the signs of the entries of h ^xy, the row of the H matrix corresponding to the ‘ x versus y’ comparison. Edges are given a direction upon the definition of flows; the network itself corresponds to an undirected graph.

Set of flows

The set of flows is defined as F = { f _uv, ∀ uv ∈ E} where f _uv is equal to | h _uv|.

Comparison graph

A comparison graph is defined as a graph G _xy = ( V, E, F) constructed from a row of the H matrix, h ^xy; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F.

Source

Source is defined as a vertex with no incoming edges.

Sink

Sink is defined as a vertex with no outgoing edges.

Path

A path π _i is defined as a sequence of connected directed edges belonging to E.

Stream

A stream S _i is defined as the composition of a path and its associated flow, S _i = ( φ _i, π _i). with i = 1, ... , I where I is the total number of streams.

Comparison graph

König et al. showed that every row of the H matrix, h ^xy, can be interpreted as a flow network with source x and sink y, and visualised in a directed acyclic graph (DAG) ⁴. Thus, we create a graph G _xy = ( V, E, F) from h ^xy; its definition derives from a set of vertices V, a set of edges E, and a set of flows, F. The set of vertices is defined as the set of treatments examined in the network, V = { x, y, u, v ...}. Set E is defined as a set of directed edges that correspond to observed direct comparisons respecting the signs of the entries of h ^xy. To simplify the notation, we drop from now on all superscripts assuming they all refer to xy. Then, the set E contains uv if h _uv > 0 or contains vu if h _uv < 0. The set of flows is defined as F = { f _uv, ∀ uv ∈ E} where f _uv is equal to | h _uv|.

The following conditions hold for the elements of set F (see Supplementary File 1 for proof):

      a. The sum of outflows of node x (source) is 1

$${\displaystyle \sum _{xu\in E}{f}_{xu}=1}$$

      b. The sum of inflows of node y (sink) is 1

$${\displaystyle \sum _{uy\in E}{f}_{uy}=1}$$

      c. The flow passing through each internal node (any node except x or y) is conserved

$$\forall z\in V\backslash \{x,y\},{\displaystyle \sum _{v\in V}{f}_{vz}}={\displaystyle \sum _{u\in V}{f}_{zu}}$$

      d. G _xy is acyclic; there is no path (sequence of edges) that visits the same vertex twice.

Consider, for example, the graph G _xy in Figure 1b, which corresponds to the xy comparison of the network of four treatments of Figure 1a. The set of vertices is V = { x, y, u, v} and the set of directed edges is E = { xy, xv, vy, vu, uy}. Flows f _uv are given along the edges; their numerical values are equal to the respective absolute entries of h ^xy and the direction of their corresponding edge is indicated in the subscript. As properties (a) to (d) imply, the arrows in Figure 1b indicate that the outflows of x, as well as the inflows of y, equal 1, and that the inflows equal the outflows in the intermediate nodes u and v.

Streams

In Figure 1b there are three different paths from x to y, one based on direct evidence, { xy}, and two based on indirect evidence, { xv, vy} and { xv, vu, uy}. A path is a sequence of connected directed edges belonging to E, and we denote it as π _i. As property (d) implies, each node occurs at most once in π _i. Then, given the above properties of f _uv, we can assign a flow φ _i to each path π _i. Flow φ _i is equal to the smallest f _uv in the path π _i. Figure 1c shows the three paths from x to y; π ₁, π ₂ and π ₃, and their corresponding flows. Path π ₁ corresponds to xy and its flow, φ ₁, equals the flow of the single edge in path, f _xy = 0.635. Path π ₂ is constituted from two edges, xv and vy; thus, flow φ ₂ = min( f _xv, f _vy) = 0.251. The flow corresponding to the third path π ₃ is φ ₃ = min( f _xv, f _vu, f _uy) = 0.114.

We define a stream, S _i, as the composition of a path and its associated flow, S _i = ( φ _i, π _i) with i = 1, ..., I where I is the total number of streams; here I = 3. Note that it holds ${\displaystyle {\sum }_{i=1}^I{\phi }_i=1}$.

Proportion contributions of direct comparisons

In order to assign proportion contributions to each direct comparison, we need to split each stream’s flow to the involved edges in the stream’s path. Equation 3 can be re-written as

$${\hat{\theta }}_{xy}^N={\phi }_1{\hat{\theta }}_{xy}^D+{\phi }_2({\hat{\theta }}_{xv}^D-{\hat{\theta }}_{yv}^D)+{\phi }_3({\hat{\theta }}_{xv}^D-{\hat{\theta }}_{uv}^D-{\hat{\theta }}_{yu}^D)\mathbf{Equation}\mathbf{5}$$

with ${\displaystyle {\sum }_{i=1}^3{\phi }_i=1}$. It follows from the properties of the elements of set F, that each NMA effect can be written as a linear combination of direct and indirect effects, in the form of Equation 5. The effects are stochastically interdependent and, hence, their aggregation is different from the aggregation of studies in a pairwise meta-analysis.

To approximate the proportion contributions per comparison, we suggest dividing φ _i by the length of the respective path π _i, # π _i. This will leave the proportion contribution of the direct evidence of the same treatment comparison equal to the diagonal of the H matrix and assign to each comparison involved in an indirect route a portion of the respective stream’s flow. Note that directed edges might be involved in more than one path; we thus define the proportion contribution of an edge uv as

$$p_{uv}={\displaystyle \sum _{\forall iwhereuv\in {\pi }_i}{\phi }_i/\#{\pi }_i}\mathbf{Equation}\mathbf{6}$$

Figure 1d shows the derivation of the proportion contributions of each direct comparison in the network of topical antibiotics. Hence, from the row of the H matrix (0.635, 0.365, –0.114, –0.251, –0.114), which shows the absolute contributions of the direct effects ${\hat{\theta }}_{xy}^D,{\hat{\theta }}_{xv}^D,{\hat{\theta }}_{yv}^D,{\hat{\theta }}_{yu}^D,{\hat{\theta }}_{uv}^D$ to ${\hat{\theta }}_{xy}^N$, we approximated their proportion contribution as 63.5%, 16.4%, 3.8%, 12.6% and 3.8%, respectively.

Algorithm to decompose flows into proportion contributions

In this section, we present an iterative algorithm that generalizes the process outlined above to derive proportion contributions of each direct effect to the estimation of a ‘ x versus y’ NMA effect. We start by defining a graph G _xy from h ^xy.

The algorithm is described as follows:

• 0. Set initial graph G ₀ = ( V, E ₀, F ₀) = G _xy. E ₀ contains uv if h _uv > 0 or contains vu if h _uv < 0. The set of flows is F ₀ = { f _{0, uv} | uv ∈ E ₀}; numerical values of f _{0, uv} are equal to h _uv.

Then, repeat the process below I times, equal to the number of streams in G _i, until E _i = {Ø}.

• 1. In G _i–1, find the shortest path from x to y, π _i, and define its flow as φ _i = min{ f _{i–1, uv}, uv ∈ π _i}. Then, use π _i and φ _i to define the stream S _i = ( φ _i, π _i).
• 2. Recalculate the flow of edges uv ∈ π _i by subtracting φ _i from the flow of the edges of the stream found: f _i,uv = f _{i–1, uv} – φ _i ∀ uv ∈ π _i. The flow of the rest of the edges that do not belong to π _i remain unchanged: f _i,uv = f _{i–1, uv} ∀ uv ∉ π _i.
• 3. Define E _i as the set of edges uv for which f _i,uv > 0; this is E _i–1 after removing the edges with zero flow, E _i = E _i–1\{ uv | f _i,uv = 0}. Collect f _i,uv to form the set F _i = { f _i,uv | uv ∈ E _i}.
• 4. If E _i ≠ {Ø} define G _i = ( V, E _i, F _i) and go to step 1.

When the algorithm terminates, all streams S _i = ( φ _i, π _i) have been identified and Equation 6 is used to derive the proportion contributions p _uv.

Repeating the same process for all NMA effects, we derive all $p_{uv}^{xy}$ and collect them in a matrix P of the same dimensions as H. The presented algorithm could be described as a reverse maximum flow Edmonds Karp algorithm ⁸, but instead of adding we remove augmenting paths.

It is possible that multiple shortest paths exist; in this case, the order in which one chooses such a path could in principle result in different proportion contributions per comparison. We can, thus, use the following modification in the algorithm to impose consistency. Instead of selecting the shortest path, we assign cost values c _i,uv to each edge uv as follows: c _i,uv = 2 – f _{i–1, uv}. Then, we select the path from x to y with the minimum cost across comparisons included in π _i. The definition of the cost values c _i,uv assures that paths are selected from shortest to longest and removes any ambiguity regarding the selection of paths.

The starting point for the developed algorithm was a simplified version of the H matrix that does not consider the correlation induced by multi-arm trials. Alternatively, one could use the H matrix as described by König et al. ⁴ that extends the definition of the matrix for multi-arm designs. Note that any matrix whose rows can be interpreted as flow networks can be used as the starting point of the algorithm. The estimator of heterogeneity as well as the assumption of a different or common heterogeneity across comparisons in the network does not modify any aspect of the method.

Calculations in this paper were performed using R.

Application

We apply the algorithm described above to the network of topical antibiotics ⁷.

Proportion contributions of direct relative treatment effects to the estimation of the NMA effect between non-quinolone antibiotic and no treatment

Direct effects are obtained using the random effects model and the H matrix of dimension 6 ×5 is calculated using Equation 2. The H matrix, along with NMA effects, is given in Table 1.

Table 1. H matrix in the network of topical antibiotics without steroids for chronically discharging ears.

Columns correspond to direct comparisons and rows correspond to network meta-analysis (NMA) effects. Direct effects along with their variances and NMA effects with 95% confidence intervals (CIs) are given in the last column. Direct and NMA effects are measured as log odds ratios. Positive values favour the first treatment.

	xy	xv	yu	yv	uv	Direct effect	Variance of direct effect	NMA effect (95% CIs)
xy	0.635	0.365	–0.114	–0.251	–0.114	–2.29	0.42	–1.86 (–3.05,–0.67)
xu	0.603	0.397	0.632	–0.029	–0.368	–	–	–1.32 (–2.61,–0.02)
xv	0.545	0.455	0.170	0.375	0.170	0.35	0.63	–0.72 (–1.95,0.52)
yu	–0.032	0.032	0.745	0.223	–0.255	0.39	0.12	0.54 (–0.07,1.16)
yv	–0.090	0.090	0.284	0.627	0.284	1.24	0.15	1.14 (0.47,1.81)
uv	–0.058	0.058	–0.462	0.404	0.538	0.53	0.22	0.60 (–0.12,1.33)

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

We begin by applying step 0 of the algorithm. We construct the network G ₀ = G _xy = ( V, E ₀, F ₀) with source x and sink y corresponding to row h ^xy. The set of vertices is V = { x, y, u, v} and the set of directed edges, taking into account the signs of the elements of h ^xy, is E ₀ = { xy, xv, vy, vu, uy}. The set of flows is F ₀ = { f _{0, uv} | uv ∈ E ₀}, where f _{0, uv} equal the respective absolute values of Table 1 and are given along the edges of Figure 1b.

Then, we apply the developed iterative algorithm until E _i = {Ø}. The iterations of the algorithm equal the number of existing streams from x to y and are illustrated in Figure 2.

Figure 2. Illustration of the steps of the algorithm for approximating proportion contributions per comparison in the network of topical antibiotics without steroids for chronically discharging ears focusing on the comparison ‘ x versus y’.

Treatment labels: x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

First iteration

• 1. In G ₀, find the shortest path from x to y, π ₁ = { xy}. Define its flow as φ ₁ = min{ f _{0, uv}, uv ∈ π ₁} = f _{0, xy} = 0.635. Define stream S ₁ = ( φ ₁, π ₁) ( Figure 2a).
• 2. Recalculate the flow of edge xy ∈ π ₁ as f _{1, xy} = f _{0, xy} – φ ₁ = 0.635 – 0.635 = 0. The flow of the rest of the comparisons remains unchanged: f _{1, uv} = f _{0, uv} ∀ uv ∉ π ₁ ( Figure 2b).
• 3. Define E ₁ as the set of edges uv for which f _{1, uv} > 0; edge xy is removed since its flow is zero, E ₁ = E ₀\{ xy}. Collect f _{1, uv} to form the set F ₁ = { f _{1, uv} | uv ∈ E ₁} ( Figure 2c).
• 4. Define G ₁ = ( V, E ₁, F ₁). As E ₁ = { xv, vy, vu, uy} ≠ {Ø}, go to step 1 ( Figure 2c).

Second iteration

• 1. In G ₁, find the shortest path from x to y, π ₂ = { xv,vy}. Define its flow as φ ₂ = min{ f _{1, uv}, uv ∈ π ₂} = f _{1, vy} = 0.251. Define stream S ₂ = ( φ ₂, π ₂) ( Figure 2d).
• 2. Recalculate the flow of edges xv and vy as f _{2, xv} = f _{1, xv} – φ ₂ = 0.365 – 0.251 = 0.114 and f _{2, vy} = f _{1, vy} – φ ₂ = 0.251 – 0.251 = 0. The flow of the rest of the comparisons remains unchanged: f _{2, uv} = f _{1, uv} ∀ uv ∉ π ₂ ( Figure 2e).
• 3. Define E ₂ as the set of edges uv for which f _{2, uv} > 0; edge vy is removed since its flow is zero and thus E ₂ = E ₁\{ vy}. Collect f _{2, uv} to form the set F ₂ = { f _{2, uv} | uv ∈ E ₂} ( Figure 2f).
• 4. Define G ₂ = ( V, E ₂, F ₂). As E ₂ = { xv, vu, uy} ≠ {Ø}, go to step 1 ( Figure 2f).

Third iteration

• 1. In G ₂, find the shortest path from x to y, π ₃ = { xv, vu, uy}. Define its flow as φ ₃ = min{ f _{2, uv}, uv ∈ π ₃} = 0.114. Define stream S ₃ = ( φ ₃, π ₃) ( Figure 2g).
• 2. Recalculate the flow of edges xv, vu and uy as f _{3, xv} = f _{3, vu} = f _{3, uy} = f _{2, xv} – φ ₃ = 0.114 – 0.114 = 0 ( Figure 2h).
• 3. Define E ₃ as the set of edges uv for which f _{3, uv} > 0; edges xv, vu and uy are removed since their flow is zero, E ₃ = E ₂\{ xv, vu, uy}. Collect f _{3, uv} to form the set F ₃ = { f _{3, uv} | uv ∈ E ₃} ( Figure 2i).
• 4. Define G ₃ = ( V, E ₃, F ₃). The set of direct edges is E ₃ = {Ø} and the algorithm is terminated at this point ( Figure 2i).

Figure 1c shows the flows of the three streams identified when applying the above algorithm. We then calculate the proportion contributions of each comparison to the ‘ x versus y’ NMA treatment effect estimate using Equation 6 ( Figure 1d). For instance, to calculate p _xv we first have to identify the relevant paths; these were π ₂ and π ₃. Consequently,

$$p_{xv}=\frac{{\phi }_2}{\left|{\pi }_2\right|}+\frac{{\phi }_3}{\left|{\pi }_3\right|}=\frac{0.251}{2}+\frac{0.114}{3}=0.164=16.4\%$$

The calculations for deriving the proportion contributions of the other comparisons are shown in Table 2. Applying the algorithm to all NMA treatment effect estimates we get the entire proportion contribution matrix P.

Table 2. Proportion contributions of direct comparisons to the ‘ x versus y’ network meta-analysis treatment effect in the network of topical antibiotics without steroids for chronically discharging ears.

	xy	xv	yu	yv	uv
xy	$p_{xy}=\frac{{\phi }_1}{\left|{\pi }_1\right|}$ = 63.5%	$p_{xv}=\frac{{\phi }_2}{\left|{\pi }_2\right|}+\frac{{\phi }_3}{\left|{\pi }_3\right|}$ = 16.4%	$p_{yu}=\frac{{\phi }_3}{\left|{\pi }_3\right|}$ = 3.8%	$p_{yv}=\frac{{\phi }_2}{\left|{\pi }_2\right|}$ = 12.6%	$p_{uv}=\frac{{\phi }_3}{\left|{\pi }_3\right|}$ = 3.8%

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Proportion study contributions

Matrix P ( Table 3) shows the proportion contributions of each direct comparison to each NMA treatment effect estimate. These proportions can be distributed to individual studies within each comparison according to their weights from direct meta-analyses. For example, p _xy = 63.5% and there are two studies examining the xy comparison. The individual study weights for the two studies are 0.69 and 1.54 resulting in study proportion contributions of $\frac{0.69}{0.69+1.54}63.5\%=19.6\%$ and $\frac{1.54}{0.69+1.54}63.5\%=43.8\%$ to the xy NMA treatment effect estimate. The application of this process to the entire matrix P leads to the matrix P* shown in Table 4. Adjusted weights as proposed by Rücker & Schwarzer ⁹ are used for multi-arm studies.

Table 3. Proportion contribution matrix P for the network of topical antibiotics without steroids for chronically discharging ears.

Cells show the proportion contribution of direct comparisons indicated in the column to the network meta-analysis treatment effects indicated in the rows.

	xy	xv	yu	yv	uv
xy	63.5%	16.4%	3.8%	12.6%	3.8%
xu	30.1%	19.4%	31.1%	1%	18.4%
xv	24.4%	45.5%	5.7%	18.8%	5.7%
yu	1.1%	1.1%	74.5%	11.1%	12.2%
yv	4.5%	4.5%	14.2%	62.7%	14.2%
uv	1.9%	1.9%	22.1%	20.2%	53.8%

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Table 4. Study proportion contribution matrix P* for the network of topical antibiotics without steroids for chronically discharging ears.

Cells show the proportion contribution of individual studies indicated in the column to the network meta-analysis treatment effects indicated in the rows.

	Study 1	Study 2	Study 3	Study 4	Study 5	Study 6	Study 7	Study 8	Study 9	Study 10	Study 11	Study 12	Study 13
xy	19.7	63.4	2.8	0.5	0.6	0.4	0.6	0.7	2.7	1.3	0.8	2.3	4.2
xu	9.3	40.4	8.8	4.4	5.3	3.5	5.1	5.4	6.9	6.5	3.9	0.2	0.3
xv	7.6	67.1	4.2	0.8	1	0.6	0.9	1	4.1	2	1.2	3.4	6.2
yu	0.3	4.63	13.6	10.5	12.7	8.4	12.2	12.9	12.2	4.3	2.6	2	3.7
yv	1.4	23.5	12.1	2	2.4	1.6	2.3	2.4	12.2	5	3	11.3	20.7
uv	0.6	8.4	19	3.1	3.8	2.5	3.6	3.8	14.4	19.1	11.5	3.6	6.7

x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Using proportion study contributions to quantify the impact of a characteristic in a direct comparison

The algorithm translating the H matrix into study proportion contributions can be applied to quantify the influence that a study-level characteristic has in the estimation of the NMA effects. For instance, if risk of bias judgements for individual studies are available, we can obtain an approximation of the proportion of each NMA treatment effect estimate that is coming from studies with a ‘high’, ‘moderate’, or ‘low’ risk of bias. Salanti et al. suggested the visualisation of this information using a bar plot, in which direct comparisons of the same risk of bias level have been grouped ³. Figure 3 shows such a bar plot using the algorithm described in this paper and distributing comparison proportion contributions to study proportion contributions; inspecting Figure 3 can support judgements regarding the importance of study limitations for different NMA treatment effect estimates. For instance, studies with high risk of bias contribute more than 50% in the estimation of the ‘ u versus v’ comparison, potentially reducing the confidence that we can place in this particular NMA treatment effect estimate.

Figure 3. Bar plot showing the study proportion contributions of direct comparisons with low (green), moderate (yellow) and high (red) risk of bias.

The bar plot has been produced in CINeMA (Confidence In Network Meta-Analysis) software ¹¹. Studies are synthesized using the random effects model. x, no treatment; y, quinolone antibiotic; u, non-quinolone antibiotic; v, antiseptic.

Proportion contributions of direct comparisons in a large complex network of interventions

So far, we have illustrated how to derive proportion contributions for a network with four treatments. However, the algorithm can be straightforwardly applied to large networks of any structure, as soon as the involved treatments are connected. Consider for example a large network examining antimanic drugs ( Figure 4) ¹⁰. Let us concentrate on the comparison PLA versus OLA (‘placebo versus olanzapine’); the algorithm starts by applying step 0 and constructing network G ₀. Then, we continue by finding the shortest path in the first iteration, which corresponds to the direct comparison, and define its flow and stream S ₁ = ( φ ₁, π ₁). The number of algorithm’s iterations is equal to the number of streams from placebo to olanzapine, which turns out to be 16. The resulting entire proportion matrix is given in Supplementary File 2.

Figure 4. Network plot for the network of antimanic drugs.

ASE, asenapine; ARI, aripiprazole; PLA, placebo; HAL, haloperidol; QUE, quetiapine; LITH, lithium; ZIP, ziprasidone; OLA, olanzapine; DIV, divalproex; RIS, risperidone; CARB, carbamazepine; LAM, lamotrigine; PAL, paliperidone; TOP, topiramate; ASE, asenapine.

Outcome data from the example network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations ⁷

Data labels: study, name of individual studies; id, id of the individual studies; t; treatment; r, number of events; n, sample size; rob, risk of bias per study.

Copyright: © 2018 Papakonstantinou T et al.

Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication).

Discussion

In this paper, we present a new approach to derive proportion contributions of individual studies to the treatment effect estimates in NMA. We made use of the fact that the composition of network treatment effect estimates can be interpreted as a flow of evidence. An assumption that underlies our algorithm is the equal split of the stream flow to the involved comparisons. Although indirect effects are not weighted averages, we find this approximation to be a pragmatic approach that reasonably reflects the amount that each comparison contributes to network effects. The derivation of the contribution of sources of evidence in a Bayesian NMA is not straightforward, as no analogue to the H matrix exists. The application of the method described in this paper, however, can be used to derive useful approximations of the contributions of studies.

Applying the algorithm to networks of interventions can be used to quantify the contribution of potential study limitations to the NMA treatment effect estimates. Study limitations may lead to biased NMA treatment effect; however, the amount and direction of bias in the NMA treatment effect as a result of the within-study bias is not straightforward to define and is not currently accommodated within the proportion contribution matrix. First, a single biased trial may affect an entire indirect route; thus, even if its proportion contribution is small, its consequences in the estimation of the NMA treatment effect may be important. Second, the direction of bias across studies involved in a stream may vary. For example, bias in two comparisons in the same stream may either cancel out or add-up in favor of one of the two treatments. We aim to extend the methods presented in this paper to develop a network meta-regression model that will use the direction and the amount of bias to determine whether and how much NMA treatment effect estimates will be biased as the result of within-study bias.

Alternative methods to derive the relative contribution of all sources of evidence have been developed ^{12, 13}. Krahn et al. define influence functions to describe the extent to which changes in study effects would be translated into NMA treatment effects ¹³. An alternative approach, based on the decomposition of Fisher’s information matrix, has been proposed to derive proportion study weights in a variety of meta-analysis models, including meta-regression, network meta-analysis and individual patient data meta-analysis ¹². Further investigation of the degree of agreement between our algorithm and that of Riley et al. ¹² would be of interest.

In the example implemented in the Application, there is no other possible set of paths, and associated streams, that could be selected from x to y in order to partition the inflow of x: π ₁, π ₂ and π ₃ is the only possible set of streams ( Figure 1c). Thus, even if we were taking paths using different criteria, i.e. from longest to shortest, according to values from the H matrix or even randomly, the proportion contributions given in Table 2 would be identical. However, cases exist where the selection of paths does influence the derivation of the P matrix. In Supplementary File 3, we elaborate on the selection of direct paths in the algorithm and discuss some alternative modifications of the algorithm. We are planning to examine the properties of the different approaches in greater detail in a follow up project.

We offer an R package ¹⁴, which we also use in the software application CINeMA (Confidence In Network Meta-Analysis) ¹¹, that aims to simplify the evaluation of confidence in the findings from NMA. While CINeMA largely follows the framework previously developed by Salanti et al. ³, the refinement of several methodological aspects is currently under development. Core aspects of the approach include the consideration of the relative contributions of each direct comparison to each NMA treatment effect estimate. To this end, CINeMA uses the proportion contribution matrix as described in this paper. The command netweight in Stata has also been updated to use the described approach.

We believe that the approach described in this paper is a useful and novel addition to network meta-analysis methodology, which allows the consistent derivation of the proportion contributions of direct evidence from individual studies to network treatment effects.

Data availability

The data referenced by this article are under copyright with the following copyright statement: Copyright: © 2018 Papakonstantinou T et al.

Data associated with the article are available under the terms of the Creative Commons Zero "No rights reserved" data waiver (CC0 1.0 Public domain dedication). http://creativecommons.org/publicdomain/zero/1.0/

Dataset 1: Outcome data from the example network of topical antibiotics for the treatment of chronic otitis media with ear discharge in patients with eardrum perforations ⁷. Data labels: study, name of individual studies; id, id of the individual studies; t; treatment; r, number of events; n, sample size; rob, risk of bias per study. DOI: 10.5256/f1000research.14770.d203174 ¹⁵.

Funding Statement

TP received funding from a Campbell Collaboration Method Grant [grant no. CMG2.04].

The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Supplementary material

Supplementary File 1. Proof that the sum of all outflows from the source is equal to the sum of all inflows to the sink and both are 1.

Supplementary File 2. Proportion contribution matrix P for the network of antimanic drugs. Cells show the proportion contribution of direct comparisons indicated in the column to the network meta-analysis treatment effects indicated in the rows. ASE, asenapine; ARI, aripiprazole; PLA, placebo; HAL, haloperidol; QUE, quetiapine; LITH, lithium; ZIP, ziprasidone; OLA, olanzapine; DIV, divalproex; RIS, risperidone; CARB, carbamazepine; LAM, lamotrigine; PAL, paliperidone; TOP, topiramate; ASE, asenapine.

Supplementary File 3. Considerations on the selection of streams