Skip to main content
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2017 Aug 1.
Published in final edited form as: J Abnorm Psychol. 2016 Jun 16;125(6):747–757. doi: 10.1037/abn0000181

Identifying Highly Influential Nodes in the Complicated Grief Network

Donald J Robinaugh 1,2, Alexander J Millner 3, Richard J McNally 3
PMCID: PMC5060093  NIHMSID: NIHMS788995  PMID: 27505622

Abstract

The network approach to psychopathology conceptualizes mental disorders as networks of mutually reinforcing nodes (i.e., symptoms). Researchers adopting this approach have suggested that network topology can be used to identify influential nodes, with nodes central to the network having the greatest influence on the development and maintenance of the disorder. However, because commonly used centrality indices do not distinguish between positive and negative edges, they may not adequately assess the nature and strength of a node’s influence within the network. To address this limitation, we developed two indices of a node’s expected influence (EI) that account for the presence of negative edges. To evaluate centrality and EI indices, we simulated single-node interventions on randomly generated networks. In networks with exclusively positive edges, centrality and EI were both strongly associated with observed node influence. In networks with negative edges, EI was more strongly associated with observed influence than was centrality. We then used data from a longitudinal study of bereavement to examine the association between (a) a node’s centrality and EI in the complicated grief (CG) network and (b) the strength of association between change in that node and change in the remainder of the CG network from 6 to 18-months post-loss. Centrality and EI were both correlated with the strength of the association between node change and network change. These findings suggest high-EI nodes, such as emotional pain and feeling of emptiness, may be especially important to the etiology and treatment of CG.

Keywords: network analysis, centrality, complicated grief, expected influence, persistent complex bereavement disorder

General Scientific Summary

Complicated grief can be conceptualized as a network of mutually reinforcing symptoms. Centrality and expected influence indices aim to use the structure of the complicated grief network to identify symptoms that should be especially important to its development and persistence. We found that change in symptoms with high expected influence was more strongly tied to change in the severity of complicated grief than was change in symptoms with low expected influence, suggesting that this index is able to identify symptoms that may play an important role in the etiology and treatment of complicated grief.


The network approach to mental disorders posits that the symptoms of a mental disorder cohere as a syndrome, not because they are caused by a common underlying disease entity, but because of causal relations among the symptoms themselves (Borsboom & Cramer, 2013). In other words, symptoms are not passive and interchangeable indicators of an underlying disorder in the way that coughing and spitting up blood are indicators of a lung tumor. Rather, they are active psychological variables capable of affecting one another and, thereby contributing to the persistent activation of the overall network. From this perspective, the symptoms of a mental disorder and the relations among them form a causal system that is, itself, constitutive of the disorder.

Networks contain two fundamental components: nodes and edges. In mental disorder networks, nodes typically represent the elements of a syndrome (“symptoms”, in traditional psychiatric parlance), such as sleep difficulty in major depressive disorder. Greater severity of the symptom (e.g., more sleep difficulty) corresponds to greater activation of the node. The relationship between two nodes is represented by an edge (i.e., the line connecting the nodes). Edges can be undirected, indicating only that two nodes are connected (e.g., sleep difficulty and poor concentration are correlated with one another), or directed, indicating the direction of the connection between two nodes (e.g., sleep difficulty leads to poor concentration). They can also be unweighted, representing only the presence of a connection, or weighted, representing the strength of the connection. Within the framework of current clinical practice, an episode of a disorder corresponds to the above-threshold activation of a specified number of nodes of the network (e.g., five of nine elements of the depression network; American Psychiatric Association [APA], 2013). Greater disorder severity corresponds to greater overall network activation (i.e., greater sum node activation). Recovery corresponds to deactivation of nodes.

Centrality

Importantly, nodes differ in the role they play in the network. High centrality nodes have strong connections to many other nodes, and act as hubs that connect otherwise disparate nodes to one another. Low centrality nodes exist on the periphery of the network, with fewer and weaker connections to other nodes of the network. Centrality is commonly assessed with three indices: strength, closeness, and betweenness. A node’s strength is the sum of the absolute value of its connections with other nodes in the network. A node’s closeness is the average shortest path between a given node and the remaining nodes in the network. Nodes with higher closeness are more proximally connected to the rest of the network. A node’s betweenness is the number of times in which a given node lies on the shortest path between two other nodes. Together, these indices quantify the position of a node within the network.

Researchers taking the network perspective of psychopathology have argued that the strong interconnectedness of high centrality nodes may make them especially important to the etiology and treatment of mental disorders (Borsboom & Cramer, 2013; Cramer & Borsboom, 2015; Fried, Epskamp, Nesse, Tuerlinckx, & Borsboom, 2016; McNally et al., 2015; Robinaugh, LeBlanc, Vuletich, & McNally, 2014). In other words, because highly central nodes have more and stronger connections to other nodes than do low-centrality nodes, they should have greater influence on overall network activation, be more influenced by overall network activation, or both. As highly central nodes go, so should go the network.

For example, we recently proposed that the bereavement-specific syndrome, complicated grief (CG; also known as Persistent Complex Bereavement Disorder; APA, 2013), is best conceptualized as a network of mutually-reinforcing cognitions, emotions, and behaviors (Robinaugh et al., 2014). The elements of this syndrome include emotional pain, intense yearning or longing for the deceased, preoccupation with thoughts about the deceased or their death, grief-related avoidance behavior, and feelings that life is empty or without meaning. The death of a loved one activates specific elements of the syndrome and the mutually reinforcing relationships among them lead to the emergence and persistence of the network that constitutes CG. In our initial examination of the CG network, we found that emotional pain was highly central to the network and concluded that it may play an important role in the persistence and remission of the CG syndrome (Robinaugh et al., 2014). Treating emotional pain, we reasoned, should diminish the activation of associated nodes, enabling therapeutic benefits to propagate throughout the network.

The hypothesis that high centrality nodes are especially important to the development, persistence, or remission of mental disorder networks remains largely untested and there is reason to believe it may not always hold true. The position of a node within a network is only one factor that determines its importance. One must also consider what is being transmitted through the network and how that that transmission occurs (Borgatti, 2005). In mental disorder networks, researchers have assumed that causal influence is being transmitted. Change in the severity of a node causally influences all nodes receiving an incoming edge from that node. Importantly, this influence can be positive (increase in Node A causes an increase in Node B) or negative (increase in Node A causes a decrease in Node C). The presence of negative edges in mental disorder networks is noteworthy because commonly used measures of node centrality, such as strength, closeness, and betweenness, are typically calculated only in networks with exclusively positive edges. To address the presence of negative edges in mental disorder networks, researchers have simply transformed negative edge weights into positive edge weights when calculating centrality indices. However, because these indices do not distinguish between positive and negative edges, two nodes with equivalent centrality may have opposing effects on the rest of the network. Moreover, a node with a comparable number of strong positive and negative edges may have little to no cumulative influence on overall network activation. Deactivating such a node would decrease activation of neighboring nodes connected by a positive edges but would increase the activation of neighboring nodes connected by a negative edges, thereby resulting in little or no cumulative change in overall network activation. In other words, a node may be highly central without being highly influential.

Expected Influence

To better identify highly influential nodes in mental disorder networks, it may be necessary to distinguish between positive and negative edges. To address this possibility, we propose two new measures of node importance. In contrast to measures of centrality, which quantify the position of a node within a network, these indices aim to assess the nature and strength of a node’s cumulative influence within the network, and thus the role it may be expected to play in the activation, persistence, and remission of the network. To reflect that aim, we will refer to these indices as measures of expected influence. In this article, we focus on the use of these indices in undirected networks (i.e., networks in which the direction of influence is not specified). The implications of using these indices in directed networks are detailed in the Discussion section.

One-step expected influence (EI1) aims to assess a node’s influence with its immediate neighbors (i.e., the nodes with which it shares an edge). The formula for EI1 is identical to the formula for node strength except that it retains the positive or negative value of the edge weight. A given Node i’s EI1 is the summed weight of its edges shared with the remaining nodes in the network (j). In formula 1 below, aij is an adjacency matrix whose elements are binary values indicating the presence (1) or absence (0) of an edge between Node i and Node j and wij is an adjacency matrix whose elements range from −1 to 1 and indicate the weight of the edge between Node i and Node j. For a node with exclusively positive edges, EI1 is equal to node strength. However, for a node with negative edges, these measures diverge. A negative edge increases a node’s strength but decreases its EI1, resulting in either diminished positive EI1 (if its positive edges outweigh its negative edges) or strengthened negative EI1 (if its negative edges outweigh the positive edges).

EI1i=j=1Naijwij (1)

EI1 provides a conceptually straightforward assessment of a node’s expected influence. However, it does not incorporate information about the expected influence of a node’s neighbors, information highly relevant to the ultimate influence of the node within the network. For example, if Node A is connected only to Node B, and Node B has low expected influence, then changes in Node A will have little influence on the remainder of the network. However, if Node B is a highly influential node with many strong edges, the changes in Node A may have a large effect on the network by virtue of its influence on the highly influential Node B. Two-step expected influence (EI2; see formula 2) accounts for both the immediate influence of a node within the network and the secondary influence on the network through its neighbors. A node’s EI2 is its EI1 plus the sum of the EI1 values of the remaining nodes in the network (EI1j=k=1Najkwjk, where ajkwjk is the weighted edge between node j and all other nodes in the network [k]) multiplied by the weighted edge between node i and node j (aijwij). The added expected influence of the neighboring node is weighted because the secondary influence of Node A through Node B will vary depending on the strength of the edge between Node A and Node B.

EI2i=j=1Naijwij+j=1Naijwijk=1Najkwjk (2)

The Current Study

The identification of highly influential nodes in mental disorder networks is of fundamental importance to our understanding of mental disorders and our ability to treat them. In the current study, we examined whether centrality and expected influence indices were able to identify highly influential nodes in a CG network calculated from empirical data and in simulated networks with characteristics comparable to those of the CG network.

Our first aim was to use simulated data to obtain an initial evaluation of how centrality and expected influence indices performed in the context of networks with negative edges. To that end, we simulated single-node interventions in randomly generated networks by “treating” or “deactivating” individual nodes (setting their value to 0) and evaluating the effect of this intervention on overall network activation (cf. "knock out" interventions in Bramson & Vandermarliere, 2016). We hypothesized that in networks with exclusively positive edges, both centrality and expected influence indices would be strongly positively associated with observed node influence (i.e., change in the overall activation of the rest of the network induced by deactivating that node). We further hypothesized that in networks with negative edges, expected influence indices would better predict observed node influence than would centrality indices.

Our second aim was to evaluate these indices in an empirical network of CG symptoms derived from a longitudinal study of conjugal bereavement in which CG was assessed at 6- and 18-months post-loss (Time 1 and Time 2, respectively). If a node is highly influential in the network, then change in the activation of that node over time should be associated with change in the activation of the remainder of the CG network. In contrast, if a node is not influential, change in that node should not be tied to changes in the overall network. Accordingly, we examined the relationship between (a) the centrality and expected influence of a node at Time 1 and (b) the correlation between change in that node (Time 2 − Time 1) and change in overall network activation (Time 2 − Time 1). We hypothesized that the higher a node’s centrality and expected influence at Time 1, the stronger the association between node change and overall network change.

Method

Aim 1

As an initial assessment of centrality and expected influence indices, we evaluated how well they measured the impact of individual nodes on the overall network in simulated single-node interventions. In these simulated single-node interventions we “treated” or “deactivated” an individual node (the target node) by setting its activation to 0. We then examined the effect of this intervention on the remainder of the network. A substantive example of this type of intervention would be treating insomnia and examining the subsequent effect on other symptoms of depression. We defined a node’s “observed influence” as the change in the sum activation of the remaining nodes in the network (i.e., change in overall network activation minus change in the target node) induced by deactivating that node. Next, we calculated the correlation between a node’s centrality and expected influence indices and its observed influence. To obtain a reliable estimate of the strength of this correlation, we repeated this procedure 500 times and examined the distribution of the correlation coefficients. Because we were especially interested in how the presence of negative edges would affect the strength of the associations among a node’s centrality, expected influence, and observed influence, we calculated this distribution of 500 correlation coefficients in each of four conditions: networks with 0%, 5%, 10%, and 25% negative edges.

Each network simulation was carried out in three steps. A complete description of these steps appears in Supplementary Materials A. We present an abbreviated description of this process here. First, we generated a random network and assessed its properties. To do so, we used the erdos.renyi.game random graph generation tool from the R package igraph (Csardi & T., 2006) to create an unweighted and undirected adjacency matrix and then assigned edge weights by multiplying that unweighted adjacency matrix by a matrix of values randomly sampled from a gamma distribution (shape = .85, rate = 12) where the minimum value of the distribution was set to .05. We used the G (n, p) variant of the Erdős-Rényi graph model, with n (the number of nodes) set to 13 and p (the probability of an edge being present) set to .50. These parameters and the parameters of the gamma distribution were selected to produce networks with a size, unweighted network density, and distribution of edge weights comparable to that of the empirical CG network examined in our second aim.

We used the R package qgraph (Epskamp, Cramer, Waldorp, Schmittmann, & Borsboom, 2012) to calculate three indices of node centrality: node strength, closeness, and betweenness. For each centrality index, higher values reflect greater centrality in the network. We then calculated EI1 and EI2 using the formulas described in the introduction. If the expected influence of a node is positive, changes in the node should produce changes in the overall network in the same direction (e.g., decreases in node activation should lead to decreases in overall network activation). If the expected influence of a node is negative, changes in the node should produce network changes in the opposite direction (e.g., decreases in node activation should lead to increases in overall network activation).

In step two, we simulated the dynamics of the network by iteratively calculating the value of each node at Time t (node it) as a function of the value of all remaining nodes in the network at the pervious iteration (node jt−1) weighted by the edges shared between node i and node j. For the purposes of these simulations, we assumed that nodes do not have negative activation and that nodes cannot increase in severity indefinitely. Accordingly, we defined nodes as ranging in activation from 0 (absent) to 1 (highest severity). We further assumed that increases in activation are not linear and that, as node activation approaches 1, more incoming influence from other nodes is needed to increase node activation than is needed at lower levels of node activation. No such assumption was made as node activation approached 0. For these simulations, we defined the value of node i at Time 2 only as a function of the remaining nodes in the network and did not incorporate external influence or self-loops (i.e., the effect of the node on itself). The implications of omitting these parameters for the simulation are discussed in the limitations section of this manuscript. We assigned all nodes an initial starting value of .50 and then performed 30 iterations of this simulation to allow the network to reach of point of stable activation in which the amount of incoming influence for each node was insufficient to further increase or decrease activation of the node.

Finally, in step 3, we evaluated the effect of “deactivating” a single node by setting the activation of the target node to 0 and continuing to simulate all other nodes as a function of the remaining nodes in the network as described in Step 2. We then assessed the change in the remainder of the network over 30 iterations post-intervention (i.e., the observed influence of the node on the network). This process was repeated for each node in the network. We then calculated the correlation between (a) centrality and expected influence indices for each node and (b) the observed influence of each node.

Aim 2

Our second aim was to evaluate these indices in an empirical CG network based on data from the Changing Lives of Older Couples (CLOC) study, a large longitudinal study of conjugal bereavement (for previous studies using this dataset to investigate mental disorder networks, including CG, see Fried et al., 2015; Robinaugh et al., 2014). In the CLOC study, researchers used a two-stage area probability sample to collect baseline data from older married adults (men aged 65 or older and women married to a man 65 or older; n = 1,532). They then inspected newspaper obituaries and public death records to identify baseline subjects who subsequently experienced the death of a spouse. Of those who had lost a spouse (n = 335), 250 participated in the first wave of follow-up interviews that occurred 6 months after the loss (Time 1) and 210 subjects participated in another interview 18 months after the loss (Time 2). A subset of subjects (n = 106) also participated in a third interview 48 months after the loss. In the current study, we analyzed data from subjects who completed both Time 1 and Time 2 assessments (n = 195). Most subjects were Caucasian (84.6%) and female (87.0%). For further information about the CLOC study, see Carr, Nesse, and Wortman (2006).

Assessment of complicated grief

The CLOC study was conducted long before diagnostic criteria for CG appeared in the 5th edition of the Diagnostic and Statistical Manual of Mental Disorders (DSM-5; APA, 2013) under the name Persistent Complex Bereavement Disorder. Nonetheless, the survey contained items that correspond closely to the symptoms enumerated in the DSM-5. We used those items to assess 13 of the 16 CG symptoms (see Robinaugh et al., 2014 for further details about the items corresponding to these 13 symptoms). For most CLOC variables, subjects either rated symptom frequency ranging from Never to Often or expressed their strength of agreement to statements ranging from Not True to Very True. In both cases, subjects used a 4-point scale. Items not rated on a 4-point scale were rescaled so that the minimum score was 1 and the maximum score was 4, thereby maintaining consistency with the other items. We used mean replacement to address the small proportion of missing data (proportion missing = 0.49%). Where multiple variables assessed a symptom, we calculated the mean of those variables. These data preparation procedures produced an assessment of node activation (i.e., symptom severity) for 13 of the 16 CG nodes, with higher scores indicating greater node activation (range = 1 – 4).

Overall CG network activation

We calculated overall CG network activation for each individual as the summed level of activation of each node using the full four-point scale for each node (range of possible scores = 13–52). This “overall network activation” score is the equivalent of using sum symptom severity (e.g., a symptom inventory total score) as an index of disorder severity. Overall network activation scores ranged from 16.13 to 48.00 at Time 1 (M = 29.02, SD = 7.19) and 14.27 to 47.67 at Time 2 (M = 26.16, SD = 6.47). Change in overall network activation (Time 2 − Time 1) ranged from −20.68 to 10.37 (M = −2.86, SD =4.88). Additional information about change in the CG network appears in Supplementary Materials B.

Network analysis

We used the R package qgraph (Epskamp et al., 2012) to calculate two networks: an association network in which edges represented the zero-order correlation between nodes and a “LASSO” (Least Absolute Shrinkage and Selection Operator) network in which edges represented the regularized partial-correlation (Friedman, Hastie, & Tibshirani, 2008) between two nodes after accounting statistically for the effect of the remaining nodes in the network. Both networks were weighted and undirected, specifying the strength but not the direction of the relationship. The graph of the “lasso” network appears in Supplementary Materials C. The graph of the association network appears in Supplementary Materials D. We then calculated three indices of node centrality (node strength, closeness, and betweenness) and two measures of node expected influence (EI1 and EI2) for both networks.

Recently, researchers have found that association networks may fail to adequately detect network structure (Schmittmann, Jahfari, Borsboom, Savi, & Waldorp, 2015). Accordingly, we will focus the presentation and discussion of our results on the lasso network. The complete results for association networks can be found in Supplementary Materials D.

Results

Aim 1

The mean correlations between centrality and expected influence indices from the network simulations appear in Table 1. In the 500 randomly generated networks with exclusively positive edges, centrality and expected influence indices tended to be strongly correlated with one another, mean rs ≥ .78. The correlations among the three centrality indices remained highly consistent across conditions. However, the correlations between centrality indices and expected influence indices diminished as the proportion of negative edges increased. In networks with 25% negative edges, measures of centrality were only moderately associated with EI1, mean rs = .36 – .44, and EI2, mean rs = .42–.52. In other words, in networks with a relatively high proportion of negative edges, these indices were no longer in strong agreement about which nodes were most important.

Table 1.

Mean correlations among centrality and expected influence indices in randomly generated Erdős-Rényi networks

Closeness Betweenness Strength EI1
Positive Edges

Betweenness .80 (.09)
Strength .89 (.06) .81 (.09)
EI1 .89 (.06) .81 (.09) 1.00 (.00)
EI2 .91 (.06) .78 (.10) .99 (.01) .99 (.01)

5% Negative Edges

Betweenness .81 (.08)
Strength .89 (.06) .81 (.09)
EI1 .75 (.18) .68 (.20) .84 (.17)
EI2 .79 (.16) .68 (.18) .86 (.14) .99 (.01)

10% Negative Edges

Betweenness .80 (.08)
Strength .89 (.06) .80 (.10)
EI1 .65 (.20) .59 (.21) .74 (.19)
EI2 .70 (.18) .61 (.19) .77 (.16) .98 (.01)

25% Negative Edges

Betweenness .80 (.07)
Strength .89 (.06) .81 (.09)
EI1 .39 (.30) .36 (.31) .44 (.30)
EI2 .47 (.29) .42 (.28) .52 (.27) .97 (.02)

Note. In each of our four conditions (i.e., 0%, 5%, 10%, and 25% negative edges) we randomly generated 500 Erdős-Rényi networks and calculated the correlations among centrality and expected influence indices within those networks. The values reported here are the mean and standard deviation from the distribution of 500 correlation coefficients calculated in each condition.

EI1 = one-step expected influence. EI2 = two-step expected influence.

Figure 1 depicts the distributions of the correlation coefficients between (a) centrality and expected influences and (b) observed node influence in the simulated single-node interventions. When networks contained only positive edges, centrality indices and expected influence indices were all consistently highly correlated with observed node influence: closeness, M (SD) = .82 (.09); betweenness, M (SD) = .86 (.09); strength, M (SD) = .96 (.03); EI1, M (SD) = .96 (.03); EI2, M (SD) = .93 (.04). However, as networks contained more negative edges, centrality indices were increasingly poor predictors of observed node influence. In networks with 25% negative edges, the mean correlation was only moderate, closeness: M (SD) =.45 (.30); betweenness, M (SD) = .44 (.33); strength M (SD) = .50 (.30). Moreover, centrality indices frequently failed to be even minimally correlated with observed node influence (r < .1 in 13%, 16%, and 10% of networks, for closeness, betweenness, and strength, respectively. In contrast, expected node influence indices continued to be strongly correlated with observed influence in networks with as much as 25% negative edges, EI1, M (SD) =.80 (.16); EI2, M (SD) = .83 (.15). Overall, these simulations suggest that expected influence indices better predict observed node influence than do centrality indices when networks contain both positive and negative edges.

Figure 1.

Figure 1

Distribution of correlations between (a) centrality and expected influence indices and (b) observed node influence. We randomly generated 500 networks in each of four conditions: 0%, 5%, 10%, and 25% negative edges. Each row depicts the findings from one condition. Each column depicts the findings for a centrality or expected influence index. Each panel depicts a histogram of the correlation coefficient between (a) the centrality and expected influence indices and (b) the observed influence of “treating” the node on the remainder of the network in the randomly generated networks. The strength of the correlation is represented on the x-axis. The frequency with which that correlation was observed is represented on the y-axis.

Aim 2

Centrality and expected influence

The centrality and expected influence indices for the CG lasso network appear in Table 2. Before evaluating our hypotheses, we examined the reliability of centrality indices at Time 1 using Spearman correlation permutation tests (cf. Courrieu, Brand-D'Abrescia, Peereman, Spieler, & Rey, 2011; Telesford et al., 2010). For this analysis, we divided the Time 1 dataset into two equally sized samples composed of independent subjects, calculated network centrality indices separately for each sample, and conducted a Spearman correlation between these indices to test whether network nodes displayed similar centrality position in the two samples. We use a Spearman correlation because the distribution of edges in our lasso network had a considerable positive skew. We then permuted this process 10,000 times to establish a distribution of Spearman values for the Time 1 network centrality indices and examined the mean Spearman. Both EI1, M (SD) = 0.58 (0.18) and EI2, M (SD) = 0.54 (0.17), exhibited moderate agreement. Node strength, M (SD) = 0.48 (0.20) and closeness, M (SD) = 0.42 (0.20) exhibited lower but still moderate agreement. However, betweenness, M (SD) = 0.36 (0.23), exhibited relatively weak agreement and there was considerable variation in Spearman values over the permuted distribution. Because the permutation analysis uses a split-half methodology, the reliability of the centrality indices in our full sample is likely higher than indicated by these results (we did not apply a Spearman-Brown correction). Nonetheless, these results suggest some concern with the reliability of these indices in our lasso network, especially for the betweenness index. The reliability for each index was notably higher in the CG association network, with strength, closeness, and both expected influence indices exhibiting strong agreement (M ≥ 0.71), and betweenness exhibiting moderate agreement, M (SD) = .46 (.22). Additional information about the reliability assessment in our association network appears in Supplementary Materials D.

Table 2.

Centrality and Expected Influence Indices for the Lasso Complicated Grief Network

Strength Closeness Betweenness EI1 EI2
B01 Yearning 0.739 0.008 6 0.739 1.390
B02 Emotional Pain 1.181 0.009 24 1.181 2.071
B03 Thoughts Person 0.707 0.007 6 0.443 0.835
B04 Thoughts Death 0.965 0.009 22 0.965 1.671
C01 Accept 0.852 0.008 12 0.852 1.525
C02 Emotional Numbness 0.974 0.009 14 0.974 1.828
C04 Bitterness 0.641 0.007 0 0.641 1.159
C05 Regret 0.292 0.006 0 0.292 0.550
C06 Avoid 0.478 0.006 2 0.215 0.473
C09 Lonely 0.670 0.007 2 0.670 1.318
C10 Emptiness 1.141 0.009 20 1.141 1.990
C11 Identity 0.651 0.008 2 0.651 1.220
C12 Future 0.645 0.007 0 0.645 1.254

Note. EI1 = one-step expected influence. EI2 = two-step expected influence.

In our lasso network, the centrality and expected influence indices were in broad agreement with one another about which nodes were most important to the network. Centrality indices were strongly correlated with one another, rs > .87, ps <.001. There was an almost perfect correlation between expected influence indices, r > .99, ps <.001. Both EI1 and EI2 were strongly correlated with measures of centrality, rs > .81, ps <.001. Similarly strong correlations among indices were observed in our association network, rs > .64, ps <.018 (see Supplementary Materials D).

Node change and network change

We calculated the correlation between change in the node activation and change in activation of the remainder of the network from Time 1 to Time 2. We used change in the remainder of the network (i.e., overall network activation minus activation of the node of interest) as our outcome variable in order to eliminate any confounding effects of including the target node in the calculation of overall network activation. The complete results for these analyses appear in Supplementary Materials B. Changes in grief-related avoidance, r(193) = .05 [−.09, .19], p = .489, and thoughts about the deceased, r(193) = .13 [−.01, .27], p = .062, were not associated with change in activation of the remainder of the network. For all other nodes, change in node activation was significantly correlated with change in activation of the remainder of the network, with correlations ranging from r(193) =.15 [.01, .28], p = .042, for thoughts about the future, to r(193) = .45 [.32, .55], p < .001 for a belief that life is empty or meaningless. Individuals who reported a high level of change in these nodes (e.g., a substantial reduction in emotional pain) tended to report change in the remainder of the CG network in the same direction (e.g., a reduction in overall CG).

Centrality, expected influence, and the node-network association

We next examined the relationship between (a) the centrality and expected influence of the node and (b) the correlation between change in the activation of that node (Time 2 − Time 1) and change in overall network activation (Time 2 − Time 1). In our lasso network, both EI1, r(11) = .74 [.33, .92], p = .003, and EI2, r(11) = .71 [.26, .91], p = .007, were strongly correlated with the strength of the relationship between node change and network change. The findings for EI1 appear in Figure 2. Each centrality index was also strongly correlated with the strength of the node change-network change association: strength, r(11) = .66 [.18, .89], p = .013; closeness, r(11) = .61 [.09, .87], p = .027; betweenness, r(11) = .62 [.10, .87], p = .024. In our association network, node strength, closeness, EI1, and EI2 were all correlated with the relationship between node change and network change, r(11) = .67 [.19, .89], p = .013. Betweenness was not, r(11) = .35 [−.25, .75], p = .247 (see Supplementary Materials D for complete results).

Figure 2.

Figure 2

Expected influence and the correlation between node change and network change. A scatter-plot depicting the correlation between a node’s one-step expected influence at Time 1 (on the x-axis) and the strength of the association between change in that node and change in the remainder of the network from Time 1 to Time 2 (on the y-axis). Change in high-expected influence nodes was more strongly associated with change in overall network activation than was change in low-expected influence nodes.

Discussion

Researchers adopting the network approach to mental disorders have suggested that high centrality nodes may be especially important to the development, persistence, and remission of mental disorder networks. Our findings provide support for this hypothesis. In simulated single-node interventions in networks with exclusively positive edges, a node’s centrality strongly predicted its observed influence (i.e., the change in the remainder of the network induced by “deactivating” that node). In an empirical network of CG symptoms that contained almost all positive edges, all three centrality indices were strongly correlated with the strength of association between change in node activation and change in the activation of the remainder of the CG network from 6-months to 18-months post-loss.

Our findings suggest that centrality indices perform reasonably well in identifying the most influential nodes in a mental disorder network such as CG. However, we also found evidence that centrality indices may be limited by their failure to account for the presence of negative edges in the network. Most notably, in simulated single-node interventions, centrality indices became less predictive of observed node influence as the proportion of negative edges increased. In contrast, expected influence indices remained strongly correlated with observed node influence in randomly generated networks with negative edges. Although the strength of this correlation did diminish with an increasing proportion of negative edges, both EI1 and EI2 remained strongly correlated with observed influence in networks that had a relatively high proportion of negative edges (25%). In addition, in our empirical network of CG symptoms, both EI1 and EI2 were strongly and significantly correlated with the strength of association between node change and network change.

Implications for Network Analysis

We found that change in high centrality and high expected influence CG nodes was more strongly associated with change in the overall CG network than was change in low centrality or low expected influence CG nodes. Consistent with this finding, Boschloo and colleagues recently reported that high centrality symptoms of depression were more strongly predictive of subsequent onset of MDD than were low centrality symptoms (Boschloo, van Borkulo, Borsboom, & Schoevers, 2016). Together, these findings suggest that network topology can be used to identify the nodes most important to the onset and remission of a mental disorder. However, our simulation findings also suggest that it may be important to account for not only the position of a node in the network, but also the nature of its relationships with the rest of the network. The EI1 and EI2 indices proposed here incorporate both of these pieces of information and, in doing so, may be better able to identify influential nodes in the context of networks that contain negative edges. There are several important areas for future research on centrality and expected influence.

Directed networks

In the current study, we examined centrality and expected influence indices in the context of undirected networks (i.e., networks in which the direction of influence is not specified). However, the full potential of these indices is likely to be realized in the context of directed networks where they can make distinct predictions about the influence of the node on the rest of the network (outgoing expected influence) and the rest of the network’s influence on the node (incoming expected influence). Although methods exist for examining directed networks using cross-sectional data, these methods often assume that the causal structure of the network is invariant across individuals and that the network is acyclic (i.e., contains no feedback loops), assumptions unlikely to hold for mental disorder networks (Cramer, Waldorp, van der Maas, & Borsboom, 2010, p. 185). In future studies, researchers should consider evaluating these indices in directed networks derived from intensive time-series data.

Intra-individual networks and the proportion of negative edges

We observed a low proportion of negative edges in our CG network, raising questions about the practical value added by accounting for negative edges in these networks. Indeed, the proportion of negative edges is typically low in mental disorder networks calculated from cross-sectional data as symptoms tend to be positively correlated with one another, especially in syndromes that do not incorporate opposing symptoms, such as hypersomnia versus insomnia in major depressive disorder. Our findings suggest that centrality indices will perform well in the context of such networks. We suspect that negative edges will be more prevalent in intra-individual networks, especially in non-clinical populations. For example, one bereaved individual may consistently respond to thoughts about the deceased with increases in emotional pain (i.e., a positive edge) whereas another may consistently respond to such thoughts with decreases in emotional pain (i.e., a negative edge). Centrality indices would treat these edges equivalently, accounting only for the presence and absolute strength of the relationship. In contrast, expected influence indices would make very different predictions about the influence of thoughts about the deceased on the remainder of the network, its importance to the development or persistence of CG, and its suitability as a target of intervention in these two individuals. Accordingly, in future studies, it may be especially informative for researchers to examine the performance of these indices in intra-individual networks in clinical and non-clinical populations.

Single-node interventions

In this study, we simulated single-node interventions in order to examine observed node influence. To more rigorously test our predictions regarding the association between expected influence and observed influence, researchers should administer single-node interventions on both high- and low-expected influence nodes, treating the individual node and examining the impact on the remainder of the network. In doing so, researchers would more rigorously test the hypothesis that change in nodes with high expected influence is causally related to change in the broader network while also clarifying the role played by these nodes in the network.

Implications for CG and CG Treatment

Our findings in the CG network suggest that reduced activation of nodes with high expected influence bodes well for reduction in overall CG. Accordingly, intervening on these nodes may produce greater reductions in overall network activation than intervening on nodes with lower expected influence. Emotional pain and a feeling that life is empty or without meaning were the nodes with the highest expected influence and were among the most strongly tied to change in overall network activation. In regard to the former, reduction in emotional pain through imaginal and in vivo exposure exercises is an important component of cognitive-behavioral treatments for CG (e.g., Bryant et al., 2014; Shear et al., 2005). In regard to the latter, this strong association is consistent with the emphasis placed on restoration of meaning in theories of natural grief resolution (Stroebe & Schut, 1999) and in prominent psychotherapies for CG (e.g., Neimeyer, 2000; Shear, Frank, Houck, & Reynolds, 2005). Accordingly, our findings suggest clinical researchers have appropriately targeted nodes that are especially important to the CG network.

Although our findings are consistent with the possibility that high expected influence nodes may be important targets of psychotherapy, it should be noted that these analyses were correlational and do not indicate a causal direction in the node-network relationship. Similarly, because our centrality and expected influence indices were undirected, it is unclear whether nodes were high on these indices because they exert greater influence on other nodes of the network or if they were especially susceptible to the influence of other nodes in the network. If a node’s importance is mainly attributable to its being the recipient, rather than the source, of activation, then targeting this node may have little impact on the activation level of the network. If a node is high in both in- and out-expected influence, modifying it may be both beneficial (due its effect on other nodes) and difficult (due the influence of other active nodes in the network). In this case, treatments may benefit by reducing activation of low-expected influence nodes before targeting high-expected influence nodes.

There are additional caveats to the conclusion that high expected influence nodes should be the primary targets for psychotherapy. First, nodes may vary in the extent to which they are amenable to change in psychotherapy for CG. For example, a patient may be better able to change a behavior (e.g., grief-related avoidance) than an emotion (e.g., emotional pain). Indeed, prominent psychotherapies for CG (e.g., Bryant et al., 2014; Shear et al., 2005) do not attempt to reduce distressing emotions directly, but rather do so indirectly, by modifying cognitions (e.g., self-blame) and behaviors (e.g., grief-related avoidance). The focus of these interventions suggests that cognitive and behavioral nodes of the CG network may be more amenable to direct intervention in psychotherapy than are emotional nodes. Finally, it may be that the most effective interventions do not target nodes at all, but rather target edges. For example, the aim of a network-informed CG intervention may not be to decrease thoughts about the deceased, but rather to modify the edge between thoughts about the deceased and intense yearning. In other words, to “decouple” these nodes (Levin, Luoma, & Haeger, 2015). These findings suggest the need to identify not only influential nodes, but also influential edges whose modification might facilitate reductions in the overall network.

Taken together, these considerations imply that the treatment implications of our findings are more complex than merely prescribing intervention on high-centrality or high-expected influence nodes, such as emotional pain or the feeling that life is empty or without meaning. Researchers adopting a network perspective should consider not only the potential impact of changing a node, but also the node’s amenability to direct intervention, the possibility that initial interventions on low centrality nodes may facilitate subsequent interventions on high centrality nodes, and the possibility that edges may be more appropriate targets of intervention than nodes. Nonetheless, our findings suggest that a node’s centrality and expected influence are important pieces of information to consider when evaluating the most effective way of intervening on the CG network and that reduction in such nodes augurs well for the prognosis of CG.

Limitations

The network simulations performed in our Aim 1 analyses did not account for external influences on nodes or self-loops (i.e., the node’s influence on itself). Consequently, our simulations almost certainly inflated the strength of the correlation between (a) centrality and expected influence indices and (b) observed node influence. Although this should not have affected our comparison of expected influence and centrality indices, it nonetheless illustrates the rudimentary nature of the simulation data presented here and the need to further improve our ability to simulate mental disorder networks in order to better evaluate hypotheses such as those examined here. To do so, it will be critical to advance our understanding of how CG and other mental disorder networks unfold over time within individuals so that network simulations can be informed by and effectively model the processes operating within these networks.

In our Aim 2 analyses, we used items from the CLOC study to assess nodes of CG network guided by the DSM-5 diagnostic criteria for this syndrome. However, the correspondence between the CLOC survey item and the DSM-5 diagnostic criterion was not always precise and we were unable to find a CLOC survey item for three of the DSM-5 diagnostic criteria. Moreover, because these criteria were not developed with the aim of providing a comprehensive list of CG network nodes, our use of the DSM-5 diagnostic criteria to define our set of network nodes may have led us to omit nodes that play a critical role in the CG network. This possibility is especially noteworthy here as the omission of such nodes would affect our assessment of node centrality. For example, we were unable to include grief-related approach behavior (i.e., proximity seeking) in our analyses. Given the plausible causal association between yearning for the deceased and grief-related approach behavior, this omission may have caused us to underestimate the centrality of yearning, a node that many grief researchers believe to be the core node of the CG syndrome (e.g., Prigerson et al., 2009). In future research on CG it will be critical for researchers to move beyond the DSM-5 and other proposed diagnostic criteria sets to include other nodes potentially operative within the CG network.

In addition, we examined the CG network in a sample of bereaved adults, most of whom would not have met criteria for the diagnosis of CG. Accordingly, our analyses may have failed to adequately assess the network structure as it exists in those who experience persistent elevated activation of the CG network. For example, in our analyses grief-related avoidance exhibited very low centrality and change in avoidance was not correlated with change in the rest of the CG network. This finding stands in stark contrast to prominent theories regarding the etiology of CG that emphasize the role of grief-related avoidance in maintaining the syndrome (Boelen, van den Hout, & van den Bout, 2006; Shear et al., 2007). It may be that the effect of avoidance is context dependent (e.g., maladaptive only in the context of elevated emotional pain) and that avoidance plays a more prominent role in CG network for those who develop the disorder than it does among the general population of bereaved adults. Accordingly, researchers should examine the CG network in those with persistently elevated activation of the network. The need for research on the CG network in other samples is especially noteworthy given that, to date, the CG network has been studied exclusively with the CLOC dataset.

In our lasso network, centrality indices exhibited poor to moderate reliability. Accordingly, some caution is warranted when interpreting these results. This limitation is mitigated by our finding that the correlation between centrality indices and the strength of the relationship between node change and network change was also observed in an association network where these indices exhibited greater reliability (see Supplementary Materials D). Reliability was especially low for our betweenness index and remained relatively low even when we examined betweenness in the association network. We consider this to be a limitation of the betweenness index in small highly inter-connected networks, such as those typically observed when examining mental disorder networks, and not a limitation specific to the current CG network. In future studies on mental disorder networks, researchers should consider the reliability of these indices when choosing a measure of node importance. Notably, the relevance of a centrality metric can vary as a function of the substantive phenomenon represented by the network (Freeman, 1978/1979). For example, closeness centrality may be especially relevant for modeling information transmission in a social network, whereas strength centrality may be most relevant for psychopathology networks. Accordingly, researchers should also consider conceptual appropriateness of these indices when choosing a measure of node importance.

Some might question whether the relationship between a node’s centrality or expected influence and its correlation with the strength of association between node change and network change is tautological. We believe this is not the case. First, the departure from perfect predictability (i.e., a correlation less than 1.0) confirms that these are not merely two different ways of saying the same thing. Second, our Aim 2 analyses concerned change in node and network activation over time and nothing in our calculation of centrality precluded the possibility of it being completely unrelated to change in network activation over time. Together, these points indicate that our findings are not a tautological consequence of our definitions of centrality and expected influence.

It is important to note that although our predictions were derived from the network approach to mental disorders, these results should not be considered evidence in support of the network approach. There are alternative explanations for these findings. For example, our calculation of node strength and expected influence is similar to the calculation of an item-rest correlation (i.e., the correlation between an item and a scale score of the remaining items from that scale; also called the “corrected item-total” correlation). Given that more reliable items may have greater predictive validity, the current findings could be seen as resulting from differences in the reliability of the items rather than their centrality. Similarly, operating from a latent construct approach, nodes that we have identified as being “highly central” may alternatively be considered to be especially good indicators of the underlying construct. From this perspective, it would not be surprising that change in these “good indicator” items is more strongly associated with change in the latent construct. We believe that the network approach provides a plausible explanation for the tendency of these nodes to hang together as a syndrome that does not suffer from limitations inherent in latent construct approach (Borsboom & Cramer, 2013) and we consider the analyses performed in this paper to be a more natural extension of the network approach than it is of alternative conceptual frameworks. Nonetheless, it remains for future research to further adjudicate between these alternative interpretations of these findings.

Finally, our study was limited by our reliance on cross-sectional analyses to calculate our centrality and expected influence indices. Findings based on inter-individual variation only correspond to findings based on intra-individual variation under very specific conditions that are rarely met in psychological research (i.e., the assumption of ergodicity rarely holds; Molenaar & Campbell, 2009). Consequently, our findings cannot be assumed to provide information about the processes operative within any specific bereaved individual. To draw conclusions about the CG network as it occurs at the level of the individual, researchers must use intra-individual network analyses to assess the CG network as it unfolds over time.

Conclusion

The network approach conceptualizes CG not as underlying latent disease entity, but rather as a causal system of mutually reinforcing nodes. In a previous study, we suggested that highly central nodes may figure prominently in the etiology and treatment of CG. Consistent with this prediction, we found that change in high centrality nodes of the CG network (e.g., feelings of emptiness and emotional pain) was more strongly associated with change in the remainder of the CG network than was change in low-centrality nodes (e.g., grief-related regret or avoidance). However, in simulated network data, we found that centrality indices may be limited by their failure to distinguish between positive and negative edges. In contrast, indices of a node’s expected influence remained strongly correlated with observed node influence in our simulated networks, even when those networks contained a relatively high proportion of negative edges. In addition, expected influence indices were strongly and significantly correlated with the strength of the relationship between node change and network change in our analysis of the CG network. Together, these findings suggest that expected influence indices can be used to identify highly influential nodes in the CG network that may play a prominent role in the etiology and treatment of CG.

Supplementary Material

S1

Acknowledgments

The Changing Lives of Older Couples study was supported by grants from the Nancy Pritzker Research Network and the National Institute of Aging (AG15948-01; AG610757-01; AG05561-01). We thank Sacha Epskamp, Eiko Fried, Susanne Hoeppner, Robin Mankel, and two anonymous reviewers for their helpful feedback on earlier drafts of this article.

References

  1. American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th. Arlington, VA: American Psychiatric Association; 2013. [Google Scholar]
  2. Boelen PA, van den Hout MA, van den Bout J. A cognitive-behavioral conceptualization of complicated grief. Clinical Psychology: Science and Practice. 2006;13:109–128. [Google Scholar]
  3. Borgatti SP. Centrality and network flow. Social Networks. 2005;27:55–71. [Google Scholar]
  4. Borsboom D, Cramer AO. Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology. 2013;9:91–121. doi: 10.1146/annurev-clinpsy-050212-185608. [DOI] [PubMed] [Google Scholar]
  5. Bramson A, Vandermarliere B. Benchmarking measures of network influence. arXiv preprint arXiv:1602.05744. 2016 doi: 10.1038/srep34052. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Bryant RA, Kenny L, Joscelyne A, Rawson N, Maccallum F, Cahill C, Nickerson A. Treating prolonged grief disorder: a randomized clinical trial. JAMA Psychiatry. 2014;71:1332–1339. doi: 10.1001/jamapsychiatry.2014.1600. [DOI] [PubMed] [Google Scholar]
  7. Carr DS, Nesse RM, Wortman CB. Spousal bereavement in late life. New York: Springer Pub. Co; 2006. [Google Scholar]
  8. Courrieu P, Brand-D'Abrescia M, Peereman R, Spieler D, Rey A. Validated intraclass correlation statistics to test item performance models. Behavior Research Methods. 2011;43:37–55. doi: 10.3758/s13428-010-0020-5. [DOI] [PubMed] [Google Scholar]
  9. Cramer AO, Borsboom D. Problems attract problems: A network perspective on mental disorders. In: Scott R, Kosslyn S, editors. Emerging Trends in the Social and Behavioral Sciences. John Wiley & Sons, Inc; 2015. pp. 1–15. [Google Scholar]
  10. Cramer AO, Waldorp LJ, van der Maas HL, Borsboom D. Comorbidity: A network perspective. Behavioral and Brain Sciences. 2010;33:137–150. doi: 10.1017/S0140525X09991567. [DOI] [PubMed] [Google Scholar]
  11. Csardi G, T N. The igraph software package for complex network research. Inter Journal, Complex Systems. 2006;1695:1–9. [Google Scholar]
  12. Epskamp S, Cramer AO, Waldorp LJ, Schmittmann VD, Borsboom D. Qgraph: Network visualizations of relationships in psychometric data. Journal of Statistical Software. 2012;48:1–18. [Google Scholar]
  13. Freeman LC. Centrality in social networks conceptual clarification. Social Networks. 1978/1979;1:215–239. [Google Scholar]
  14. Fried EI, Bockting C, Arjadi R, Borsboom D, Amshoff M, Cramer AO, Stroebe M. From loss to loneliness: The relationship between bereavement and depressive symptoms. Journal of Abnormal Psychology. 2015;124:256–265. doi: 10.1037/abn0000028. [DOI] [PubMed] [Google Scholar]
  15. Fried EI, Epskamp S, Nesse RM, Tuerlinckx F, Borsboom D. What are 'good' depression symptoms? Comparing the centrality of DSM and non-DSM symptoms of depression in a network analysis. Journal of Affective Disorders. 2016;189:314–320. doi: 10.1016/j.jad.2015.09.005. [DOI] [PubMed] [Google Scholar]
  16. Friedman J, Hastie T, Tibshirani R. Sparse inverse covariance estimation with the graphical lasso. Biostatistics. 2008;9:432–441. doi: 10.1093/biostatistics/kxm045. [DOI] [PMC free article] [PubMed] [Google Scholar]
  17. Levin ME, Luoma JB, Haeger JA. Decoupling as a mechanism of change in mindfulness and acceptance: A literature review. Behavior Modification. 2015 doi: 10.1177/0145445515603707. [DOI] [PubMed] [Google Scholar]
  18. McKinney W. Data structures for statistical computing in python; Proceedings of the 9th Python in Science Conference; 2010. pp. 51–56. [Google Scholar]
  19. McNally RJ, Robinaugh DJ, Wu GWY, Wang L, Deserno MK, Borsboom D. Mental disorders as causal systems: A network approach to posttraumatic stress disorder. Clinical Psychological Science. 2015;3:836–849. [Google Scholar]
  20. Millman KJ, Aivazis M. Python for Scientists and Engineers. Computing in Science & Engineering. 2011;13:9–12. [Google Scholar]
  21. Molenaar PCM, Campbell CG. The new person-specific paradigm in psychology. Current Directions in Psychological Science. 2009;18:112–117. [Google Scholar]
  22. Neimeyer RA. Searching for the meaning of meaning: Grief therapy and the process of reconstruction. Death Studies. 2000;24:541–558. doi: 10.1080/07481180050121480. [DOI] [PubMed] [Google Scholar]
  23. Prigerson HG, Horowitz MJ, Jacobs SC, Parkes CM, Aslan M, Goodkin K, Maciejewski PK. Prolonged grief disorder: Psychometric validation of criteria proposed for DSM-V and ICD-11. PLoS Medicine. 2009;6:e1000121. doi: 10.1371/journal.pmed.1000121. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Robinaugh DJ, LeBlanc NJ, Vuletich HA, McNally RJ. Network analysis of persistent complex bereavement disorder in conjugally bereaved adults. Journal of Abnormal Psychology. 2014;123:510–522. doi: 10.1037/abn0000002. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Schmittmann VD, Jahfari S, Borsboom D, Savi AO, Waldorp LJ. Making Large-Scale Networks from fMRI Data. PLoS One. 2015;10:e0129074. doi: 10.1371/journal.pone.0129074. [DOI] [PMC free article] [PubMed] [Google Scholar]
  26. Shear K, Frank E, Houck PR, Reynolds CF., 3rd Treatment of complicated grief: a randomized controlled trial. Journal of the American Medical Association. 2005;293:2601–2608. doi: 10.1001/jama.293.21.2601. [DOI] [PMC free article] [PubMed] [Google Scholar]
  27. Shear K, Monk T, Houck P, Melhem N, Frank E, Reynolds C, Sillowash R. An attachment-based model of complicated grief including the role of avoidance. European Archives of Psychiatry and Clinical Neuroscience. 2007;257:453–461. doi: 10.1007/s00406-007-0745-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
  28. Stroebe M, Schut H. The dual process model of coping with bereavement: Rationale and description. Death Studies. 1999;23:197–224. doi: 10.1080/074811899201046. [DOI] [PubMed] [Google Scholar]
  29. Telesford QK, Morgan AR, Hayasaka S, Simpson SL, Barret W, Kraft RA, Laurienti PJ. Reproducibility of graph metrics in FMRI networks. Frontiers in Neuroinformatics. 2010;4:117. doi: 10.3389/fninf.2010.00117. [DOI] [PMC free article] [PubMed] [Google Scholar]
  30. van der Walt S, Colbert SC, Varoquaux G. The NumPy Array: A structure for efficient numerical computation, Computing in Science & Engineering. 2011;13:22–30. [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

S1

RESOURCES