Abstract
The death of a loved one can trigger a range of emotional, behavioural, motivational and cognitive reactions collectively labelled as “grief”. There has been a recent resurgence of empirical interest in grief reactions, and there is now a general consensus that 7%–10% of bereaved individuals experience intense and chronic reactions termed Prolonged Grief (PG). However, there is ongoing debate about how best to conceptualise the construct. Recent studies have advanced this debate by applying a network, or causal systems approach. Whereas traditional views of psychopathology posit symptoms of disorders reflect the independent actions of latent entities, the network approach argues that the symptoms themselves interact to give rise to the disorder. A network conceptualisation offers novel approaches to studying the mechanisms that contribute to PG. To date, however, research has focussed only on spousal loss and only used a single archival data set. Therefore, in this paper we apply network analysis to examine relationships among PG symptoms in samples of individuals bereaved by loss of a spouse (Study 1, N=193) and a parent (Study 2, N=180). Participants completed the PG-13 and a measure of depression. A comparison test suggested the networks produced from each sample were not reliably different. The strongest link in both networks was between yearning and emotional pain. Meaninglessness was relatively central, whereas avoidance was peripheral in both networks. Findings are discussed with reference to theoretical models and the potential benefits a network approach may hold for understanding relationships between symptoms of PG.
Keywords: Grief, bereavement, network modelling
The death of a love one typically leads to a range of behavioral reactions, disturbing thoughts, and painful emotions that together constitute the experience of grief. Grief reactions may be characterized by yearning for the deceased and/or distress at the lost relationship, difficulty accepting the death, a sense of shock, avoidance of reminders, a sense that life lacks meaning without the deceased, self-identity confusion, emotional numbness, bitterness, loss of trust, and a difficulty re-engaging with life. There is variation in the frequency, duration and intensity with which individuals experience grief reactions. Approximately 7–10% of individuals will experience an extreme and prolonged reaction, termed Prolonged Grief (PG; or complicated grief or Persistent Complex Bereavement Disorder; American Psychiatric Association, 2013; Maciejewski, Maercker, Boelen, & Prigerson, 2016; Nielsen et al., 2016; Prigerson et al., 2009; Shear et al., 2011). PG has received increased attention in recent years as it is linked with a range of negative health outcomes independent of bereavement-related depression or anxiety (Bonanno et al., 2007; Maciejewski et al., 2016; Prigerson et al., 2009; Shear et al., 2011) and shows better outcomes in response to grief-targeted treatments (Bryant et al., 2014; Shear, Frank, Houck, & Reynolds, 2005; Shear et al., 2016).
Despite the weight of evidence regarding the harms associated with PG, there has been significant debate about the construct (see Bryant, 2014 for discussion). In part this stems from the traditional view of psychopathology, which assumes that symptoms of a disorder co-occur as a result of the expression of an unobserved disease process or latent entity. This view typically postulates that mental health disorders represent different categories of functioning from healthy functioning. More recently, Borsboom and colleagues (e.g., Borsboom & Cramer, 2013; Boschloo et al., 2015; Fried et al., 2016; Hofmann, Curtiss, & McNally, 2016; McNally, Robinaugh, Wu, Wang, & Deserno, 2015) have offered a fundamentally different approach to understanding psychopathology based on complex network relationships or causal systems. A network consists of two components: Nodes which represent the individual items included in the analysis, and edges which represent the relationships between the nodes. In contrast to the “latent entity” model of psychopathology, the network approach proposes that symptoms comprising mental health disorders arise and cluster together as a result of direct causal relationships among the symptoms themselves (Borsboom & Cramer, 2013). That is, “depression” is not an entity that causes insomnia, fatigue and concentration problems. Rather, these symptoms give rise to and subsequently interact to reinforce each other: Insomnia causes fatigue, fatigue causes concentration problems, concentration problems trigger worry and rumination, worry and rumination fuel insomnia etc. From this perspective, disorders develop when such functionally interrelated symptoms settle into a pathological equilibrium and become mutually reinforcing, not because of the action of any latent entity (Borsboom & Cramer, 2013; McNally, 2016; McNally et al., 2015).
The network approach to modeling psychopathology has several potential advantages for understanding grief-related syndromes. A particular advantage lies in its ability to map the clear causal relationships observed between symptoms in many mental health disorders (Hofmann et al., 2016; McNally, 2016). For example, Fried et al. (2015) compared the ability of a network approach and a traditional latent modeling approach to explain the pattern of depressive symptoms amongst spousally-bereaved and non-bereaved older adults. Compared to the non-bereaved group, bereaved individuals scored significantly higher on items assessing loneliness, sadness, depressed mood, and appetite loss, and significantly lower on items assessing happiness and enjoyment of life. Importantly, Fried et al. (2015) found that modeling the direct relationships between spousal loss and symptoms using network analysis provided a better fit of the data than modelling indirect effects or depression as a latent diagnostic entity. Network modeling also showed that bereavement primarily impacted the symptom of loneliness, which in turn activated the other depressive symptoms.
A key implication of network modeling is that by identifying the most influential symptoms, or nodes, within the network it should be possible to target these nodes, disrupt the equilibrium, and so reduce (or increase) overall levels of psychopathology. In this way, the network approach is highly compatible with cognitive-behavioral models of psychopathology and treatment which seek to target the key mechanism thought to underlie symptom maintenance (e.g., Hawton, Salkovskis, Kirk, & Clark, 1989; Hofmann, Asmundson, & Beck, 2013; Hofmann et al., 2016; Leahy, 2006; Mennin, Ellard, & Fresco, 2013). Robinaugh, LeBlanc, Vuletich, and McNally (2014) also used network modelling to investigate bereavement-related psychopathology. Rather than depression, they examined a set of grief-specific items selected to match symptoms included in the DSM-5 Persistent Complex Bereavement Disorder (American Psychiatric Association, 2013). Robinaugh, et al’s. (2014) analysis provided additional support for the integrity PG, finding that the symptoms of PG clustered separately from symptoms of depression. Also, consistent with Fried et al. (2015), Robinaugh et al. (2014) identified loneliness as a potentially important bridge symptom between PG and depression within this sample. Further, the analysis shed light on the possible differential importance of individual PG symptoms: in this analysis emotional pain emerged as the most central (or connected) node within the network, whereas avoidance was found to be relatively peripheral. This suggests that when emotional pain was present many other nodes within the network were also activated, however, when avoidance was present other nodes may or may not have been activated to any great extent.
Although the findings from Fried et al. (2015) and Robinaugh et al. (2014) highlight the potential insights that may be gained from applying the network approach to understanding bereavement-related psychopathology, both these analyses were based on data from the Changing Lives of Older Couples (CLOC) study. This study was conducted in Michigan during the 1990’s and comprised men aged 65 or over and women whose husbands were 65 years or older. The extent to which these findings are generalizable to other bereaved populations has yet to be explored. It is possible that the networks derived from this data reflect loss-type or age specific relationships; for example, loneliness may be more prominent among older adults who lose a spouse. Also, the CLOC study was conducted before systematic study was undertaken to identify key grief-specific symptoms predictive of poor outcomes, and so analyses did not include all symptoms proposed for inclusion in a PG diagnosis. Given the relative novelty of the network approach, there is an imperative to explore the extent to which conclusions drawn from these initial analyses, conducted using the same data set, may extend to other samples of bereaved individuals. Accordingly, in the current investigation we applied the network approach to analyze data from two different samples of bereaved individuals. Our first aim was to explore the extent to which patterns observed in Robinaugh et al. (2014) and Fried et al. (2015) generalized to other bereaved samples. We were also interested in examining the extent to which any conclusions drawn from the produced networks represented reliable associations. To this ends we applied recently developed methods of exploring network stability and reliability (Epskamp, Borsboom, & Fried, 2016). Data were taken from two ongoing bereavement-related projects being conducted at the Loss Trauma and Emotion Lab, Teachers College, Columbia University. In the first study, we examined networks of PG and depression in a sample of spousally bereaved individuals who were all under 65 years of age. In the second study we examined the same networks in a sample of younger individuals who had experienced the death of a parent.
Study 1: Loss of a Spouse
Method
Participants
We analyzed data from 193 individuals (66.5% female, 33.5% male) from the New York metropolitan area who had lost a spouse. The mean age of participants was 57.04 (SD = 6.58). The majority of the sample identified as Caucasian (92%; African American 4.8%; Asian 1.6%; Other 1.6%), and had a college degree or above (High School only 9.4%; Some College 23.8%; Bachelor’s degree 35.4%; Graduate degree or above 31.4%). Most individuals were working full time (61%; Part-time 12%; Unemployed or Retired 27%), with an expected median yearly family income of USD $90′000 (IQR: 51′500–150′000). Participants had volunteered to participate in a study investigating bereavement outcomes. Mean time since death was 14.00 months (SD = 0.88). The study was approved by Teachers College Institutional Review Board (14-146: Project to Understand Reactions to Loss).
Measures
Prolonged Grief-13 (PG-13) (Prigerson et al., 2009)
The PG-13 is a self-report measure that indexes symptoms of PG. The measure assesses for yearning, emotional distress at the lost relationship, difficulty accepting the death, shock, avoidance of reminders, numbness, bitterness, difficulty engaging in life, identity disturbance, and a sense of meaninglessness on 5-point scales (not at all – several times a day/overwhelmingly, respectively) and for the presence of functional impairment (yes/no). Scores on the PG-13 are predictive of greater impairment and poorer quality of life 12–24 months post-bereavement (Maciejewski et al., 2016; Prigerson et al., 2009). The functional impairment item is not a discrete symptom; instead, it indexes the combined impact of all other items on functioning. For this reason, we did not include this as an item in the network. Cronbach alpha for the scale in this sample was α = .92. The inclusion/exclusion of functional impairment did not change scale reliability.
Center for Epidemiologic Studies-Depression Scale (CES-D; Radloff, 1977)
Depressive symptoms were measured using 9-item version of the self-report CES-D (Kohout, Berkman, Evans, & Cornoni-Huntley, 1993). Items are scored on a 3-point scale. Two items were excluded as they indexed interpersonal problems rather than core symptoms of depression (see also Robinaugh et al., 2014). Cronbach alpha for the 7 and 9 item scale in this sample was α = .85 and α = .82, respectively.
Statistical Approach
A series of networks of regularized partial correlation coefficients were computed using the glasso procedure from the R package qgraph (Epskamp, Cramer, Waldorp, Schmittmann, & Borsboom, 2012; Epskamp & Fried, 2016). In a regularized partial correlation network each edge represents the partial correlation between nodes after controlling for all other variables in the dataset. In addition, the procedure uses a LASSO correction to shrink very small connections to be exactly zero, producing a more parsimonious network (Epskamp & Fried, 2016). This approach avoids spurious connections between nodes based on indirect relationships with a third node (Epskamp & Fried, 2016; Friedman, Hastie, & Tibshirani, 2008). When the partial correlation is zero, no edge is drawn between the nodes, indicating that the two variables are independent of one another after controlling for relationships between other variables in the network.
Measures of node centrality
To assess the importance of each node within the network, we calculated three indices of node centrality: Strength, closeness and betweenness. Strength equals the sum of the weights of the edges attached to that node. Closeness is the inverse of the sum of the distances of the node from all other nodes in the network. A closeness-central node is one that is likely to be quickly affected by changes in other nodes directly or indirectly. Betweenness is defined as the number of times in which a given node lies on the shortest path between two other nodes.
Network stability
The extent to which estimated network structures are susceptible to variations in sampling error and estimation methods has yet to be established. As an exploratory measure of the stability of the network produced in this analysis we used the R package bootnet (Epskamp et al., 2016). This program uses bootstrapping procedures to compute 95% confidence intervals around each edge of the network and estimates of node centrality stability. To quantify node centrality stability, bootnet calculates a correlational estimate, the centrality stability coefficient (CS-coefficient). The CS-coefficient represents the proportion of participants that can be dropped from the analysis while maintaining a very large correlation with the original centrality indices. The bootnet program also provides significance tests for differences in edge strength and differences in strength centrality.
Comorbidity and node communities
To explore the relationship between symptoms of PG and depression we repeated lasso regularized partial correlation network estimation including PG and depressive symptoms (18 nodes in total). We examined the community structure of these networks using the spin glass algorithm (Reichardt & Bornholdt, 2006) with gamma= 1, start temperature = 1, stop temperature = .01, cooling factor = .99, spins = 19). The analysis was conducted with the R package igraph (Csardi & Nepusz, 2006).
Results
Mean scores for each of the items on the PG 13 are presented in Table 1. Almost all participants in this sample reported some degree of yearning for the deceased (98%) and emotional pain at the lost relationship (95%) in the last month. The least frequently endorsed items were difficulties with trust (30%) and avoidance of reminders (40%). The mean number of items endorsed as being experienced at least once in the past month (or slightly) was 6.74 (SD = 3.02), the modal number was 9. The mean sum score on this measure, an indication of network activation, was 24.44 (SD=9.11). Forty-nine (24%) participants scored in a range suggestive of possible diagnosis of PG (Prigerson et al., 2009).
Table 1.
PG Item | Spousal loss Mean (SD) |
Parental Loss Mean (SD) |
---|---|---|
Yearning | 3.62 (1.11) | 2.94 (1.21) |
Emotional Pain | 3.04 (3.62) | 2.54 (1.21) |
Avoidance | 1.67 (1.01) | 1.72 (1.08) |
Shocked and stunned | 2.10 (1.18) | 1.80 (1.08) |
Role confusion | 2.21 (1.29) | 2.08 (1.25) |
Difficulty accepting loss | 2.17 (1.18) | 2.24 (1.22) |
Difficulty trusting | 1.53 (0.94) | 1.81 (1.16) |
Bitterness and anger | 2.06 (1.10) | 2.06 (1.10) |
Difficulty re-engaging | 2.02 (1.13) | 2.02 (1.22) |
Numbness | 2.04 (1.16) | 2.08 (1.21) |
Sense of meaningless | 1.99 (1.21) | 1.91 (1.16) |
Relationship among PG symptoms
The regularized partial correlation network is presented in Figure 1a. The strength, closeness and betweenness indices of centrality are presented in Figure 2a. The four items showing the highest centrality were Yearning, Emotional Numbness, Meaninglessness and Difficulty Accepting the Loss. As can also be seen in Figure’s 1a and 2a, Avoidance appeared relatively unconnected to the rest of the network; the strongest connections involving Avoidance were between Avoidance and Role Confusion and Avoidance and Difficulty Accepting the Loss.
Network stability
The full results from the network stability analysis are presented in the supplementary material (Figure S1–4). Significance tests of edge differences indicated that the two thickest edges (between Yearning and Emotional Pain and between Meaninglessness and Role Confusion) were significantly stronger than most other edges in the network; many of the remaining edges were shown not to be reliably different from one another (see Figure S2). This suggests caution should be taken in drawing conclusions about the differential strength of relationships for all but the strongest edges. Consistent with previous findings, strength centrality was shown to be the most robust index of centrality. Significance test for differences in node strength centrality indicated that Yearning, Numbness and Meaninglessness were significantly stronger than Avoidance, but did not reveal reliable differences between the strength of other nodes. The CS-coefficients for Strength, Closeness, and Betweenness were 0.33, 0.18 and 0.07, respectively.
Comorbidity
The LASSO association network including items assessing PG and depression is presented in Figure 3a. The spin glass algorithm detected five communities of nodes in this network. No community contained a combination of grief and depression items. In other words, grief and depression symptoms formed distinct network structures. One community comprised Yearning, Emotional Pain and Shock/Stunned, a second contained Role Confusion, Difficulties Re-Engaging with Life, Numbness and Meaninglessness, the third cluster contained Avoidance, Difficulty Accepting the Loss, Bitterness and Trust Difficulties. A fourth community contained Sadness, Loneliness and Sleeping Difficulties, and the final community comprised the remainder of the depression symptoms, Appetite Changes, Depression, Effort, and Difficulties “Getting-Going”. Bootstrapped confidence intervals were calculated for this larger network and are presented in the supplementary material (S5).
Discussion
This study examined relationships between individual symptoms of PG and depression in a sample of spousally-bereaved individuals using a network analysis framework. Supporting the construct of PG, we found that the nodes comprising PG clustered separately from those comprising depression (see also Robinaugh et al., 2014). Yearning, Emotional Numbness, Meaninglessness and Difficulties Accepting the Loss appeared to be the most central nodes within the PG network. Avoidance showed the lowest centrality across the three indices of network centrality. The CS coefficients indicated that differences in centrality on the betweenness and closeness indices fell below the .25 guideline proposed by Epskamp et al. (2016) for interpretability. By the same guidelines, the CS-coefficient for strength centrality was more trustworthy (Epskamp, et al 2016). This is consistent with studies showing that node strength is the most stable measure of centrality (Epskamp, et al 2016). Accordingly, we focus our discussion of centrality on those findings related to strength.
In contrast to the findings of Robinaugh et al.’s (2014), Emotional Pain did not show dominant centrality in the network. Only Yearning, Emotional Numbness, and Meaninglessness were found to have significantly greater strength than the lowest node, Avoidance; Emotional Pain was, however, strongly linked with Yearning. The edges between Yearning and Emotional Pain, and between Meaninglessness and Role Confusion were significantly stronger than most other edges in the network. The networks in this study were based on regularized partial correlations constructed using a LASSO penalty, a recent analytic development likely to result in a network with fewer and less spurious connections (Constantini et al., 2015; Epskamp & Fried, 2016). When we re-ran the analysis using the partial-correlations procedure used in Robinaugh et al. (2014), the centrality of Emotion Pain increased somewhat, but remained less than that of Numbness and Meaninglessness. In terms of the relationships between nodes representing PG and depression, Sadness appeared as the strongest “gateway” or linking node. Yearning and Meaninglessness were the grief items most strongly connected to Sadness. Interestingly, in this analysis Loneliness clustered with the depression nodes and did not act as a clear link between PG and depression. However, there was a strong edge from Loneliness to Sadness.
Both previous studies that have applied network analysis to bereavement related symptomatology (Fried et al., 2015; Robinaugh et al., 2014) have used data from the CLOC study, a large population survey of older adults conducted in the 1990’s. Empirical understanding of the phenomenology of grief reactions has increased significantly since this time and not all phenomena subsequently identified as predictive of poor bereavement outcomes are included in the CLOC dataset (e.g., difficulties trusting others since the loss). Further, the DSM 5 criteria for PCBD on which Robinaugh et al. (2014) was based combines items that are separated on the PG-13 (Shock/Stunned and Emotional Numbness), and includes items not assessed by the PG-13 (Preoccupation, Regret). Recent findings suggest that differences between PG and PCBD are largely semantic in terms of predictive validity (Maciejewski et al., 2016), however, it is yet to be established the extent to which inclusion or exclusion of items within a network may change the nature of the other observed relationships. In this study, Difficulties with Trust showed moderate strength compared with other nodes. Perhaps not surprisingly, Difficulties with Trust’s strongest relationship was with Bitterness. Interestingly, however the two nodes Numbness and Shock/Stunned were connected, but evidenced different relationships with other nodes, raising the possibility that they may function differently within the network. Numbness also appeared to have greater strength centrality than Shock/Stunned.
In summary, while this analysis showed several similarities with previous network analyses of bereavement symptomatology, there were also several differences. Before drawing further conclusions, however, we sought to explore the consistency of these observed relationships by repeating our analysis in a sample of individuals who had experienced the death of a parent.
Study 2: Loss of a Parent
Method
Participants
This sample comprised 180 participants who completed the PG-13 online while participating in a study on decision making. Participants were recruited via Mechanical Turk using procedures outlined in (Maccallum & Bonanno, 2016)1. 1800 individuals first participated in a “Life Events Survey” in which they were asked to indicate whether they had experienced a number of positive and negative life events. Individuals who indicated they had experienced the death of a loved on within the last 1–3 years were then re-contacted and invited to participate in an additional Decision Making Study. Each survey included questions to check for attention and consistency of responding (see Goodman, Cryder, & Cheema, 2013; Paolacci, Chandler, & Ipeirotis, 2010). For the purposes of this analysis, we selected data from individuals who had experienced the loss of a parent only. The mean age of participants was 42.35 (SD = 10.94, range 25–65). Forty-five percent were female. Fifty percent had a college degree or above (High school 12%, Some college 32%, College degree 39%, graduate degree or above 17%). Eighty-six percent identified as Caucasian, 8% as African American, five percent as Asian and one percent as Hispanic. The average time since loss was 24 months [SD = .08; range 12–36 months]. The study was approved by Teachers College Institutional Review Board (15–260: Decision Making and Life Events).
Procedure
The measures and statistical procedures were identical to those used in Analysis 1 with the following addition: We compared the network connectivity between the spousal loss and parental loss samples using the Network Comparison Test (NCT) implemented in the Rpackage NCT (van Borkulo et al., 2015). The NCT tests for a difference in the overall network connectivity (or global strength) by calculating the difference in the weighted sum of the absolute connections for repeated samples of randomly regrouped individuals. For this analysis the sampling procedure was repeated 1000 times and gamma was set to 0. This process results in a distribution of network differences under the null hypothesis (that both groups are equal), which can be used to test the significance of observed differences between the two groups. The observed difference is considered significant at the threshold of .05 (see van Borkulo et al., 2015). Extensions of the package test for network structure invariance and sample differences in individual edge strength using the same procedure (van Borkulo, 2016). We also examined the similarity of the network edge weights using Spearman Brown correlations.
Results
Mean scores for each of the items on the PG 13 are presented in Table 1. Cronbach alpha for the scale in this sample was α = .92. Again, the inclusion/exclusion of functional impairment did not change scale reliability. The most frequently endorsed items at any level were Yearning (88%) and Emotional Pain at the lost relationship (80%). The least frequently endorsed were Avoidance (38%), Difficulties with Trust (41%) and Difficulties Re-Engaging with Life (41%). The mean number of items endorsed at least once in the past month (or slightly) was 5.66, the modal number was 10. The mean sum score (network activation) was 22.09 (SD=9.59). Forty-eight (25%) participants scored in a range suggestive of possible diagnosis of PG (Prigerson et al., 2009).
Relationship among symptoms
The LASSO regularized partial correlation network for parent loss is presented in Figure 1b. The strength, closeness and betweenness indices of centrality are presented in Figure 2b. No single node appeared to have unique centrality. The two strongest nodes on all three measures of centrality were Emotional Pain and Shock. Difficulties with Trust was similarly high on strength and closeness, but low on betweenness. Again, the weakest node in the network was Avoidance. The second weakest appeared to be Role Confusion, then Yearning.
Stability
The full results of the stability analysis are presented in the supplementary material (Figure S6–9). As in Study 1, the strongest edge, between Yearning and Emotion Pain was significantly stronger than most other edges in the network. The edges between Emotional Pain and Shock, and Meaninglessness and Difficulty Re-engaging were also stronger than about half of the remaining edges. Many of the remaining edges were not reliably different from one another. As with Study 1, this suggests caution should be taken in drawing conclusions about the differential strength of many relationships within the network. Consistent with previous findings, strength centrality was shown to be the most robust index of centrality. Significance tests for differences in node strength centrality indicated that Emotional Pain was significantly more central than Yearning, Avoidance, and Role Confusion. Role Confusion was also significantly less central than Shock/Stunned and Difficulties with Trust. No other reliable differences between strength centrality were identified. The CS-coefficients for Strength, Closeness and Betweenness were 0.34, 0.17, and 0.10, respectively.
Comorbidity
The LASSO regularized partial correlation network including the 11 PG-13 and 7 depression symptoms is presented in Figure 3b (Cronbach alpha for the 7 and 9 item version of the CES-D in this sample was α = .86 and α = .88, respectively). The spin glass algorithm detected 3 communities of nodes in this network. Again, no community of nodes in this sample of parentally bereaved individuals contained a combination of grief and depression symptoms. One community comprised Yearning, Emotional Pain and Shock/Stunned, another contained the remainder of the PG-13 symptoms. A third community contained all depression items. Bootstrapped confidence intervals for this larger network are presented in Figure S10.
Network comparison
The NCT did not reveal any significant differences between the produced networks in terms overall global strength (difference =.43, p < 0.19) or network structure (.24 p< 0.42). Spearman Brown correlations indicated a high degree of similarity in edge weights across networks (r = .75). Overall, this analysis suggests that although there were some differences in node strength and in the relationships between nodes across spousal and parental loss, the networks from the two studies were generally similar.
Discussion
In this analysis we examined network relationships between symptoms of PG and depression in a sample of parentally bereaved individuals. Again, we observed a strong relationship between Yearning and Emotional Pain, and moderate relationships between Meaninglessness and Difficulties Re-Engaging, Emotional Pain and Stunned/Shock. Across centrality indices, Emotional Pain and Shock/Stunned were the most central nodes. As with Study 1, strength centrality was the most stable centrality index and again indicated that strength findings were interpretable with some care. The CS-coefficients for betweenness and closeness again fell below the suggested guidelines for interpretation (Epskamp, et al., 2016). Also consistent with Study 1, Avoidance appeared the weakest node in the network (lowest strength centrality), and nodes representing PG and depression clustered separately. There were no strong links between the symptom groups, and there appeared to be fewer non-zero edges between the groups than in the Spousal Loss sample.
Study 1 and Study 2 participants differed both in terms of the type of lost relationship, and the sample selection method. Nonetheless, the resulting networks showed many similarities: Yearning, Emotional Pain and Shock/Stunned clustered together and Avoidance was relatively peripheral in both networks. The strongest edge in both networks was between Yearning and Emotional Pain. Meaninglessness also had edges that were significantly stronger in both networks. In Study 1, Meaninglessness formed a cluster with Role Confusion, Emotional Numbness and Difficulties Re-Engaging with Life. In Study 2, these links were also evident, if not as strong.
There were, however, some differences between the networks: In contrast to Study 1, in Study 2, Yearning and Role Confusion showed somewhat lower strength centrality. Together with Avoidance, our test of significance found that these nodes showed significantly less strength centrality than Emotional Pain. Difficulties with Trust also showed somewhat higher strength centrality in Study 2 than in Study 1 (see supplementary material). This later finding may reflect differences in recruitment methods, and we discuss this more in the general discussion. Overall, however the produced networks were shown to be largely similar.
General Discussion
Although the past decade has witnessed a surge of interest in PG, gaps in our understanding remain, and the best methods of conceptualising the construct have been hotly debated (Bryant, 2014; Maciejewski et al., 2016; Maercker et al., 2013; Prigerson et al., 2009; Shear et al., 2011). Recent studies have attempted to advance our understanding of bereavement using a novel network analysis approach to understanding psychopathology (Fried et al., 2015; Robinaugh et al., 2014). The network approach argues that rather than reflecting the action of a latent disease or entity, “symptoms” cluster together and become a “disorder” as a result of direct causal relationships between each other (Borsboom & Cramer, 2013; McNally, 2016; McNally et al., 2015). A key implication of this approach is that by identifying and modifying the most influential nodes within the network it should be possible to change the functioning of the network, and so reduce (or increase) activation of other nodes. This approach to understanding psychopathology is consistent with the clear causal relationships observed across many mental health conditions (Hofmann et al., 2016; McNally, 2016). To date, studies applying network approaches to bereavement have shown some promise, in terms of exploring how symptoms may be connected to each other (Fried et al., 2015) and identifying the most influential nodes (Robinaugh et al., 2014). They have, however, been limited in their reliance on a single, archival data set and analysis of only one type of bereavement, spousal bereavement. In the current investigation, we addressed these limitations by applying network analyses to newly collected data sets that examined samples that had experienced different kinds of bereavements. We also applied newly developed techniques to assess the stability of the estimated networks. In doing so, we observed a number of similarities and several differences with previous analyses. Emotional Pain, the node with highest centrality in Robinaugh et al. (2014) had high strength centrality in our parental loss sample (Study 2), where it was found to have significantly greater strength than several other nodes within the network. In contrast, Emotional Pain evidenced less strength centrality in our analysis of spousal loss (Study 1). Nonetheless, the edge between Emotional Pain and Yearning was the strongest edge in both studies, being significantly stronger than nearly every other edge in either network. Questions have been raised about the extent to which node centrality, of itself, provides clarification about network influence and specific activation pathways. In these analyses, as with a previous network stability analyses (e.g. Epskamp, et al 2016) the stability of the closeness and betweenness indexes was low. Theoretically, closeness and betweenness are interesting network properties; however, ongoing research will be needed to determine extent to which these indices represent reliable and interpretable measures. Moreover, a node may be central to a network because it is activated by many other nodes, rather than being a cause of activation. Indeed, “emotional pain”, or distress, is likely to be a core consequence of many psychological conditions. The cross-sectional nature of these analyses means that we cannot determine the direction of the relationship between these, or other nodes in the networks. Stronger conclusions about causal relationships among nodes will need to await future studies involving longitudinal assessment of symptoms (e.g., Bringmann, Lemmens, Huibers, Borsboom, & Tuerlinckx, 2015) or experimental manipulations. However, the pattern of findings observed in this study suggests further investigation is warranted as to the extent to which Yearning and Emotional Pain represent separate or overlapping symptoms.
Consistent with Robinaugh et al. (2014), Avoidance was the least connected node within the network, evidencing significantly lower strength than the strongest nodes in both networks. The role of avoidance in PG is of interest, both in terms of theoretical models of PG and treatment approaches (Boelen, de Keijser, van den Hout, & van den Bout, 2007; Bryant et al., 2014; Maccallum & Bryant, 2013; Shear et al., 2007; Stroebe, Schut, & Boerner, 2010). The relationship between avoidance behaviors and PG has been the focus of a number of investigations using both self-report and experimental paradigms (Baker et al., 2016; Boelen, van den Bout, & van den Hout, 2010; Boelen & van den Hout, 2008; Maccallum, Sawday, Rinck, & Bryant, 2015; Schnider, Elhai, & Gray, 2007). Results have been mixed; some of these studies have found a relationship between avoidance and PG and others have not. Our findings suggest that avoidance, as indexed in these studies, had little consistent relationship with activation in the rest of the network. This does not rule out the possibility that avoidance assessed at different time scales, via other methods (e.g. experimental paradigms), or in a different sample (e.g. treatment vs non-treatment seeking individuals) may impact network activation to a greater degree. Indeed, there is increasing recognition that avoidance per se is neither adaptive nor maladaptive; rather, it’s relationship with outcomes is dependent on individual difference factors and the context is which it is used (Bonanno, Papa, Lalande, Westphal, & Coifman, 2004). For some individuals, avoidance may be associated with lower activation of other items in the network, for others it may be associated with increased activation. Together, these findings suggest the need to adopt a more nuanced approach to understanding avoidance in PG.
The development of new sources of meaning and life goals is thought to be a key task in successful adaption to loss, and is a focus of effective treatments for PG (Bryant et al., 2014; Maccallum & Bryant, 2013; Neimeyer, 2016; Shear et al., 2016). Consistent with this view, experiencing a sense of meaninglessness emerged as a key node in both the spousal and parental loss networks. Meaninglessness showed relationships with Role Confusion, Numbness and Difficulties reengaging with life. Again, the application of longitudinal network modeling approaches may offer a novel method of examining temporal changes in these relationships to ascertain the most effective pathways for reducing meaninglessness. Although these connections appeared somewhat stronger in the spousal loss sample than the parental loss sample, the network comparison test suggested these were not reliably significant differences. Network analysis is primarily concerned with the relationships between nodes rather than absolute severity. We note however, that the two samples did not differ significantly in terms of mean severity on these four nodes. The spousal loss group did endorse significantly greater yearning, emotional pain and loneliness than the parental loss group (p <.001).
Overwhelmingly, investigations of bereavement outcomes have focused on spousal loss. Although there is some evidence that absolute rates of distress may not differ between loss types (Maccallum, Galatzer-Levy, & Bonanno, 2015), the extent to which different loss types may be associated with different symptom relationships has yet to be established. We were interested in investigating networks associated with different loss types, as it is possible that loss type may impact functioning in different ways. For example, death of a spouse may have a greater impact on social networks and social functioning, impacting yearning, loneliness, and re-engagement more so than the death of a parent. The current study provides some evidence that there are many similarities between the network relationships across loss types, as assessed via the PG-13. We have noted, however, some differences between the results of the current study and that of Robinaugh et al. (2014). Emotional Pain did not emerge as the central node in our spousal loss sample. Further, loneliness did not appear to function as a gateway node between PG and depression in either network; although in our spousal loss sample it was strongly linked with Sadness, which did connect to PG symptoms. However, there are a number of factors that limit any direct comparisons that can be made with previous studies, including the analytic and node inclusion differences outlined previously. There were also sampling differences across the studies reported in this paper, and in prior studies. Participants in Study1 agreed to volunteer in a study investigating grief responses, participants in Study 2 had volunteered to participate in research that was not grief focused, whereas, participants in the CLOC study, examined in previous papers, were initially enrolled in a population based survey. With respect to the samples investigated in this paper, there are likely a number of differences between participant groups that may impact network relationships. For example, those who lost a spouse were somewhat older (M=57 years) than those who lost a parent (M=46 years). Although the current pattern of findings suggests there may be some differences related to sample features, there are also similarities across bereavement types. For these analyses, we took advantage of recently collected data sets, which limits to some extent our ability to make direct comparisons. Future studies that collect data specifically for the purpose network analyses could explore these issues more thoroughly.
Until recently there has been no way of assessing the stability or reliability of estimated networks (Constantini et al., 2015; Epskamp et al., 2016). This software is now available (Epskamp et al., 2016) and was used in the present study to analyze the stability of networks produced in these analyses. Findings suggested many of the apparent differences between moderate edges and nodes in networks in both Study 1 and 2 were not stable in response to sampling variation (Epskamp et al., 2016). For this reason, we have not interpreted differences between moderate edges. We were also able to assess the stability of our centrality indices. Strength centrality was the most stable index. Our findings indicate that one third of participants could be dropped from Study 1 and 2, respectively, while maintaining 95% probability of a very large correlation with the strength centrality of the original network. Epskamp et al., (2016) presented findings from a simulation study to assist interpretation of CS coefficients; based on their recommendations our strength centrality coefficients fell within the range considered interpretable, although with some care (0.25 < CS coefficient > 0.5). This is perhaps consistent with our inclusion of participants along the continuum of PG severity, and the finding that many of the moderate edges in our networks were not reliably different from one another. As noted by the developers, however, these cut offs were somewhat arbitrary and do not represent definitive guidelines (Epskamp, et al., 2016). We present the networks, stability and comparison results in this paper, as at present the field of network analysis is primarily data driven and exploratory (Epskamp et al., 2016). It is through the application of such approaches to existing data sets that the field will develop and generate norms.
Recent simulation studies now provide greater direction to researchers on issues of network estimation reliability and power (Epskamp et al., 2016). These indicate that power is impacted both by sample size and the number of nodes (and hence edges to be estimated) included in the networks. These simulations place our sample sizes towards the lower end of the acceptable power range. Similarly, the network comparison test represents a new addition to the field. Recommendations relating to power have yet to be determined. Van Borkulo and colleagues (Van Borkulo, Boschloo, Kossakowski, & Borsboom, 2017) have explored data simulation to examine the boundaries of power associated with changes in sample size, network sparsity, and sample size differences. Their simulations again suggest our sample size is towards the lower end of the acceptable power range to detect a large difference between networks. A consideration in our decision not to compare networks containing both PG and depression was the reduction in power that accompanies an increased number of nodes. Based on these evolving recommendations, future studies would benefit from larger sample sizes to enhance the potential stability of estimated networks, and enable inclusion of additional items. For example, consistent with Robinaugh et al. (2014) we excluded items on the CES-D that did not closely resemble symptoms of Major Depression. However, it would be interesting in future investigations to examine the extent to which changes in social functioning following a bereavement link to PG and depressive symptomatology.
We also note that the networks described in this investigation were constructed based on self-report symptom measures focussing on diagnostic criteria. The PG-13 is a self-report measure that has demonstrated relationships with functional outcomes (Maciejewski et al., 2016; Prigerson et al., 2009). It will be interesting to compare the extent to which the same relationships may be observed following clinical interview. Moreover, the PG-13 indexes symptoms predictive of poor outcomes, and was not intended to index causal mechanisms. While network modeling has focussed predominantly on the relationships between symptoms, it is its potential to examine relationships between proposed mechanisms of change that is its most tangible advance. Future longitudinal studies that move beyond symptom measures and include theoretically relevant nodes within the network (e.g., intolerance of distress, attentional biases), will help to shed further light on the nature of the relationships and core mechanisms involved in the maintenance of PG.
Supplementary Material
General Scientific Summary.
Prolonged grief is a potential chronic and debilitating consequence of bereavement. This study applies a novel approach that maps the syndrome as a network of direct relationships between symptoms. Findings indicate that symptoms such as yearning, emotional pain, and meaningless may be more central to the network than others and highlights similarities between the networks produced following loss of a spouse and parent.
Acknowledgments
This research is funded by an NIH grant R01MH091034 awarded to George A Bonanno
Footnotes
Of the 180 participants included in the current study, 80 participated in Maccallum and Bonanno (2016).
This paper represents original work. At the time of submission, the data and ideas appearing in the manuscript have not been presented, published or shared in any other form.
References
- American Psychiatric Association. Diagnostic and statistical manual of mental disorders. 5th. Washington, DC: American Psychiatric Association; 2013. [Google Scholar]
- Baker WA, Keshaviaha A, Horenstein A, Goetter EM, Mauro C, Reynolds CF, Simon NM. The role of avoidance in Complicated Grief: A detailed examination of the Grief-Related Avoidance Questionnaire (GRAQ) in a large sample of individuals with Complicated Grief. Journal of Loss and Trauma. 2016:533–547. doi: 10.1080/15325024.2016.1157412. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Boelen PA, de Keijser J, van den Hout MA, van den Bout J. Treatment of complicated grief: A comparison between cognitive-behavioral therapy and supportive counseling. Journal of Consulting and Clinical Psychology. 2007;75:277–284. doi: 10.1037/0022-006x.75.2.277. [DOI] [PubMed] [Google Scholar]
- Boelen PA, van den Bout J, van den Hout MA. A prospective examination of catastrophic misinterpretations and experiential avoidance in emotional distress following loss. Journal of Nervous and Mental Disease. 2010;198:252–257. doi: 10.1097/NMD.0b013e3181d619e4. [DOI] [PubMed] [Google Scholar]
- Boelen PA, van den Hout MA. The role of threatening misinterpretations and avoidance in emotional problems after loss. Behavioural and Cognitive Psychotherapy. 2008;36:71–87. doi: 10.1017/s1352465807004079. [DOI] [Google Scholar]
- Bonanno GA, Neria Y, Mancini A, Coifman KG, Litz B, Insel B. Is there more to complicated grief than depression and posttraumatic stress disorder? A test of incremental validity. Journal of Abnormal Psychology. 2007;116:342–351. doi: 10.1037/0021-843x.116.2.342. [DOI] [PubMed] [Google Scholar]
- Bonanno GA, Papa A, Lalande K, Westphal M, Coifman K. The importance of being flexible: The ability to both enhance and suppress emotional expression predicts long-term adjustment. Psychological Science. 2004;15:482–487. doi: 10.1111/j.0956-7976.2004.00705.x. [DOI] [PubMed] [Google Scholar]
- Borsboom D, Cramer AOJ. Network analysis: An integrative approach to the structure of psychopathology. Annual Review of Clinical Psychology. 2013;9:91–121. doi: 10.1146/annurev-clinpsy-050212-185608. [DOI] [PubMed] [Google Scholar]
- Boschloo L, van Borkulo CD, Rhemtulla M, Keyes KM, Borsboom D, Schoevers RA. The network structure of symptoms of the Diagnostic and Statistical Manual of Mental Disorders. PLOS One. 2015;10 doi: 10.1371/journal.pone.0137621. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bringmann LF, Lemmens LHJM, Huibers MJH, Borsboom D, Tuerlinckx F. Revealing the dynamic network structure of the Beck Depression Inventory II. Psychological Medicine. 2015;45:747–757. doi: 10.1017/s0033291714001809. [DOI] [PubMed] [Google Scholar]
- Bryant RA. Prolonged grief: where to after Diagnostic and Statistical Manual of Mental Disorders, 5th Edition? Current Opinion in Psychiatry. 2014;27:21–26. doi: 10.1097/YCO.0000000000000031. [DOI] [PubMed] [Google Scholar]
- Bryant RA, Kenny L, Joscelyne A, Rawson N, Maccallum F, Cahill C, Nickerson A. Treating prolonged grief disorder: A randomized controlled trial. JAMA: Psychiatry. 2014;71:1332–1339. doi: 10.1001/jamapsychiatry.2014.1600. [DOI] [PubMed] [Google Scholar]
- Constantini G, Epskamp S, Borsboom D, Perguini M, Mottus R, Waldorp LJ, Cramer AOJ. State of the aRt personality research: A tutorial on network analysis of personality data in R. Journal of Research in Personality. 2015;54:13–29. [Google Scholar]
- Csardi G, Nepusz T. The igraph software package for complex network research. International Journal of Complex Systems. 2006:1695. [Google Scholar]
- Epskamp S, Borsboom D, Fried EI. Estimating psychological networks and their stability: A tutorial paper. arXiv. 2016 doi: 10.3758/s13428-017-0862-1. 1604.08462 [stat.AP] [DOI] [PMC free article] [PubMed] [Google Scholar]
- Epskamp S, Cramer AOJ, Waldorp LJ, Schmittmann VD, Borsboom D. qgraph: Network visualizations of relationships in psychometric data. Journal of Statistical Software. 2012;48:1–18. [Google Scholar]
- Epskamp S, Fried EI. A tutorial on regularized partial correlation networks. 2016 doi: 10.1037/met0000167. arxiv.org/abs/1607.01367 [stat.AP] [DOI] [PubMed]
- Fried EI, Bockting C, Arjadi R, Borsboom D, Amshoff M, Cramer AOJ, Stroebe M. From loss to loneliness: The relationship between bereavement and depressive symptoms. Journal of Abnormal Psychology. 2015;124:256–265. doi: 10.1037/abn0000028. [DOI] [PubMed] [Google Scholar]
- Fried EI, van Borkulo CD, Cramer AOJ, Boschloo L, Schoevers RA, Borsboom D. Mental disorders as networks of problems a review of recent insights. Social Psychiatry and Psychiatric Epidemiology. 2016;52:1–10. doi: 10.1007/s00127-016-1319-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Friedman J, Hastie T, Tibshirani R. Sparse inverse covariance estimation with the graphical lasso. Biostatistics. 2008;9:432–441. doi: 10.1093/biostatistics/kxm045. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Goodman JK, Cryder CE, Cheema A. Data collection in a flat world: The strengths and weaknesses of Mechanical Turk samples. Journal of Behavioral Decision Making. 2013;26:213–224. doi: 10.1002/bdm.1753. [DOI] [Google Scholar]
- Hawton K, Salkovskis PM, Kirk J, Clark DM, editors. Cognitive Behaviour Therapy for Psychiatric Problems: A Practical Guide. Oxford, UK: Oxford University Press; 1989. [Google Scholar]
- Hofmann SG, Asmundson GJG, Beck AT. The science of cognitive therapy. Behavior Therapy. 2013;44:199–212. doi: 10.1016/j.beth.2009.01.007. [DOI] [PubMed] [Google Scholar]
- Hofmann SG, Curtiss J, McNally RJ. A complex network perspective on clinical science. Perspective of Psychological Science. 2016;11:597–605. doi: 10.1177/1745691616639283. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kohout FJ, Berkman LF, Evans DA, Cornoni-Huntley Two shorter forms of the CES_D Depression Symptoms Index. Journal of Aging and Health. 1993;5:179–193. doi: 10.1177/089826439300500202. [DOI] [PubMed] [Google Scholar]
- Leahy RL, editor. Contemporary Cognitive Therapy: Theory Research and Practice. New York, NY: Guilford Press; 2006. [Google Scholar]
- Maccallum F, Bonanno GA. The economics of losing a loved one: Delayed reward discounting in prolonged grief. Clinical Psychological Science. 2016;4:683–690. doi: 10.1177/2167702615605827. [DOI] [Google Scholar]
- Maccallum F, Bryant RA. A Cognitive Attachment Model of prolonged grief: Integrating attachments, memory, and identity. Clinical Psychology Review. 2013;33:713–727. doi: 10.1016/j.cpr.2013.05.001. [DOI] [PubMed] [Google Scholar]
- Maccallum F, Galatzer-Levy IR, Bonanno GA. Trajectories of depression following spousal and child bereavement: A comparison of the heterogeneity in outcomes. Journal of Psychiatric Research. 2015;69:72–79. doi: 10.1016/j.jpsychires.2015.07.017. [DOI] [PubMed] [Google Scholar]
- Maccallum F, Sawday S, Rinck M, Bryant RA. The push and pull of grief: Approach and avoidance in bereavement. Journal of Behavior Therapy and Experimental Psychiatry. 2015;48:105–109. doi: 10.1016/j.jbtep.2015.02.010. [DOI] [PubMed] [Google Scholar]
- Maciejewski PK, Maercker A, Boelen PA, Prigerson HG. “Prolonged grief disorder” and “persistent complex bereavement disorder”, but not “complicated grief”, are one and the same diagnostic entity: an analysis of data from the Yale Bereavement Study. World Psychiatry. 2016;15:266–275. doi: 10.1002/wps.20348. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Maercker A, Brewin CR, Bryant RA, Cloitre M, Reed GM, van Ommeren M, Saxena S. Proposals for mental disorders specifically associated with stress in the International Classification of Diseases-11. The Lancet. 2013;381(12):1683–1685. 62191–6. doi: 10.1016/S0140-6736. [DOI] [PubMed] [Google Scholar]
- McNally RJ. Can network analysis transform psychopathology? Behavioural Research and Therapy. 2016;86:95–104. doi: 10.1016/j.brat.2016.06.006. [DOI] [PubMed] [Google Scholar]
- McNally RJ, Robinaugh DJ, Wu GWY, Wang L, Deserno MKB,D. Mental disorders as causal systems: A network approach to Posttraumatic Stress Disorder. Clinical Psychology Science. 2015;3:836–849. doi: 10.1177/2167702614553230. [DOI] [Google Scholar]
- Mennin D, Ellard KK, Fresco DM. United we stand: Emphasizing commonalities across cognitive-behavioral therapies. Behavior Therapy. 2013;44:234–248. doi: 10.1016/j.beth.2013.02.004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Neimeyer RA. Meaning Reconstruction in the Wake of Loss: Evolution of a Research Program. Behaviour Change. 2016;33:65–79. doi: 10.1017/bec.2016.4. [DOI] [Google Scholar]
- Nielsen MK, Neergaard MA, Jensen AB, Vedsted P, Bro F, Guldin MB. Predictors of complicated grief and depression in bereaved caregivers: A nationwide prospective cohort study. Journal of Pain and Symptom Management. 2016;53:540–550. doi: 10.1016/j.jpainsymman.2016.09.013. [DOI] [PubMed] [Google Scholar]
- Paolacci G, Chandler J, Ipeirotis PG. Running experiments on Amazon Mechanical Turk. Judgment and Decision Making. 2010;5:411–419. [Google Scholar]
- Prigerson HG, Horowitz MJ, Jacobs SC, Parkes CM, Aslan M, Goodkin K, Maciejewski PK. Prolonged Grief Disorder: Psychometric validation of criteria proposed for DSM-V and ICD-11. Plos Medicine. 2009;6:e1000121. doi: 10.1371/journal.pmed.1000121. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Radloff LS. The CES-D Scale: A self report depression scale for research in the general population. Applied Psychological Measurement. 1977;1:385–401. [Google Scholar]
- Reichardt J, Bornholdt S. Statistical mechanics of community detection. Physical Review E. 2006;74 doi: 10.1103/PhysRevE.74.016110. [DOI] [PubMed] [Google Scholar]
- Robinaugh DJ, LeBlanc NJ, Vuletich HA, McNally RJ. Network Analysis of Persistent Complex Bereavement Disorder in conjugally bereaved adults. Journal of Abnormal Psychology. 2014;123:510–522. doi: 10.1037/abn0000002. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Schnider KR, Elhai JD, Gray MJ. Coping style use predicts posttraumatic stress and complicated grief symptom severity among college students reporting a traumatic loss. Journal of Counseling Psychology. 2007;5:344–350. doi: 10.1037/00220167.54.3.344. [DOI] [Google Scholar]
- Shear MK, Frank E, Houck PR, Reynolds CF., III Treatment of complicated grief: A randomized controlled trial. JAMA: Journal of the American Medical Association. 2005;293:2601–2608. doi: 10.1001/jama.293.21.2601. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shear MK, Monk T, Houck P, Melhem N, Frank E, Reynolds C, Sillowash R. An attachment-based model of complicated grief including the role of avoidance. European Archives of Psychiatry and Clinical Neuroscience. 2007;257:453–461. doi: 10.1007/s00406-007-0745-z. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shear MK, Reynolds CF, Simon NM, Zisook S, Wang Y, Mauro C, Skritskaya N. Optimizing treatment of Complicated Grief: A randomized clinical trial. JAMA psychiatry. 2016;73:685–694. doi: 10.1001/jamapsychiatry.2016.0892. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Shear MK, Simon N, Wall M, Zisook S, Neimeyer R, Duan N, Keshaviah A. Complicated grief and related bereavement issues for DSM-5. Depression and Anxiety. 2011;28:103–117. doi: 10.1002/da.20780. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Stroebe MS, Schut H, Boerner K. Continuing bonds in adaptation to bereavement: Toward theoretical integration. Clinical Psychology Review. 2010;30:259–268. doi: 10.1016/j.cpr.2009.11.007. [DOI] [PubMed] [Google Scholar]
- van Borkulo CD. Statistical Comparison of Two Networks Based on Three Invariance Measures 2016 [Google Scholar]
- van Borkulo CD, Boschloo L, Borsboom D, Penninx BWJH, Waldorp LJ, Schoevers RA. Association of symptom network structure with the course of longitudinal depression. JAMA Psychiatry. 2015;72:1219–1296. doi: 10.1001/jamapsychiatry.2015.2079. [DOI] [PubMed] [Google Scholar]
- Van Borkulo C, Boschloo L, Kossakowski JJ, Borsboom D. Comparing network structures on three aspect: A permutation test. 2017 doi: 10.1037/met0000476. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.