Abstract
Background:
There has been increasing interest in classifying inflammatory phenotypes of depression. Most investigations into inflammatory phenotypes only have tested whether elevated inflammation is associated with elevated levels of depression symptoms, or risk for a diagnosis. This study expanded the definition of phenotype to include the structure of depression symptoms as a function of inflammation.
Methods:
Network models of depression symptoms were estimated in a sample of 4,157 adults (mean age = 47.6, 51% female) from the 2015-2016 National Health and Nutrition Examination Survey (NHANES). Analyses included comparisons of networks between those with elevated (C-reactive protein (CRP) values ≥3.0 mg/L; N = 1,696) and non-elevated CRP (N = 2,841) as well as moderated network models with CRP group status and raw CRP values moderating the associations between depression symptoms.
Results:
Differences emerged at all levels of analysis (global, symptom-specific, symptom—symptom associations). Specifically, the elevated CRP group had greater symptom connectivity (stronger total associations between symptoms). Further, difficulty concentrating and psychomotor difficulties had higher expected influence (concordance with other symptoms) in the elevated CRP group. Finally, there was evidence that several symptom—symptom associations were moderated by CRP.
Conclusions:
This study provides consistent evidence that the structure of depression symptoms varies as a function of CRP levels. Greater symptom connectivity might contribute to why elevated CRP is associated with treatment-resistant depression. Additionally, differences in symptom structure might highlight different maintenance mechanisms and treatment targets for individuals with compared to those without elevated CRP. Finally, differences in symptom structure as a function of CRP highlight a potential misalignment of standard depression measures (the structure of which are evaluated on groups unselected for CRP levels) and the presentation of depression symptoms in those with elevated CRP.
Keywords: Depression, inflammation, CRP, structure, network analysis, immunology
Introduction
Elevated inflammatory physiology (e.g., concentrations of C-reactive protein (CRP)) is an established correlate of (e.g., Haapakoski et al., 2015) and risk factor for (Moriarity et al., 2020b) depression. However, it is present in only a subset of individuals with Major Depressive Disorder (MDD) (Raison & Miller, 2011). Additionally, there is inconsistency in the presence, size, and direction of associations between depression and inflammatory proteins (see Horn et al., 2018 for a meta-analysis focusing on CRP). This has led to calls for the inflammatory phenotyping of depression (Felger et al., 2018) and increased modeling of specific symptoms and symptom subtypes in immunopsychiatry (Moriarity & Alloy, 2020). These pursuits have included profiling symptoms associated with inflammation (Fried et al., 2019; Jokela et al., 2016; Kappelmann et al., 2020; Moriarity et al., 2020a) as well as inflammatory proteins and cell counts associated with MDD (Lynall et al., 2020). Although there are many different inflammatory proteins, CRP, an acute phase reactant, arguably is the most widely studied. Partially, this is attributable to CRP being a good marker of general inflammatory levels, a relatively consistent predictor of depression symptoms, and is the reason that CRP is the focus of this study.
Inflammatory Phenotype of Depression Symptoms
Initial studies have found support for differential relationships between CRP and specific depression symptoms. For example, higher CRP has been associated with more depressed mood (indexed by dysphoria subscales, Niles et al., 2018; White et al., 2017) and greater anhedonia and reward abnormalities (Felger et al., 2016; Haroon et al., 2016; Moriarity et al., 2019). It also has been associated with somatic complaints including fatigue and sleeping problems (Fried et al., 2019; Jokela et al., 2016; White et al., 2017) as well as changes in appetite (Jokela et al., 2016; Lamers et al., 2018). In fact, Madj and colleagues (2020) found that, out of all inflammatory proteins, CRP was associated the most consistently with sleeping problems and fatigue. Several studies also have found associations between CRP and cognitive symptoms (Gimeno et al., 2009) and suicidality (Kim et al., 2007). Studies such as these set the foundation for the characterization of an inflammatory phenotype of depression.
Increased understanding of the specific facets of depression for which inflammation confers risk promises a wide variety of benefits. For example, it could help clinicians identify depressed patients for whom inflammation might be a factor and, consequently, who might benefit from anti-inflammatory interventions. Further, it can help generate hypotheses about which specific depression symptoms are most likely to respond to these treatment options. Also, analyses examining specific symptoms or symptom dimensions are inherently transdiagnostic; a finding that CRP is associated with heightened anhedonia might be as relevant for any condition involving low approach motivation or reward sensitivity as it is for depression. Additionally, if inflammation only is associated with specific facets of depression, aggregation of these dimensions with those that have no relationship with inflammation increases the noise-to-signal ratio in analyses, exacerbating risk of Type-II error and deflating effect sizes. Consequently, the classification of an inflammatory phenotype of depression has the potential to meaningfully impact research and public health.
Inflammatory Symptom Phenotypes: A Network Perspective
Most research investigating inflammatory phenotypes of depression have tested whether inflammation is associated with higher levels of depression. However, “phenotypes” need not be defined solely through the presence/severity of symptoms; they should include the structure of psychopathology itself. The network theory of psychopathology posits that symptoms cluster because they directly influence one another (e.g., sleeping problems and fatigue are correlated because sleeping problems cause fatigue; Borsboom & Cramer, 2013). Consequently, increases or decreases in one symptom can influence the presence/severity of connected symptoms, contributing to the onset and maintenance of psychopathology. Symptoms with many and/or strong connections are referred to as central symptoms. Importantly, some metrics of centrality (e.g., expected influence (EI), used in this study) might highlight symptoms with prognostic utility (Elliott et al., 2020). Even if the associations between symptoms do not represent causal relationships in line with network theory, expected influence (the sum of all associations between one symptom and the rest) highlights symptoms with particularly high rates of co-occurrence with other symptoms in the network. The characterization of inflammatory network phenotypes of depression can complement existing methods by investigating whether the structure of depression symptoms varies as a function of inflammation. Given that investigation of inter-item relations is standard practice for measurement construction, differences in symptom structure based on CRP-levels could highlight measurement issues in this literature. These advances could increase insight into depression classification, inflammation-related variability in symptom structure, as well as symptom-level maintenance processes and treatment targets for individuals with depression and elevated inflammation (e.g., individuals with comorbid chronic medical conditions).
The Current Study
This study is an initial exploration of potential differences in depression network structure as a function of CRP levels. In line with discussion about whether the association between inflammation and depression is continuous or categorical (i.e., CRP only is associated with depression during “inflamed” states or among those with clinically-elevated CRP), analyses included group difference and continuous moderation (CRP as a moderator of symptom-symptom associations) analyses. Group-differences analyses separated participants into those with CRP values ≥ and <3.0 mg/L (an established medical cut-off per the American Heart Association). This cut-off was chosen because it is associated with increased risk for several negative outcomes, namely depression and heart disease (Biasucci, 2004; Liukkonen et al., 2006) and is a good estimate for the lower bound of the uppermost tertile of CRP in population studies (Pearson et al., 2003).
There were three a priori hypotheses for the group-differences analyses. 1) Symptom connectivity (the weighted absolute sum of all associations in a network, in other words, how densely connected the symptoms are) would be higher for the elevated CRP group. The rationale for this hypothesis is two-fold. First, heterogeneity in symptom structure could dilute the magnitude of the most extreme associations and the group selected for a unified risk factor (elevated CRP) should have less heterogeneity. Second, greater symptom connectivity at baseline is associated with treatment-resistant depression (van Borkulo et al., 2015), as is elevated inflammation (Sluzewska et al., 1997). 2) If there were significant differences in expected influence (centrality of symptoms) between networks, the group with elevated CRP would have the higher expected influence because of the reduced heterogeneity rationale. 3) If there is a significant difference between groups with respect to any of the symptom—symptom associations (a measure of structural invariance), the stronger association (or associations) is/are expected to be in the elevated CRP group because of the reduced heterogeneity rationale. As there have been no studies about how the structure of symptoms differs as a function of CRP, there were no specific hypotheses regarding which symptoms’ expected influence or which symptom—symptom associations would be moderated in the models.
Methods and Materials
Participants and Procedures
This study utilized the National Health and Nutrition Examination Survey (NHANES) 2015-2016 dataset, a nationally representative community sample of the United States. This dataset was designed by the National Center of Health Statistics (NCHS) at the Centers for Disease Control and Prevention to examine many physical and mental health constructs in the United States. The NCHS oversaw data collection and approved the NHANES study protocol (for more details please see: Centers for Disease Control and Prevention, 2009; Chen et al., 2018; Zipf et al., 2013). 9,544 participants were examined during the 2015-2016 cohort; however, participants had to be older than 1 year old to be eligible for CRP measurement and at least 18 years old for their depression symptom data (Patient Health Questionnaire; PHQ-9) to be publicly available. The PHQ-9 was collected during the same visit as CRP. The final analytic sample consisted of 4,157 adults (see Table 1 for descriptive information). For group-differences analyses, 1,574 participants with CRP values ≥3.0 mg/L (an established medical cut-off per the American Heart Association) were selected for the elevated CRP group. The remaining 2,583 participants comprised the non-elevated CRP group. Group comparisons of age, sex, levels of symptoms, and impairment due to symptoms were conducted using independent samples t-tests (or Welch’s t-test when there were non-equal variances) and Pearson’s Chi-Square Tests to test for differences at p < .05 in SPSS Version 24 (IBM Corp, 2018).
Table 1.
Variable | Entire Sample (N = 4,157) |
EIevated CRP (n = 1,574) |
Non-elevated CRP (n = 2,583) |
---|---|---|---|
M (SD) and range for continuous variables or % for categorical variables | |||
Total Dep. Symptoms | 3.18 (4.18) | 3.78 (4.68) | 2.82 (3.80) |
Range: 0-27 | Range: 0-27 | Range: 0-26 | |
Age | 47.59 (16.33) | 49.16 (15.69) | 46.63 (16.63) |
Range: 20-79 | Range: 20-79 | Range: 20-79 | |
Sex | |||
Female | 51.3 % | 60.5 % | 45.6 % |
Race | |||
Mexican American | 18.6 % | 20.5 % | 17.5 % |
Other Hispanic | 13.6 % | 15.2 % | 12.7 % |
Non-Hispanic White | 32.4 % | 31.3 % | 33.0 % |
Non-Hispanic Black | 20.5 % | 22.9 % | 19.0 % |
Non-Hispanic Asian | 11.1 % | 5.4 % | 14.6 % |
Other | 3.8 % | 4.6 % | 3.3 % |
Note. M = Mean, SD = Standard deviation, CRP = C-reactive protein, Dep = depression
Measures
Depression symptoms
The Patient Health Questionnaire (PHQ-9; Kroenke et al., 2001) is a nine-item self-report measure that was administered to assess the frequency of nine DSM-IV depression diagnostic criteria during the past two weeks. The nine items specifically measured sadness, anhedonia, sleep problems, fatigue, psychomotor difficulties, feeling bad about oneself, difficulty concentrating, changes in appetite, and thoughts of death. This questionnaire also included an additional question asking about the degree of impairment/distress experienced due to depression symptoms. Participants were asked to rate each item using a 4-point Likert scale ranging from 0 (not at all) to 3 (nearly every day), resulting in a range of 0-27. Cronbach’s α was high in this sample (α = .84). 12.2% of the entire sample had a PHQ-9 score higher than 8, a cut-off for clinically-relevant depression (for meta-analysis see Manea et al., 2012). 15.3% of the elevated CRP group was above this cut-off compared to 10.4% of the non-elevated group.
C-reactive protein
Blood was drawn via venipuncture and high sensitivity CRP levels were measured using the SYNCHRON System(s) High Sensitivity C-Reactive Protein reagent. The system portioned out one-part sample to 26-parts reagent into a cuvette and monitored change in absorbance at 940 nanometers. This change is proportional to the concentration of CRP and is used to calculate the concentration based on a single-point adjusted, pre-determined calibration curve. There was a change in lab equipment during the 2015-2016 survey cycle from the Beckman Coulter UniCel DxC 600 Synchron chemistry analyzer to the Beckman Coulter UniCel DxC 600i Synchron chemistry analyzer. An internal comparison study was reported to indicate no statistical adjustment was required to correct for this change. Specimens were frozen at −70 °C until the day of the assay. Samples were run singly as part of a Multi-analyte Biochemistry Panel. Lower limit of detection for CRP was .08 mg/L (values lower than this were set to .08 mg/L). Participants were asked to fast the morning of the blood draw.
Statistical Analyses
Categorical CRP group differences.
In network models, variables are described as “nodes”, and “edges” are the pairwise associations between nodes, controlling for all other associations in the network (Epskamp & Fried, 2016). We estimated two (one for each group) Gaussian graphical models (GGMs), using the R package bootnet (Version 1.3; CRAN link: http://cran.r-project.org/package=bootnet). Because items were ordinal, Spearman correlations were used to estimate the network, consistent with guidelines (see Epskamp & Fried, 2018 for a tutorial). In light of work suggesting that nonregularized network models outperform regularized models when using psychopathology data (Williams et al., 2019), models were not regularized. It is important to note that the package mgm (used for the continuous moderation model described below) uses a slightly different method for calculating edge weights. Specifically, a penalty is applied to reduce the smallest edges to zero, resulting in a not completely nonregularized network. To facilitate comparison, these models were re-estimated in bootnet with the method set to mgm as a sensitivity analysis (see Supplemental Results).
To assess symptom centrality, we calculated expected influence using qgraph (Version 1.6.5; Epskamp et al., 2012). Expected influence quantifies the sum of the edge weights between a given node and all other nodes in the network. In other words, expected influence is how positively (or negatively) a node is connected to all other nodes. Before interpretation, it is necessary to test the stability of the centrality parameters. This was done with a case dropping procedure in bootnet that tested the proportion of the sample that can be dropped from the analysis and result in a .7 correlation between centrality estimates of the original sample and the new sample, resulting in a stability coefficient. A stability coefficient of .25 (25% of the sample dropped) is considered somewhat stable and .50 is considered stable. Given the cross-sectional data, these networks are undirected, meaning that the direction of association cannot be determined. Expected influence estimates were correlated with node variances to evaluate the extent to which they might be driven by measurement characteristics. Large, positive correlations would indicate that expected influence might be driven by variability.
Both groups’ edge weight matrices and expected influences were correlated using Spearman correlations to assess similarity. Between-group comparisons were made using a permutation-based analysis (1,000 iterations) with the R package NetworkComparisonTest (Version 2.2.1; Borkulo et al., 2016), allowing comparison of symptom connectivity (weighted absolute sum of all edges in the network), node expected influence, and network structure (defined as a difference in at least one of the edge weights between the networks; Borkulo et al., 2016).
All network models (including those below) controlled for impairment/distress due to depression symptoms (both to control for levels of severity and because inflammation is a stress sensitive construct that could be influenced by distress experienced secondary to symptoms), age (Czarkowska-paczek, 2016), exercise (metabolic equivalent of tast (MET) minutes; Kasapis and Thompson, 2005), BMI (Mac Giollabhui et al., 2020), smoking ("never", "past" or, "current"; Kushner et al., 2006), sex (Moriarity et al., 2019), race (Khera et al., 2005), and number of chronic illnesses. Variance associated with these covariates was removed from the symptom data before estimating the networks in the group-difference analyses using NetworkComparisonTest because including the covariates as nodes would have compared CRP-group differences in symptom and covariate networks, rather than just symptom networks. This was done by regressing each symptom onto these covariates and using the symptom-residuals as nodes in the network models. This was done separately for each group. Specification of specific edges to moderate is possible in mgm; thus, covariates were included as nodes in the moderated network models described below.
Moderated network models.
R package mgm (Version 1.2.7; Haslbeck & Waldorp, 2020) can estimate moderated network models, making it preferable to bootnet (which cannot) for this aim. Two nonregularized moderated network models were estimated with all nine depression symptoms, the covariates described above, and a moderator (either CRP group status or raw CRP values). Estimating a moderated network model with CRP group status as a moderator does not allow for tests of the hypotheses about differences in global symptom connectivity, expected influence, or structural invariance to be tested, but does facilitate comparability and continuity between the different estimation methods. The moderator was specified to only moderate symptom—symptom edges. Raw CRP had a skewed distribution and was normalized using the nonparanormal transformation (Liu et al., 2009). The moderator function in mgm does not currently handle moderated centrality measures, so only moderated edge weights are reported. To reduce the risk of false positives, mgm employs a threshold (Loh & Wainwright, 2012), even in nonregularized networks, to set small edge weights to zero. Thus, these networks were still subject to some level of regularization, which might be less preferrable than the truly nonregularized networks capable with bootnet. To quantify the uncertainty of the edge weights, 500 bootstraps were estimated and the percentage of bootstraps in which moderated edges were above the threshold (and consequently not set to zero) are reported. Moderated network models were visualized using qgraph. To maximize visual comparison, identical maximum edge weights were imposed on the networks.
Results
Preliminary Analyses
The average individual in the elevated CRP group had a higher PHQ-9 score (Mdiff = .96 points, SEdiff = .14, p < .001, 95% CI = 0.68-1.23, d = .05) and were older at the time of screening (Mdiff = 2.53 years, SEdiff = .51, p < .001, 95% CI = 1.52-3.54, d = .01) compared to the participants in the non-elevated group. Both differences had small effect sizes. Individuals in the elevated group reported more impairment/distress due to depression symptoms, but this difference was not significant (Mdiff = .03, SEdiff = .02, p = .054, 95% CI = .001-.063, d = .13). Additionally, the elevated CRP group had a greater proportion of females (X2(1) = 87.38, p < .001) and there were differences in the racial make-up of the two groups (X2(5) = 96.13, p < .001). See Table 1 for full descriptive statistics for the total sample and each group.
Primary Analyses
Group differences.
Bootstrapped correlation-stability analyses found excellent stability for edge weights and expected influence (EI; 75% for both edges and EI in both networks). The edges between the two networks shared a large correlation (rs = .60, p < .001) and the ordinal ranking of EI was highly congruent between the two networks (rs = .82, p = .011, Table 2). The symptoms with the highest EI in the elevated CRP network were sadness, difficulty concentrating, and feeling bad about oneself. Conversely, the symptoms with the highest EI in the non-elevated CRP network were sadness, feeling bad about oneself, and anhedonia. However, it is important to note that differences in EI ranking are descriptive, not inferential, in nature. Expected influence was not significantly positively correlated with node variance in either the elevated (rs = −.23, p = .552) or non-elevated CRP networks (rs = .15, p = .708), suggesting that EI was not driven by measurement properties. Descriptive statistics for the nodes in all networks can be found in Table 2.
Table 2.
Node | Total Sample Mean (Variance) |
Elevated CRP Mean (Variance) |
Non-elevated CRP Mean (Variance) |
Elevated CRP EI |
Non- elevated CRP EI |
EI group difference p-value |
---|---|---|---|---|---|---|
1. Anhedonia | .39 (.60) | .45 (.70) | .35 (.54) | .77 | .86 | .195 |
2. Sadness | .34 (.48) | .39 (.58) | .31 (.42) | .96 | 1.02 | .373 |
3. Sleep problems | .60 (.85) | .71 (.98) | .54 (.75) | .65 | .76 | .059 |
4. Fatigue | .76 (.83) | .89 (.96) | .68 (.74) | .61 | .58 | .470 |
5. Changes in appetite | .39 (.59) | .51 (.75) | .33 (.49) | .71 | .73 | .720 |
6. Feeling bad for oneself | .24 (.39) | .29 (.49) | .21 (.33) | .88 | .98 | .094 |
7. Difficulty concentrating | .25 (.43) | .29 (.54) | .22 (.36) | .96 | .80 | .007** |
8. Psychomotor problems | .15 (.26) | .18 (.33) | .13 (.22) | .82 | .69 | .019* |
9. Thoughts of | .05 (.10) | .06 (.12) | .05 (.09) | .43 | .47 | .444 |
Note: Means and variances describe raw variables. EI = expected influence
p < .05
p < .01
Consistent with hypotheses, between-group comparisons found that the elevated CRP network had significantly higher global strength than the non-elevated CRP network (global strength difference = .44, p < .001). Both difficulty concentrating and psychomotor problems had higher EI in the elevated CRP group compared to the non-elevated CRP group (difference = .16, p = .007; difference = .13, p = .019, respectively). Further, the structural invariance test was significant, indicating that some of the edge weights differed between networks (maximum difference in edge weights = .18, p < .001). Follow-up pairwise comparisons of edge weights found 10 out of 36 possible edges differed between networks. Specifically, anhedonia—psychomotor difficulties (difference = .12, p = .004), feeling bad for oneself—psychomotor difficulties (difference = .10, p = .018), difficulty concentrating—thoughts of death (difference = .17, p < .001), and psychomotor difficulties—thoughts of death (difference = .13, p = .001) were stronger in the elevated CRP network. Conversely, sadness—changes in appetite (difference = .17, p < .001), sleep problems—psychomotor difficulties (difference = .09, p = .014), changes in appetite—psychomotor difficulties (difference = .11, p = .004), anhedonia—thoughts of death (difference = .18, p < .001), sleeping problems—thoughts of death (difference = .12, p < .001), and feeling bad for oneself— thoughts of death (difference = .15, p < .001) were stronger in the non-elevated CRP network.
CRP group moderated network models.
The moderated network models with CRP group as a moderator found 7 out of 36 symptom—symptom edges were moderated by CRP group (Figure 1). The mgm package z-standardizes and centers all continuous nodes before computing edge weights, so the moderated edge weights below can be interpreted as standardized beta coefficients. Specifically, the following four edges were stronger (i.e., more positive) in the elevated CRP group: sadness—fatigue, anhedonia—changes in appetite, difficulty concentrating—changes in appetite, and difficulty concentrating—sleep problems (moderated weights = .04, .10, .10, .06, respectively). Conversely, three edges were weaker (i.e., less positive) in the elevated CRP group: sadness—changes in appetite, psychomotor problems—changes in appetite, and anhedonia—thoughts of death (moderated weights = −.13, −.09, and −.03, respectively). Stability analyses indicated that all moderated edges were nonzero in at least half of the bootstrapped models (nonzero in 57%, 93%, 90%, 66%, 98%, 71%, and 51% of bootstraps, respectively).
Continuous CRP moderated network models.
The continuous CRP moderated network analysis found that 5 out of 36 symptom—symptom edges were moderated by CRP (Figure 2). Specifically, the following three edges were stronger in individuals with higher CRP: sadness—fatigue, anhedonia—changes in appetite, and difficulty concentrating—changes in appetite (moderated weights = .04, .05, .05, respectively). Conversely, two edges were weaker in individuals with higher CRP: sadness— changes in appetite and anhedonia— fatigue (moderated weights = −.08 and −.03, respectively). Stability analyses indicated that all moderated edges, except anhedonia—fatigue, were nonzero in at least half of the bootstrapped models (nonzero in 58%, 71%, 79%, 96%, 48% of bootstraps, respectively).
Discussion
This study is the first to broaden the concept of “inflammatory phenotype” beyond the presence or severity of symptoms to include the relationships among symptoms themselves. We found that several structural characteristics (i.e., global strength, expected influence, and symptom—symptom associations) of depression networks are moderated by CRP. This extends previous work that found that CRP is not equally associated with all symptoms of depression (e.g., Fried et al., 2019; Haroon et al., 2016; Jokela et al., 2016; Moriarity et al., 2020a), to demonstrate that differences in CRP might be associated with differences in depression symptom structure. Two of the three group-differences hypotheses were supported, with mixed support for the third. First, the elevated CRP group had greater symptom connectivity, indicating stronger total symptom-symptom associations in this group. Importantly, higher symptom connectivity at baseline is associated with treatment resistant depression (van Borkulo et al., 2015), as is elevated CRP (Sluzewska et al., 1997). This result highlights symptom structure (specifically high symptom connectivity) as one way that CRP might influence depression course and treatment outcomes.
Additionally, both difficulty concentrating and psychomotor problems had significantly higher expected influence in the elevated CRP group, suggesting that these symptoms are more central to the symptom structure of depression in individuals with elevated vs. normative levels of inflammation. The result with difficulty concentrating dovetails with research finding that CRP is associated with cognitive deficits (Krogh et al., 2014) and psychomotor problems (Majd et al., 2020). However, it is important to note that Krogh and colleagues (2014) only found an association between CRP and a couple of the cognitive measures they tested (specifically, trail making A (Fitzhugh et al., 1962) and design fluency (Fitzhugh et al., 1962)). Additionally, the majority of evidence (see Majd et al., 2020 for a review) suggests a relationship between CRP and specifically psychomotor retardation (the PHQ-9 did not differentiate between agitation and retardation). Consequently, future research is necessary to investigate potential specificity of cognitive dysfunction domains/psychomotor difficulties related to these findings. Interestingly, Majd and colleagues (2020) highlight several measurement issues common in inflammatory studies of psychomotor problems, including that most measures do not distinguish between physical and cognitive slowing. It is possible that the elevated expected influence of both difficulty concentrating and psychomotor problems reflect shared variance reflective of cognitive slowing/impairment, a robust predictor of depression course and functional impairment (e.g., Haro et al., 2019). Considered in light of theory (Borsboom & Cramer, 2013) and recent empirical work (Elliott et al., 2020) suggesting that central symptoms might have prognostic utility, these symptoms might be particularly relevant for disease course/intervention planning in individuals with elevated CRP. However, longitudinal data is necessary to evaluate the prognostic utility of these findings.
There was mixed support for the hypothesis that stronger symptom—symptom associations would be found in the elevated vs. non-elevated CRP group. The group-difference analysis using NetworkComparisonTest supported variability in depression symptom structure between the two groups (defined as a difference in at least one of the edge weights between the networks); however six out of the ten edges were larger in the non-elevated CRP group. Conversely, both moderated network models (one featuring group status as a moderator, the other featuring continuous CRP, discussed in more detail below) found the majority of moderated edges became stronger at higher levels of CRP. Thus, this project consistently supports the existence of symptom—symptom level differences based on CRP; however, it does not provide support that the majority of symptom—symptom associations strengthen as CRP increases.
Given lack of understanding about whether the relationship between inflammation and depression is categorical or continuous in nature, moderated network models also were estimated. Moderated network models do not allow for some of the comparisons described above (i.e., symptom connectivity, expected influence, structural invariance); however, they can identify associations between specific depression symptoms that vary as a function of CRP levels/group status. The symptoms involved in the most moderated edges were changes in appetite, anhedonia, psychomotor problems, and thoughts of death (nine, six, six, and six edges, respectively, across all three models). When isolating frequencies of symptoms involved in edges that were stronger vs. weaker at higher levels of CRP, changes in appetite, psychomotor difficulties, and anhedonia were among the most frequently involved symptoms for both categories (involved in four, three, and three edges that were stronger at higher levels of CRP and five, three, and three edges that were stronger at lower levels of CRP, respectively). This suggests that CRP does not uniformly strengthen the associations between these depression symptoms. Notable exceptions to this pattern are difficulty concentrating (involved in four edges stronger at higher levels of CRP and zero stronger at lower levels) and thoughts of death (involved in four edges stronger at lower levels of CRP and two stronger at lower levels), suggesting more unidirectional CRP moderations for these two symptoms. In line with network theory (Borsboom & Cramer, 2013), future work should extend this study using longitudinal data to see whether CRP moderates symptom-level mechanisms of depression risk and maintenance. It is worth noting that both moderated network models found fewer differences in edge weights than the NetworkComparisonTest results using the GGMs. There are several potential reasons for this including: 1) the moderated network models were estimated with some level of regularization, reducing some small edge weights to zero; and 2) the moderated network models included many more parameters (edges were estimated for each of the covariates in the moderated network models, whereas their variance was removed prior to network model estimation in the GGMs), reducing power.
There was substantial overlap between the symptoms whose expected influence (i.e., difficulty concentrating, psychomotor difficulties) and symptom—symptom associations (primarily changes in appetite, anhedonia, psychomotor problems, and thoughts of death) were moderated by CRP in this study and symptoms associated with elevated CRP in previous work (Felger et al., 2016; Krogh et al., 2014; Majd et al., 2020; Park and Kim, 2017). This suggests that CRP confers risk for these symptoms as well as modulates their role in the structure of depression symptoms; however, it is noteworthy that the strength of some symptom—symptom associations decreased as CRP increased. This underscores the fact that understanding whether CRP is associated with the severity of a symptom does not elucidate all the ways CRP might influence the presentation of depression.
In conclusion, network characteristics at every level (i.e., global, symptom—rest of network, symptom—symptom) were moderated by CRP, providing consistent evidence that the structure of depression might vary as a function of CRP levels. In addition to implications for treatment and the classification of an inflammatory phenotype of depression, this highlights the need for further research into how psychopathology structures might vary as a function of inflammation. Investigation of symptom structure is standard practice for the development and evaluation of psychopathology measures. If the structure of psychopathology varies as a function of psychopathological risk factors (e.g., inflammation), extant measurement techniques may not be ideal for particular populations of interest (e.g., individuals with comorbid depression and chronic medical conditions).
This study had several important strengths. First, it included a large, demographically diverse sample, maximizing generalizability. This is particularly important given the power constraints of network analyses. Second, unlike many network studies, nonregularized networks were used where possible, which are preferable for psychopathology networks (Williams et al., 2019). Third, the use of an established medical cutoff for elevated low-grade inflammation, rather than choosing a specific proportion of the sample to include in each group (e.g., top vs. bottom 1/3rd of CRP values), maximizes interpretability of results in the context of the extant health psychology literature and increases the reproducibility of this study’s methodology. Additionally, it is not currently known whether the inflammation—depression association is categorical (i.e., only in individuals with clinical levels of inflammation) or continuous in nature. Including both group-comparisons and continuous statistical approaches maximizes this study’s relevance as understanding of this relationship evolves. Additionally, inclusion of both approaches prevents limitations germane to group-differences studies (e.g., the heterogeneity problem; Feczko et al., 2019) from applying to the entire study.
However, results should be considered in light of several limitations. First, although the diversity of the sample improves generalizability, there might be important demographic (e.g., sex, race, age) moderators of the characteristics tested in this study. Specifically, sex has been identified as a moderator of the relationship between inflammation and depression in several studies (Gimeno et al., 2009; Moriarity et al., 2019; Niles et al., 2018). Moderated network models involving two moderators (e.g., CRP and sex) have not yet been implemented and cannot be tested. However, this concern is somewhat ameliorated by controlling for variance associated with several potential confounds. Second, this dataset does not have repeated measures, so no inferences about directionality or how these associations influence course of illness can be made. Prospective studies are an important next step for this line of research; however, some of the cross-sectional metrics reported in this study (i.e., symptom connectivity, expected influence) have been demonstrated to have implications for the course of psychopathology, supporting the utility of these analyses. Third, three PHQ-9 items are double-barreled (changes in appetite, sleep problems, and psychomotor difficulties). This might obscure important differences in how CRP moderates depression symptom structure. For example, there is growing evidence that CRP is more consistently associated with increased, rather than decreased, appetite (Hickman et al., 2014; Lamers et al., 2018). Future research should include more precise symptom measures. Fourth, although CRP generally is considered a good indicator of general inflammation, it is only one of many inflammatory proteins associated with depression symptoms. Although CRP is one of, if not the, most important inflammatory protein to use to test the inflammatory phenotype of depression symptom structure given its status as a pan-inflammatory marker and prevalence in the literature, there are many other proteins not in this dataset (e.g., interleukin-6, tumor necrosis factor) that could, and should, be used in extensions of this work. Further, although 3.0 mg/L is an established cut-off for elevated CRP and is associated with negative outcomes (e.g., depression; Liukkonen et al., 2006), it is possible that another cut-off (e.g., 10 mg/L) better demarcates at what level CRP moderates the structure of depression symptoms. Fifth, although using a sample reporting the full range of depression severity buffers against Berkson’s bias (de Ron et al., 2019), there is the potential that results might not generalize to a clinical sample. Recent work testing the relationship between CRP and individual symptoms of depression found that results are largely replicable between clinical and community samples (Moriarity et al., 2020a), but the current study should be replicated in a sample with a higher rate of clinically-relevant depression to ensure clinical relevance of these results. Finally, there was a change in the chemistry analyzer during data collection, which could have confounded CRP measurement; however, this concern is ameliorated by comparisons of assay values finding that no statistical correction was necessary.
Conclusions
This study demonstrates a novel approach to characterizing an inflammatory phenotype of depression. Results suggest that structural characteristics at all levels (global, symptom-level, and specific edges) differed as a function of CRP. Specifically, greater symptom connectivity in the network of individuals with elevated CRP might explain one mechanism through which inflammation is associated with treatment-resistant depression. Additionally, greater expected influence of difficulty concentrating and psychomotor problems in the group with elevated CRP, and evidence that CRP moderates several symptom—symptom associations, might provide insight into differential symptom-level risk/maintenance pathways in those with elevated inflammation compared to those without this risk factor. At least, this project provides consistent evidence that depression symptom structure varies as a function of CRP, highlighting the potential for differential validity of extant depression measurement techniques based on inflammatory status. Given these implications, characterization of the inflammatory network phenotype of depression could be beneficial for disease classification, prevention, measurement, and treatment.
Supplementary Material
Elevated CRP was associated with symptom density, which predicts treatment resistant depression
Difficulty concentrating and psychomotor problems were more central to symptom structure in the group with CRP > 3.0 mg/L
Several symptom—symptom associations (particularly those involving changes in appetite) were moderated by CRP
Acknowledgments
We would like to thank Jonas Haslbeck for his help in using his mgm package, including several updates and a new feature to facilitate this work.
Funding: Daniel P. Moriarity was supported by National Research Service Award F31MH122116. Lauren B. Alloy was supported by National Institute of Mental Health R01MH101168.
Footnotes
Disclosures
The authors report no conflicts of interest.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- Biasucci LM, 2004. CDC/AHA Workshop on Markers of Inflammation and Cardiovascular Disease: Application to Clinical and Public Health Practice: clinical use of inflammatory markers in patients with cardiovascular diseases: a background paper. Circulation 110, 560–567. 10.1161/01.CIR.0000148983.88334.80 [DOI] [PubMed] [Google Scholar]
- Van Borkulo C, Boschloo L, Kossakowski J, Tio P, Schoevers R, Borsboom D, Waldorp L, 2016. Comparing network structures on three aspects: A permutation test. 10.13140/RG.2.2.29455.38569 [DOI] [PubMed] [Google Scholar]
- Borsboom D, Cramer AOJ, 2013. Network Analysis: An Integrative Approach to the Structure of Psychopathology. Annu. Rev. Clin. Psychol 9, 91–121. 10.1146/annurev-clinpsy-050212-185608 [DOI] [PubMed] [Google Scholar]
- Centers for Disease Control and Prevention, 2009. National Health and Nutrition Examination Survey (NHANES) stored biologic specimens: Guidelines for proposals to use samples and proposed cost schedule.
- Chen TC, Clark J, Riddles MK, Mohadjer LK, Fakhouri THI, 2018. Vital and Health Statistics National Health and Nutrition Examination Survey , 2015 – 2018 : Sample Design and Estimation Procedures. [PubMed] [Google Scholar]
- Czarkowska-paczek AWB, 2016. Inflammatory Markers Change with Age, but do not Fall Beyond Reported Normal Ranges. Arch. Immunol. Ther. Exp. (Warsz) 64, 249–254. 10.1007/s00005-015-0357-7 [DOI] [PMC free article] [PubMed] [Google Scholar]
- de Ron J, Fried EI, Epskamp S, 2019. Psychological networks in clinical populations: Investigating the consequences of Berkson’s bias. Psychol. Med 10.1017/S0033291719003209 [DOI] [PubMed] [Google Scholar]
- Elliott H, Jones PJ, Schmidt U, 2020. Central Symptoms Predict Posttreatment Outcomes and Clinical Impairment in Anorexia Nervosa: A Network Analysis. Clin. Psychol. Sci 8, 139–154. 10.1177/2167702619865958 [DOI] [Google Scholar]
- Epskamp S, Fried EI, 2018. A Tutorial on Regularized Partial Correlation Networks. Psychol. Methods 23, 617–634. [DOI] [PubMed] [Google Scholar]
- Feczko E, Miranda-dominguez O, Marr M, Graham AM, Nigg JT, Fair DA, 2019. The Heterogeneity Problem: Approaches to Identify Psychiatric Subtypes. Trends Cogn. Sci 1–18. 10.1016/j.tics.2019.03.009 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Felger JC, Haroon E, Miller AH, 2018. What’s CRP got to do with it? Tackling the complexities of the relationship between CRP and depression. Brain Behav. Immun 73, 163–164. 10.1016/j.bbi.2018.08.003 [DOI] [PubMed] [Google Scholar]
- Felger JC, Li Z, Haroon E, Woolwine BJ, Jung MY, Hu X, Miller AH, 2016. Inflammation is associated with decreased functional connectivity within corticostriatal reward circuitry in depression. Mol. Psychiatry 21, 1358–1365. 10.1038/mp.2015.168 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Fitzhugh KB, Fitzhugh LC, Reitan RM, 1962. Relation of acuteness of organic brain dysfunction to Trail Making Test performances. Percept. Mot. Skills 15, 399–403. 10.2466/pms.1962.15.2.399 [DOI] [PubMed] [Google Scholar]
- Fried EI, von Stockert S, Haslbeck JMB, Lamers F, Schoevers RA, Penninx BWJH, 2019. Using network analysis to examine links between individual depressive symptoms, inflammatory markers, and covariates. Psychol. Med 10.31234/osf.io/84ske [DOI] [PubMed] [Google Scholar]
- Gimeno D, Kivimäki M, Brunner EJ, EIovainio M, De Vogli R, Steptoe A, Kumari M, Lowe GDO, Rumley A, Marmot MG, Ferrie JE, Kivimaki M, Brunner EJ, Elovainio M, De Vogli R, Steptoe A, Kumari M, Lowe GDO, Rumley A, Marmot MG, Ferrie JE, 2009. Associations of C-reactive protein and interleukin-6 with cognitive symptoms of depression: 12-year follow-up of the Whitehall II study. Psychol. Med 39, 413–23. 10.1017/S0033291708003723 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Haapakoski R, Mathieu J, Ebmeier KP, Alenius H, Kivimäki M, 2015. Cumulative meta-analysis of interleukins 6 and 1β, tumour necrosis factor α and C-reactive protein in patients with major depressive disorder. Brain. Behav. Immun 49, 206–215. 10.1016/j.bbi.2015.06.001 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Haro JM, Hammer-Helmich L, Saragoussi D, Ettrup A, Larsen KG, 2019. Patient-reported depression severity and cognitive symptoms as determinants of functioning in patients with major depressive disorder: A secondary analysis of the 2-year prospective PERFORM study. Neuropsychiatr. Dis. Treat 15, 2313–2323. 10.2147/NDT.S206825 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Haroon E, Fleischer CC, Felger JC, Chen X, Woolwine BJ, Patel T, Hu XP, Miller AH, 2016. Conceptual convergence: increased inflammation is associated with increased basal ganglia glutamate in patients with major depression. Mol. Psychiatry 1–7. 10.1038/mp.2015.206 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Haslbeck JMB, Waldorp LJ, 2020. mgm: Estimating Time-Varying Mixed Graphical. J. Stat. Softw 93, 1–46. [Google Scholar]
- Hickman RJ, Khambaty T, Stewart JC, 2014. C-reactive protein is elevated in atypical but not nonatypical depression: Data from the National Health and Nutrition Examination Survey (NHANES) 1999-2004. J. Behav. Med 37, 621–629. 10.1007/s10865-013-9510-0 [DOI] [PubMed] [Google Scholar]
- Horn SR, Long MM, Nelson BW, Allen NB, Fisher PA, Byrne ML, 2018. Replication and reproducibility issues in the relationship between C-reactive protein and depression: A systematic review and focused meta-analysis. Brain Behav. Immun 73, 85–114. 10.1016/j.bbi.2018.06.016 [DOI] [PMC free article] [PubMed] [Google Scholar]
- IBM Corp, 2018. IBM SPSS Statistics for Windows, Version 26.0.
- Jokela M, Virtanen M, Batty GD, Kivimäki M, 2016. Inflammation and Specific Symptoms of Depression. JAMA Psychiatry 73, 87. 10.1001/jamapsychiatry.2015.1977 [DOI] [PubMed] [Google Scholar]
- Kappelmann N, Arloth J, Georgakis MK, Czamara D, Rost N, Ligthart S, Khandaker GM, Binder EB, 2020. Dissecting the association between inflammation, metabolic dysregulation, and specific depressive symptoms: A genetic correlation and 2-sample mendelian randomization study. JAMA Psychiatry. 10.1001/jamapsychiatry.2020.3436 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kasapis C, Thompson PD, 2005. The effects of physical activity on serum C-reactive protein and inflammatory markers: A systematic review. J. Am. Coll. Cardiol 45, 1563–1569. 10.1016/j.jacc.2004.12.077 [DOI] [PubMed] [Google Scholar]
- Khera A, Ms C, Mcguire DK, Mhs C, Murphy SA, Stanek HG, Das SR, Vongpatanasin W, Wians FH, Grundy SM, De Lemos JA, 2005. Race and Gender Differences in C-Reactive Protein Levels. J. Am. Coll. Cardiol 46, 464–469. 10.1016/j.jacc.2005.04.051 [DOI] [PubMed] [Google Scholar]
- Kim YK, Jung HG, Myint AM, Kim H, Park SH, 2007. Imbalance between pro-inflammatory and anti-inflammatory cytokines in bipolar disorder. J. Affect. Disord 104, 91–95. 10.1016/j.jad.2007.02.018 [DOI] [PubMed] [Google Scholar]
- Kroenke K, Spitzer RL, Williams JB, 2001. The PHQ-9: validity of a brief depression severity measure. J. Gen. Intern. Med 16, 606–613. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Krogh J, Benros ME, Jørgensen MB, Vesterager L, Elfving B, Nordentoft M, 2014. The association between depressive symptoms, cognitive function, and inflammation in major depression. Brain. Behav. Immun 35, 70–76. 10.1016/j.bbi.2013.08.014 [DOI] [PubMed] [Google Scholar]
- Kushner I, Rzewnicki D, Samols D, 2006. What does minor elevation of C-reactive protein signify? Am. J. Med 119, 166.e17–166.e28. 10.1016/j.amjmed.2005.06.057 [DOI] [PubMed] [Google Scholar]
- Lamers F, Milaneschi Y, De Jonge P, Giltay EJ, Penninx BWJH, 2018. Metabolic and inflammatory markers: Associations with individual depressive symptoms. Psychol. Med 48, 1102–1110. 10.1017/S0033291717002483 [DOI] [PubMed] [Google Scholar]
- Liu H, Lafferty J, Wasserman L, 2009. The nonparanormal: Semiparametric estimation of high dimensional undirected graphs. J. Mach. Learn. Res 10, 2295–2328. 10.1184/r1/6610712 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Liukkonen T, Silvennoinen-Kassinen S, Jokelainen J, Räsänen P, Leinonen M, Meyer-Rochow VB, Timonen M, 2006. The association between C-reactive protein levels and depression: Results from the Northern Finland 1966 Birth Cohort Study. Biol. Psychiatry 60, 825–830. 10.1016/j.biopsych.2006.02.016 [DOI] [PubMed] [Google Scholar]
- Loh P, Wainwright MJ, 2012. Structure estimation for discrete graphical models: Generalized covariance matrices and their inverses. Adv. Neural Inf. Process. Syst 2087–2095. 10.1214/13-AOS1162 [DOI] [Google Scholar]
- Lynall ME, Turner L, Bhatti J, Cavanagh J, de Boer P, Mondelli V, Jones D, Drevets WC, Cowen P, Harrison NA, Pariante CM, Pointon L, Clatworthy MR, Bullmore E, 2020. Peripheral Blood Cell–Stratified Subgroups of Inflamed Depression. Biol. Psychiatry 88, 185–196. 10.1016/j.biopsych.2019.11.017 [DOI] [PubMed] [Google Scholar]
- Mac Giollabhui N, Swistun D, Murray S, Moriarity D, Kautz M, Ellman L, Olino T, Coe C, Abramson L, Alloy L, 2020. Executive dysfunction in depression in adolescence: The role of inflammation and higher body mass. Psychol. Med 50, 683–691. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Majd M, Saunders EFH, Engeland CG, 2020. Inflammation and the dimensions of depression: A review. Front. Neuroendocrinol 56. 10.1016/j.yfrne.2019.100800 [DOI] [PubMed] [Google Scholar]
- Manea L, Gilbody S, McMillan D, 2012. Optimal cut-off score for diagnosing depression with the Patient Health Questionnaire (PHQ-9): a meta-analysis. Can. Med. Assoc. J 184, 191–196. 10.1503/cmaj.112004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Moriarity DP, Alloy LB, 2020. Beyond diagnoses and total symptom scores: Diversifying the level of analysis in psychoneuroimmunology research. Brain. Behav. Immun 89, 1–2. 10.1016/j.bbi.2020.07.002 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Moriarity DP, Horn SR, Kautz MM, Haslbeck, Alloy LB, 2020a. How handling extreme C-reactive protein (CRP) values and regularization influences CRP and depression criteria associations in network analyses. Brain. Behav. Immun 10.1016/j.bbi.2020.10.020 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Moriarity DP, Kautz MM, Mac Giollabhui N, Klugman J, Coe CL, Ellman LM, Abramson LY, Alloy LB, 2020b. Bidirectional associations between inflammatory biomarkers and depressive symptoms in adolescents: Potential causal relationships. Clin. Psychol. Sci 8, 690–703. 10.1017/CBO9781107415324.004 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Moriarity DP, Mac Giollabhui N, Ellman LM, Klugman J, Coe CL, Abramson LY, Alloy LB, 2019. Inflammatory proteins predict change in depressive symptoms in male and female adolescents. Clin. Psychol. Sci 7, 754–767. 10.1177/2167702619826586 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Niles AN, Smirnova M, Lin J, O’Donovan A, 2018. Gender differences in longitudinal relationships between depression and anxiety symptoms and inflammation in the health and retirement study. Psychoneuroendocrinology 95, 149–157. 10.1016/j.psyneuen.2018.05.035.Gender [DOI] [PMC free article] [PubMed] [Google Scholar]
- Park RJ, Kim YH, 2017. Association between high sensitivity CRP and suicidal ideation in the Korean general population. Eur. Neuropsychopharmacol 27, 885–891. 10.1016/j.euroneuro.2017.06.010 [DOI] [PubMed] [Google Scholar]
- Pearson TA, Mensah GA, Alexander RW, Anderson JL, Cannon RO, Criqui M, Fadl YY, Fortmann SP, Hong Y, Myers GL, Rifai N, Smith SC, Taubert K, Tracy RP, Vinicor F, 2003. Markers of inflammation and cardiovascular disease: Application to clinical and public health practice: A statement for healthcare professionals from the centers for disease control and prevention and the American Heart Association. Circulation 107, 499–511. 10.1161/01.CIR.0000052939.59093.45 [DOI] [PubMed] [Google Scholar]
- Raison CL, Miller AH, 2011. Is depression an inflammatory disorder? Current 13, 467–475. 10.1109/ICCEREC.2016.7814953 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sluzewska A, Sobieska M, Rybakowski JK, 1997. Changes in Acute-Phase Proteins during Lithium Potentiation of Antidepressants in Refractory Depression. Neuropsychobiology 35, 123–127. [DOI] [PubMed] [Google Scholar]
- van Borkulo C, Boschloo L, Borsboom D, Penninx BJH, Waldorp LJ, Schoevers RA, 2015. Association of Symptom Network Structure With the Course of Depression. JAMA Psychiatry 72, 1219–1226. 10.1001/jamapsychiatry.2015.2079 [DOI] [PubMed] [Google Scholar]
- White J, Kivimäki M, Jokela M, Batty GD, 2017. Association of inflammation with specific symptoms of depression in a general population of older people: The English Longitudinal Study of Ageing. Brain. Behav. Immun 61, 27–30. 10.1016/j.bbi.2016.08.012 [DOI] [PubMed] [Google Scholar]
- Williams DR, Rhemtulla M, Wysocki AC, Rast P, 2019. On Nonregularized Estimation of Psychological Networks. Multivariate Behav. Res 54, 719–750. 10.1080/00273171.2019.1575716 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Zipf G, Chiappa M, Porter KS, Ostchega Y, Lewis BG, Dostal J, 2013. National Health and Nutrition Examination Survey: Plan and Operations, 1999 – 2010. [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.