Predictive Performance of Exposome Score for Schizophrenia in the General Population

Abstract

Previously, we established an estimated exposome score for schizophrenia (ES-SCZ) as a cumulative measure of environmental liability for schizophrenia to use in gene–environment interaction studies and for risk stratification in population cohorts. Hereby, we examined the discriminative function of ES-SCZ for identifying individuals diagnosed with schizophrenia spectrum disorder in the general population by measuring the area under the receiver operating characteristic curve (AUC). Furthermore, we compared this ES-SCZ method to an environmental sum score (Esum-SCZ) and an aggregate environmental score weighted by the meta-analytical estimates (Emet-SCZ). We also estimated ORs and Nagelkerke’s R² for ES-SCZ in association with psychiatric diagnoses and other medical outcomes. ES-SCZ showed a good discriminative function (AUC = 0.84) and statistically significantly performed better than both Esum-SCZ (AUC = 0.80) and Emet-SCZ (AUC = 0.80). At optimal cut point, ES-SCZ showed similar performance in ruling out (LR− = 0.20) and ruling in (LR+ = 3.86) schizophrenia. ES-SCZ at optimal cut point showed also a progressively greater magnitude of association with increasing psychosis risk strata. Among all clinical outcomes, ES-SCZ was associated with schizophrenia diagnosis with the highest OR (2.76, P < .001) and greatest explained variance (R² = 14.03%), followed by bipolar disorder (OR = 2.61, P < .001, R² = 13.01%) and suicide plan (OR = 2.44, P < .001, R² = 12.44%). Our findings from an epidemiologically representative general population cohort demonstrate that an aggregate environmental exposure score for schizophrenia constructed using a predictive modeling approach—ES-SCZ—has the potential to improve risk prediction and stratification for research purposes and may help gain insight into the multicausal etiology of psychopathology.

Introduction

Schizophrenia spectrum disorder is a heterogeneous phenotype with a complex pathoetiology that involves a multitude of genetic and environmental risk factors and their interaction.¹ The progress of genome-wide association studies (GWAS) has paved the way for polygenic risk estimation of schizophrenia (polygenic risk score [PRS]: a weighted sum of trait-associated alleles) to measure molecular genetic liability as a single metric.² The recent release of the Psychiatric Genomics Consortium (PGC3) shows that PRS for schizophrenia (PRS-SCZ) explains up to 7.7% of variation on the liability scale to schizophrenia.³ Recent studies investigating the electronic health records in the United States showed that PRS-SCZ was associated with schizophrenia diagnosis in the population.^4,5 Of PRS for mental disorder phenotypes that have been approximated thus far, PRS-SCZ seems to be the most outstanding for testing PRS-based prediction for population risk stratification and clinical applications. However, PRS-SCZ is not distinctly associated with schizophrenia but is also associated with several psychiatric and other medical conditions^4,6,7 and subclinical multidimensional phenotypes,^7–10 as well as general mental and physical health outcomes.^11,12

Schizophrenia spectrum disorder, similar to its polygenic composition, has been associated with several environmental exposures, including cannabis use, childhood adversities (eg, sexual abuse, peer-bullying, and emotional neglect), obstetric and pregnancy complications, proxies of social exclusion (eg, ethnic minority and hearing impairment immigration), and season of birth (winter birth).¹³ These environmental factors are often correlated to a degree and comprise a network at population level: the exposome.^14,15 To supplement PRS-SCZ in gene–environment interaction studies and risk stratification in population cohorts, we have estimated an exposome score for schizophrenia (ES-SCZ) as a cumulative measure of environmental liability for schizophrenia.¹⁶ Recently, we have demonstrated that the ES-SCZ is associated with psychosis risk states,¹⁷ as well as mental and physical health in the general population.¹²

In the present study, we aimed to examine the discriminative function of ES-SCZ for identifying individuals diagnosed with schizophrenia spectrum disorder in the general population by measuring the area under the receiver operating characteristic (AUC), sensitivity, specificity, and positive and negative likelihood ratios. We also estimated ORs and Nagelkerke’s R² for ES-SCZ in association with psychiatric diagnoses and other medical outcomes.

Methods

Study Population

The Netherlands Mental Health Survey and Incidence Study-2 (NEMESIS-2) was conducted to study the prevalence, incidence, course, and consequences of mental disorders in the Dutch general population. The study was approved by the Medical Ethics Review Committee for Institutions on Mental Health Care and written informed consent was collected from participants at each wave. A multistage random sampling procedure was applied to ensure the representativeness of the sample in terms of age (between the ages of 18 and 65 at baseline), region, and population density. Dutch illiteracy was an exclusion criterion. Details of NEMESIS-2 were provided elsewhere.^18,19 From 2007 to 2009, the first wave (T0) enrolled 6646 participants (response rate 65.1%; average interview duration: 95 min) who were followed up at 3 visits within 9 years: successive response rates at year 3 (T1), year 6 (T2), and year 9 (T3) were 80.4% (n = 5303; excluding those who deceased; interview duration: 84 min), 87.8% (n = 4618; interview duration: 83 min), and 86.8% (n = 4007; interview duration: 102 min), respectively. Rates at baseline reflect lifetime occurrence; rates at T1–T3 reflect 3-year interval (T0–T1, T1–T2, and T2–T3) occurrence. Attrition between T0 and T3 was not significantly associated with any of the individual 12-month mental disorders at T0 after controlling for sociodemographic characteristics.²⁰ For this cross-sectional analysis, data from baseline (n = 6646) were utilized.

Exposome Score

The exposome score in the current analyses was calculated based on our previously validated estimates¹⁶ for constructing cumulative environmental load. Using the log odds from our previous report, we generated the ES-SCZ by summing log-odds-weighted environmental exposures (cannabis use, winter birth, hearing impairment, and childhood adversities [emotional neglect, psychological abuse, physical abuse, sexual abuse, and peer victimization]) at baseline (supplementary methods and supplementary table 1). For comparison, an environmental sum score (hereafter, Esum-SCZ) by adding each binary exposure per individual as 0 = absent and 1 = present (ranging from 0 to 8) and an aggregate environmental score weighted by the meta-analytical estimates for each exposure (hereafter, Emet-SCZ) conforming to a previous study were generated.²¹

Outcomes

Nonclinician, trained interviewers applied the Composite International Diagnostic Interview version 3.0.^22,23 To examine the discriminative function of ES-SCZ for identifying individuals diagnosed with schizophrenia spectrum disorder, lifetime Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition (DSM-IV) diagnosis of schizophrenia at baseline was used. To examine the association of ES-SCZ with clinical risk strata, we used psychosis risk strata that were previously defined based on the degree of positive psychotic symptomatology, help-seeking attempt, antipsychotic treatment, and service use and admission for psychotic symptomatology. Psychosis risk strata consisted of 5 distinct categories (no-risk, low-risk, moderate-risk, high-risk, and clinical psychosis). For further details, see elsewhere.¹⁷ For the outcome-wide association of ES-SCZ, we used lifetime diagnoses of DSM-IV disorders, lifetime suicide thoughts, plans, and attempts, self-reported chronic somatic health problems in the last 12 months, and general traits of neuroticism and extraversion as listed (supplementary method and supplementary table 1).

Statistical Analyses

All analyses were performed using Stata, version 16.²⁴P < .05 (2 tailed) was considered nominally statistically significant. To determine the discriminative function of ES-SCZ for identifying individuals diagnosed with schizophrenia, receiver operating characteristic (ROC) analysis was performed using the ROCREG command²⁵ that applies a nonparametric estimator of the 95% CIs around the AUC using a bootstrap method (n = 1000 repetitions). The ROCCOMP command was used to compare the ROC areas of ES-SCZ, Esum-SCZ, and Emet-SCZ.²⁶ By applying the ROCTAB command, the optimal cut point for ES-SCZ was estimated using the Liu method that maximizes the product of sensitivity and specificity.²⁷ The sensitivity, specificity, correct classification rate, and positive (LR+) and negative likelihood ratios (LR–) were reported. Given that the covariates may influence the discriminative function of ES-SCZ, we performed ROC analysis controlled for the covariates (age, sex, and education) using the CTRLCOV option.²⁸ We performed multinomial logistic regression models using the MLOGIT command to analyze the association of ES-SCZ at the optimal cut point with psychosis risk strata (“no-risk” group as the reference). Finally, we applied logistic regression models to test the association of ES-SCZ with psychiatric diagnoses and other medical outcomes. Unadjusted OR with corresponding 95% CI and Nagelkerke’s R² of ES-SCZ were reported for each model. Each model was also controlled for covariates (age, sex, and education).

Results

Baseline frequencies of demographic variables, exposures, outcome variables, and missing values of the total sample are shown in the supplementary table 1. At baseline, 43 individuals (0.7%) were diagnosed with a lifetime schizophrenia spectrum disorder.

Predictive Performance of the ES-SCZ

Distinguishing individuals diagnosed with schizophrenia spectrum disorder from controls, the unadjusted ROC analyses indicated the highest AUC of 0.84 (95% CI: 0.77; 0.91) for the ES-SCZ, whereas AUCs were similar for Emet-SCZ with 0.80 (95% CI: 0.73; 0.87) and for Esum-SCZ with 0.80 (95% CI: 0.72; 0.87). AUCs comparisons demonstrated a significant difference between ES-SCZ and Emet-SCZ (x² = 7.29, P = .007), as well as between ES-SCZ and Esum-SCZ (x² = 6.66, P = .010). No significant difference was found between Emet-SCZ and Esum-SCZ (x² = 0.14, P = .711). For visualization, covariate-adjusted ROC of ES-SCZ against a reference line is shown in figure 1.

Figure 1.

Receiver operating characteristic of ES-SCZ on schizophrenia spectrum disorder. Adjusted for sex, age (continuous), and 4-level education (1—primary school, 2—lower secondary education, 3—higher secondary education, and 4—higher professional education). Dashed line indicates the reference line.

Table 1 reports the sensitivity, specificity, likelihood ratios, and classification accuracy at the >50%, >75%, and the optimal cut point for the ES-SCZ. The models indicated higher sensitivity (84.62%–89.74%) relative to specificity (43.73%–78.06%), with the classification accuracy varying between 44.04% and 78.10%. At the optimal cut point (3.22), ES-SCZ showed high sensitivity (84.62%) and specificity (78.06), with 78.10% correct classification. The positive and negative likelihood ratios were 3.86 and 0.20, respectively.

Table 1.

Predictive ability of exposome score for schizophrenia on schizophrenia spectrum disorder at different cut points

	Cut points	Sensitivity %	Specificity %	Positive likelihood ratio	Negative likelihood ratio	Correctly classified %
<50%	2.03	89.74	43.73	1.59	0.23	44.04
<75%	2.86	84.62	73.67	3.21	0.21	73.74
Optimal cut point	3.22	84.62	78.06	3.86	0.20	78.10

Association Between ES-SCZ and Psychosis Risk Strata

ES-SCZ at the optimal cut point was significantly associated with the low-risk, moderate-risk, high-risk, and clinical psychosis strata (table 2). Additional post hoc group comparisons across strata showed significant differences for the low-risk vs moderate-risk, low-risk vs high-risk, low-risk vs clinical psychosis, moderate-risk vs high-risk, and moderate vs clinical psychosis. The comparison between high-risk vs clinical psychosis was not significantly different. The results were similar in the covariate-adjusted analyses (supplementary table 2).

Table 2.

Associations between exposome score for schizophrenia and psychosis risk strata

	Reference group (“no-risk”)			Psychosis low-risk state		Psychosis moderate-risk state		Psychosis high-risk state
	RRR	95% CI	P	Wald χ 2	P	Wald χ 2	P	Wald χ 2	P
Psychosis low-risk state	1.53	1.23–1.90	<.001	—	—	—	—	—	—
Psychosis moderate-risk state	2.79	2.17–3.89	<.001	13.64	<.001	—	—	—	—
Psychosis high-risk sate	4.06	3.15–5.23	<.001	35.18	<.001	4.52	.033	—	—
Clinical psychosis	7.27	3.58–14.73	<.001	17.34	<.001	6.35	.012	2.35	.125

Note: Reference group = 84.1%; psychosis low-risk state = 6.9%; psychosis moderate-risk state = 4.3%; psychosis high-risk state = 4.1%; clinical psychosis = 0.5%.

RRR, relative risk ratio.

Association Between ES-SCZ and Multiple Outcomes

In the univariate analyses, ES-SCZ was significantly (Bonferroni corrected P < .05/33) associated with 23 out of 33 outcomes (figure 2), while the multivariate analyses indicated that ES-SCZ was significantly associated with 25 outcomes (supplementary figure 1). Table 3 shows the outcome-wide association of ES-SCZ. Unadjusted significant ORs varied between 0.89 and 2.76, with the explained variance varying between 0.28% and 14.03%. The association between ES-SCZ and schizophrenia spectrum disorder indicated the highest OR (2.76 [95% CI: 2.20; 3.46], P < .001) with an explained variance of 14.03%. This was followed by bipolar disorder (OR = 2.61 [95% CI: 2.19; 3.10], P < .001, R² = 13.01%), suicide plan (OR = 2.44 [95% CI: 2.16; 2.75], P < .001, R² = 12.44%), suicidal thoughts (OR = 2.39 [95% CI: 2.19; 2.60], P < .001, R² = 13.58%), and suicide attempt (OR = 2.24 [95% CI: 1.95; 2.57], P < .001, R² = 9.46%). The analyses adjusted for covariates showed similar results for the top 5 associations (supplementary table 3): schizophrenia spectrum disorder (OR = 2.71 [95% CI: 2.16; 3.40], P < .001, R² = 15.87%), bipolar disorder (OR = 2.59 [95% CI: 2.17; 3.09], P < .001, R² = 14.69%), suicide plan (OR = 2.46 [95% CI: 2.17; 2.78], P < .001, R² = 12.88%), suicidal thoughts (OR = 2.41 [95% CI: 2.21; 2.64], P < .001, R² = 14.08%), and suicide attempt (OR = 2.24 [95% CI: 1.95; 2.57], P < .001, R² = 10.56%).

Figure 2.

Unadjusted variances and ORs of the association between ES-SCZ and multiple mental and physical health outcomes. The figure shows 23 significant associations after Bonferroni correction (P < .05/33); COPD, Chronic obstructive pulmonary disease; ES-SCZ: Exposome score for schizophrenia; R², Nagelkerke’s R².

Table 3.

Unadjusted outcome-wide association of exposome score for schizophrenia

Outcome variable	OR	95% CI	P	R 2
Extraversion	0.89	0.83–0.95	<.001	0.28
Poor eyesight	0.90	0.57–1.44	.666	0.05
High blood pressure	0.98	0.88–1.08	.652	0.01
Cancer	1.12	0.85–1.48	.429	0.09
Diabetes	1.13	0.97–1.32	.108	0.15
Thyroid abnormality	1.15	0.98–1.35	.097	0.18
Minor depressive disorder	1.15	0.98–1.35	.081	0.19
Heart disease	1.20	0.96–1.49	.105	0.27
Joint wear	1.20	1.06–1.36	.004	0.40
Back pain or hernia	1.25	1.11–1.40	<.001	0.61
Joint inflammation	1.26	1.09–1.46	.002	0.57
Alcohol abuse	1.32	1.22–1.43	<.001	1.31
COPD	1.35	1.17–1.57	<.001	1.01
Other physical problems	1.38	1.26–1.51	<.001	1.58
Migraine	1.39	1.23–1.58	<.001	1.34
Asthma	1.42	1.25–1.61	<.001	1.52
Intestinal disorder	1.49	1.23–1.80	<.001	1.62
Antisocial personality disorder	1.51	1.29–1.77	<.001	1.89
Ulcers	1.52	1.14–2.02	.005	1.51
Specific phobia	1.59	1.46–1.74	<.001	3.52
Panic disorder	1.59	1.42–1.79	<.001	2.93
Neuroticism	1.73	1.62–1.85	<.001	6.39
Generalized anxiety disorder	1.79	1.61–1.99	<.001	5.11
Major depressive disorder	1.80	1.68–1.93	<.001	7.03
Dysthymia	1.95	1.62–2.35	<.001	5.37
Alcohol dependence	1.95	1.67–2.29	<.001	5.77
Agoraphobia	1.99	1.69–2.34	<.001	6.04
Social phobia	2.01	1.85–2.18	<.001	8.79
Suicide attempt	2.24	1.95–2.57	<.001	9.46
Suicidal thoughts	2.39	2.19–2.60	<.001	13.58
Suicide plan	2.44	2.16–2.75	<.001	12.44
Bipolar disorder	2.61	2.19–3.10	<.001	13.01
Schizophrenia spectrum disorder	2.76	2.20–3.46	<.001	14.03

23 significant associations after Bonferroni correction (P < .05/33);

COPD, chronic obstructive pulmonary disease; R², Nagelkerke’s R².

Discussion

In this study, we investigated the discriminative capacity and risk stratification properties of ES-SCZ in the general population. Our findings were that ES-SCZ showed a good discriminative function (AUC: 84) for identifying schizophrenia in the general population. The AUC comparison showed that ES-SCZ significantly performed better than both the Esum-SCZ (AUC: 80) and Emet-SCZ (AUC: 80). At optimal cut point, ES-SCZ showed similar performance in ruling out (LR− = 0.20) and ruling in (LR+ = 3.86) schizophrenia. ES-SCZ at optimal cut point showed a progressively greater magnitude of association with increasing psychosis risk strata.

This is the first study investigating the discriminative function of an aggregate environment risk score for schizophrenia, the ES-SCZ, which is generated using a training sample to estimate the sum of the weighted effect sizes of environmental exposures. Unlike other previous approaches, the ES-SCZ uses estimates from the multivariate model that takes into account the interdependency of exposures. ES-SCZ showed significantly better discriminative function than the Esum-SCZ and Emet-SCZ. This finding provides further support that approaches that take into account the correlation between exposures prevent overestimation of the weights per exposure and achieve better predictive performance than those assuming independence (eg, simple summation of exposures or weighted estimates of individuals exposures from meta-analyses).¹⁶ The finding should be anticipated given the fact that environmental risk factors for mental disorder phenotypes, such as schizophrenia, are often linked with each other through causal and noncausal paths observed in the general population.¹⁴

Our findings showed that although AUC results are generally considered very good for values between 0.8 and 0.9, ES-SCZ at the optimal cut point generated small to moderate changes in probability for predicting schizophrenia. There have been no comparable studies for ES-SCZ; according to a proposed guideline for a clinically relevant risk prediction, the LRs should be optimally over 10 for LR+ and under 0.1 for LR− for decisive shifts from pretest to posttest probability.²⁹ Therefore, ES-SCZ cannot provide the risk prediction utility that is required for predicting individual diagnosis in the general population. However, as an environmental liability index for schizophrenia, ES-SCZ may offer improved solutions in research settings. First, ES-SCZ can be particularly useful for risk stratification in large population data sets as evidenced by our present findings showing a progressively greater magnitude of the association between increasing psychosis risk strata and ES-SCZ at the optimal cut point. Second, ES-SCZ may be useful for risk enrichment to target selective smaller samples for expensive, experimental, or time-consuming methods. Finally, the quantification of environmental liability as a single metric enhances statistical power over multiple testing of each exposure.^12,30 Certainly, the integration of ES-SCZ with electronic health records and molecular genetic markers, such as PRS-SCZ, has the potential to boost prediction in the future.

ES-SCZ was associated with schizophrenia diagnosis with the highest OR (2.76) and greatest explained variance (Nagelkerke R² = 14%) among all outcomes. However, in line with converging evidence suggesting that environmental exposures are not distinctly associated with psychosis spectrum phenotype only but instead are more universally related to broad psychopathology,^12,31,32 ES-SCZ was also associated with several psychiatric diagnoses and other medical outcomes in the general population. It should be noted that results on physical and mental health outcomes may not be directly comparable as mental disorders reflect lifetime diagnoses, whereas physical health outcomes reflect the previous 12-month period.

Pleiotropy is a rule rather than an exception for psychiatric diagnoses and behavioral phenotypes as also demonstrated in GWAS.³³ Similar to PRS-SCZ, ES-SCZ can provide only little to no benefit in discriminating schizophrenia from another adjacent diagnostic category, such as bipolar disorder. However, it may be used for risk stratification of broader mental ill health in the general population.

A major strength of our approach was constructing ES-SCZ in independent training and validation case-control samples and consequently testing ES-SCZ in a large population data set derived from the same country of origin with matching environmental assessment. However, several limitations should be noted. Although the random sampling procedure applied in this data set increases epidemiological representativeness, individuals who refrained from participating in this study (eg, because of trust issues) may be slightly underrepresented. Additionally, ES-SCZ is inherently limited to the exposure estimates derived from the original model using reliably measured and equally available exposures in the training and the validation case-control samples. Several other exposures associated with schizophrenia can be considered for addition to ES-SCZ. However, some of these exposures are largely unavailable in collected or ongoing cohort studies, such as obstetric and pregnancy complications, which are also extremely difficult to reliably assess without detailed birth records and maternal interviews and, therefore, impossible to collect in retrospect. Furthermore, some of these exposures, such as urbanicity, do not display a consistent association with psychosis phenotypes across countries and ethnic groups.³⁴ Also, given the lower predictive performance of genetic scores due to limited population diversity in GWAS,³⁵ the addition of some exposures, such as ethnic minority and migration, may decrease the utility of ES-SCZ when integrated with genetic data. Although generating a universal environmental loading score is extremely challenging, increasing efforts to the harmonization of available samples and determining a limited set of measures for core environmental assessment for epidemiological cohorts will pave the way for wider application.^15,36

In conclusion, our findings from an epidemiologically representative general population cohort demonstrate that an aggregate environmental exposure score for schizophrenia constructed using a predictive modeling approach—ES-SCZ—has the potential to improve risk prediction and stratification for research purposes and may help gain insight into multicausal etiology of pluripotent psychopathology.

Funding

Financial support for this study has been received from the Ministry of Health, Welfare and Sport, The Netherlands, with supplementary support from the Netherlands Organization for Health Research and Development. The study was also supported by a grant agreement HEALTH-F2-2009–241909 (Project EU-GEI) from the European Community’s Seventh Framework Program. B.P.F.R. was funded by a VIDI award number 91718336 from the Netherlands Scientific Organization. S.G. and J.v.O. were supported by the Ophelia research project, Netherlands Organization for Health Research and Development grant number 636340001.

Data Availability

Data available on reasonable request from the authors.