Toward High Reproducibility and Accountable Heterogeneity in Schizophrenia Research

Schizophrenia is a neuropsychiatric illness with substantial individual variability. The heterogeneity spans most aspects of the illness: genetics, environmental risk factors, age at onset, symptoms, treatment response, and long-term prognosis. The causative mechanisms of these heterogeneities have remained elusive from Bleuler’s definition of the disorder¹ to modern clinical and imaging studies. While variability may be because of distinct clinical or neurobiological subtypes, efforts to confirm this hypothesis have not yet succeeded. In fact, heterogeneity has prevented reproducible research on the effects of candidate genes and clinical, neuroanatomical, and functional findings. As a result, some call to retire the terms schizophrenia or schizoaffective disorder in favor of psychosis spectrum or psychosis syndrome, arguing that the former diagnoses incorrectly imply discrete illnesses.^2,3 Recent big data studies, such as those conducted by Alnæs et al⁴ in this issue of JAMA Psychiatry, provide a new way of examining individual heterogeneity and reproducibility of observations associated with this illness.

Meta-analysis has long offered a principled way to screen for false-positive findings and identify reproducible ones. Traditional meta-analyses focus on aggregating previously published mean effect sizes in group-level comparisons. By contrast, Alnæs et al⁴ focus on quantifying the sources of the patient-control differences in individual deviations from the mean. The authors hypothesize that individual deviations in brain structure may be influenced by genetic factors—specifically, those captured by variability in polygenic risk scores for schizophrenia. Genome-wide association studies indicate that schizophrenia is a highly heritable, polygenic disorder, whereby genetic risks are conferred by a large number of alleles with small effect sizes each. The Psychiatric Genomics Consortium has performed a schizophrenia genome-wide association study⁵ that revealed many disease-associated loci. Polygenic risk scores calculated using a genome-wide association study help establish an individual’s genetic risk better than any single candidate risk allele.⁶

Overall, the study by Alnæs et al⁴ showed that patients with schizophrenia had higher heterogeneity in brain structural measurements than control participants did. Ultimately, effect sizes of the patient-control differences in heterogeneity were much smaller than the effect size of the group difference effect (ie, t = 3.24 for heterogeneity and t = −17.05 for mean patient-control difference in cortical volume). Counter to the main hypothesis, the effects were not significantly associated with individual variances in polygenic risk scores. The replication analysis in 12 490 participants in the UK Biobank showed that polygenic risk scores were not associated with individual heterogeneity in healthy participants and were quite small compared with the mean effect size. With respect to the individual heterogeneity measure used by Alnæs et al, it is unclear if the approach is reproducible, given the smaller effect sizes. Would future studies require even more powerful samples?

It is not unusual that a group of patients with a complex illness exhibits more variances than a group of healthy individuals (eg, reported range and deviations of glucose levels in patients with diabetes are higher than those in healthy control participants). Investigators may recruit larger samples of patients than control participants to boost the statistical power of their research, in part to account for more variances in the patients. A general linear model and its derivative statistical methods are routinely used to probe individual variances, so it remains to be seen whether recalculating heterogeneity in the manner proposed by Alnæs et al⁴ provides additional advantages. Another potential limitation of the study is that, contrary to its hypothesis, the article did not explicitly probe the effects of the genotype-by-environment interactions.

This study provides an opportunity to reexamine the mean effects of patient-control differences and test the premise of improved reproducibility via big data neuroimaging research. The underlying methods used by the study were similar to those used in 2 prior studies: van Erp et al,⁷ which assessed 2028 patients and 2540 control participants from 15 centers worldwide, and Okada et al,⁸ which analyzed 884 patients and 1680 control participants from 15 cohorts assessed in Japan. We plotted the t score of the patient control differences in Alnæs et al⁴ vs the effect sizes of these 2 studies: the scatterplot of regional deficits from the 3 studies aligns with remarkable correlation at coefficients of approximately 0.98 (Figure). The studies by van Erp et al⁷ and Alnæs et al⁴ shared about 35% of the sample, while the sample by Okada et al⁸ was independent from both. Excellent consistency of schizophrenia-associated deficits was also reported in a multisite white matter study⁹ that replicated the regional pattern reported by another big data study¹⁰ at correlation coefficients of r equal to about 0.90. High reproducibility in imaging observations can be achieved using traditional mean-and-SD–based measures when large big data are aggregated.

Figure. Scatterplot and Linear Regression Analysis.

The scatterplot and linear regression analysis of t scores of patient-control differences on regional subcortical volumes reported by Alnæs et al⁴ vs van Erp et al⁷ and Okada et al.⁸ The sample used by Alnæs et al⁴ shared about 35% of its participants with the sample used by van Erp et al.⁷ The study by Okada et al⁸ used an independent sample. L indicates structures on the left side of the brain; R, structures on the right side of the brain.

Consistent replicability is uncommon in most patient-oriented measurements in schizophrenia, prompting the heterogeneity argument. Using aggregated neuroimaging, we are beginning to identify patterns of group differences at the level of approximately 90% replicability, with heterogeneity accounted for within the traditional statistical framework. This replicability was not achieved by comparing the measurements for each regional structure independently, but rather by evaluating all traits simultaneously. The Figure plots suggest a pattern of neuroanatomical deficits that separate patients from control participants. Meta-analytic aggregation has removed locally introduced heterogeneity, yielding schizophrenia-associated neurobiology shared across patients worldwide. High heterogeneity in patients with schizophrenia is well documented, and the analyses performed by Alnæs et al⁴ underscore the need to quantify heterogeneity on an individual level. Research using large or aggregate samples now yield both high reproducibility and reveal robust trends in regional abnormality patterns in schizophrenia.