Skip to main content
Human Brain Mapping logoLink to Human Brain Mapping
. 2017 Oct 28;39(1):575–587. doi: 10.1002/hbm.23867

Individualized prediction of schizophrenia based on the whole‐brain pattern of altered white matter tract integrity

Yu‐Jen Chen 1, Chih‐Min Liu 2,3, Yung‐Chin Hsu 1, Yu‐Chun Lo 1,4, Tzung‐Jeng Hwang 2,3, Hai‐Gwo Hwu 2,3, Yi‐Tin Lin 2, Wen‐Yih Isaac Tseng 1,3,5,6,
PMCID: PMC6866439  PMID: 29080229

Abstract

Background

A schizophrenia diagnosis relies on characteristic symptoms identified by trained physicians, and is thus prone to subjectivity. This study developed a procedure for the individualized prediction of schizophrenia based on whole‐brain patterns of altered white matter tract integrity.

Methods

The study comprised training (108 patients and 144 controls) and testing (60 patients and 60 controls) groups. Male and female participants were comparable in each group and were analyzed separately. All participants underwent diffusion spectrum imaging of the head, and the data were analyzed using the tract‐based automatic analysis method to generate a standardized two‐dimensional array of white matter tract integrity, called the connectogram. Unique patterns in the connectogram that most accurately identified schizophrenia were systematically reviewed in the training group. Then, the diagnostic performance of the patterns was individually verified in the testing group by using receiver‐operating characteristic curve analysis.

Results

The performance was high in men (accuracy = 0.85) and satisfactory in women (accuracy = 0.75). In men, the pattern was located in discrete fiber tracts, as has been consistently reported in the literature; by contrast, the pattern was widespread over all tracts in women. These distinct patterns suggest that there is a higher variability in the microstructural alterations in female patients than in male patients.

Conclusions

The individualized prediction of schizophrenia is feasible based on the different whole‐brain patterns of tract integrity. The optimal masks and their corresponding regions in the fiber tracts could serve as potential imaging biomarkers for schizophrenia. Hum Brain Mapp 39:575–587, 2018. © 2017 Wiley Periodicals, Inc.

Keywords: diffusion magnetic resonance imaging, schizophrenia, individualized prediction, white matter tracts, tract‐based automatic analysis, diffusion spectrum imaging

INTRODUCTION

Schizophrenia, a complex mental disorder, is characterized by distortions in thinking, perception, language, emotions, and behaviors [World Health Organization, 1992]. At present, a schizophrenia diagnosis relies solely on identifying characteristic symptoms and behaviors through interviews conducted by trained physicians. However, substantial heterogeneity in symptom manifestation, indistinct symptomatic differentiation from other mental disorders, and subjectivity involved in the interviews indicated that a more objective assessment of schizophrenia is warranted [Zarogianni et al., 2013].

Recent progress in neuroscience research has provided evidence that schizophrenia is a dysconnectivity syndrome; in other words, it is a disorder arising from abnormal neuronal connectivity, which leads to associated symptoms and cognitive impairments (Stephan et al., 2009). Because the white matter comprises numerous neuronal axons and forms structural connections between brain regions, alterations in white matter integrity has been a key focus of schizophrenia research (Chiappelli et al., 2015; Kubicki et al., 2005; Oh et al., 2009). Diffusion tensor imaging (DTI) is useful for probing the microstructural properties of white matter (Alexander et al., 2007; Basser et al., 1994; Basser and Pierpaoli, 1996) and has been widely used to investigate alterations in white matter integrity in patients with schizophrenia (Fujino et al., 2014; Pettersson‐Yeo et al., 2011; Roalf et al., 2013; Scheel et al., 2013; Walther et al., 2011). One of the most common indices, fractional anisotropy (FA), represents the degree of spatial coherence among microstructures (Basser and Pierpaoli, 1996), and it changes when the microstructure undergoes alterations, such as axonal degeneration, demyelination, or inflammation (Alexander et al., 2007; Basser et al., 1994). Decreased FA within prefrontal and temporal lobes and abnormalities in the fiber bundles connecting these regions are the most commonly reported findings in schizophrenia research (Fujino et al., 2014; Pettersson‐Yeo et al., 2011; Roalf et al., 2013; Scheel et al., 2013; Walther et al., 2011), which supports the notion that schizophrenia is a dysconnectivity syndrome involving the prefrontal and temporal lobes (Harrison, 1999). In addition, studies have reported that schizophrenia is associated with decreased FA in the parietal and occipital cortices, suggesting widespread white matter alterations in patients with schizophrenia (Fujino et al., 2014; Roalf et al., 2013; Scheel et al., 2013; Walther et al., 2011).

Despite extensive results from magnetic resonance imagining (MRI) studies, a limited number of studies have developed individualized diagnosis procedures for schizophrenia on the basis of structural alterations. One reason may be the widespread alterations of brain structures in schizophrenia, which preclude the use of a univariate statistical approach (Ardekani et al., 2011; Caan et al., 2006; Caprihan et al., 2008; Kambeitz et al., 2015). Recognizing that schizophrenia is associated with a distributed localization of alterations, several studies have adopted multivariate statistical approaches to distinguish patients with schizophrenia from healthy people on an individual level (Kambeitz et al., 2015). Typically, quantitative values of brain structures at each region of interest (ROI) or image voxel are selected as features. The features then undergo dimension reduction, such as by using principal component analysis (PCA), to determine which features are most relevant to the disease. Finally, the selected components are trained by a classifier, such as the linear discriminant classifier or support vector machine, to achieve an optimal function that best differentiates schizophrenic brains from healthy brains.

In addition to cortical thickness, some DTI‐derived indices such as FA and mean diffusivity (MD) have been used as features. Using PCA and linear discrimination analysis (LDA) to automatically extract the combination of relevant brain regions on FA maps, Caan et al. (2006) reported a classification accuracy of 75% in a cohort of 34 patients with schizophrenia and 24 controls. Elsewhere, Caprihan et al. (2008) modified the PCA method to maximize the Mahalanobis distance between two comparison groups, and used Fisher's linear discriminant (FLD) analysis as a classifier. These researchers achieved a classification accuracy of 80% in a sample of 45 patients and 45 healthy volunteers. By applying PCA and FLD analysis on MD maps, Ardekani et al. (2011) achieved a classification accuracy of 98% in a sample of 50 patients and 50 healthy controls.

However, although multivariate statistical approaches appear to be able to identify patients with schizophrenia to a classification accuracy of 75%–98% at an individual level, it is difficult or even impossible to localize the resulting function computed from the classifier back to image space. This limitation prevents us from relating the predictive function to the structural attributes of the disease, which would provide notable insight into the altered connectivity of schizophrenic brains (Wolfers et al., 2015). To retain the connectional attributes of the features for the individualized prediction of schizophrenia, we proposed resampling the diffusion indices along the course of major fiber tracts. This resampling procedure is helpful in two ways. First, it extracts the white matter pixels located in the major fiber pathways of the brain, and filters out unrelated pixels. Second, the extracted pixels are reorganized according to their location in the “tract space,” which ensures that pixels in the same fiber tracts are connected as neighbors despite physical distance. We hypothesized that the features following this resampling procedure could bypass machine learning algorithms, and be used directly to predict schizophrenia at the individual level.

We recently established an automatic tract‐specific analysis of diffusion MRI, called tract‐based automatic analysis (TBAA), to resample diffusion indices in the “tract space” (Chen et al., 2015). The TBAA method was developed based on three key factors: a diffusion spectrum imaging (DSI) template; a tract atlas of 76 major fiber tract bundles segmented in the template; and nonlinear transformation, which registers individual diffusion data sets to the template (Hsu et al., 2012). TBAA automatically assesses the 76 fiber tracts covering the whole brain and provides a standardized output for each data set. The standardized output of TBAA is a two‐dimensional (2D) matrix, called the 2D connectogram. The connectogram contains 76 tracts (rows) and 100 steps (columns) in each tract, which represent the information that travels along each fiber pathway from beginning to end. This connectogram enables us to observe the global pattern of white matter alterations in patients with schizophrenia and determine the pattern that most accurately differentiates patients with schizophrenia from healthy controls.

The aim of this study was to identify the unique pattern of white matter alterations that yielded the most desirable performance of individualized schizophrenia prediction. We selected patients who had been diagnosed with schizophrenia at least 2 years before the recruitment period, because there was sufficient observation time to confirm the diagnosis of schizophrenia among such patients. Because sex‐associated differences in brain alterations were implicated based on cognitive domains, including immediate and delayed memory in schizophrenia (Han et al., 2012), patients were stratified according to sex for the analysis of white matter alteration patterns.

METHODS AND MATERIALS

Participants

From January 2010 to April 2016, we recruited 176 patients with schizophrenia and 207 age‐matched healthy controls. Eight patient data sets and three control data sets were excluded during the image quality assurance procedures (refer to the section “Image Acquisition”). In total, 168 patients and 204 controls were analyzed.

Our study patients met the criteria listed in the Diagnostic and Statistical Manual of Mental Disorders, Fourth Edition, for schizophrenia, as confirmed by attending psychiatrists at National Taiwan University Hospital. All patients were medicated and had been diagnosed as having schizophrenia for at least 2 years. The healthy controls were interviewed using the Diagnostic Interview for Genetic Studies (Nurnberger et al., 1994). All the healthy controls had no lifetime history of any psychiatric disorders. Additionally, all the participants were Han Chinese; right‐handed; and free of neurological disorders, major systemic illness, substance abuse problems, or intellectual disability. The institutional review board of National Taiwan University Hospital approved the study, and all participants provided informed consent before the study began.

Image Acquisition

Images were acquired on a 3 T MRI system (TIM Trio, Siemens, Erlangen, Germany) with a 32‐channel phased‐array head coil. T1‐weighted imaging was performed using a three‐dimensional (3D) magnetization‐prepared rapid gradient echo sequence (repetition time [TR]/echo time [TE] = 2000 ms/3 ms, flip angle = 9°, field of view [FOV] = 256 × 192 × 208 mm3, and matrix = 256 × 192 × 208). DSI was performed using a single‐shot spin‐echo planar imaging sequence with twice‐refocused diffusion‐sensitive gradients to reduce distortions induced by the eddy current (Reese et al., 2003). The sequence comprised 102 diffusion‐encoding directions and a maximal b value of 4000 s/mm−2 (Kuo et al., 2008): TR/TE = 9600/130 ms, FOV = 200 mm, matrix = 80 × 80, slice thickness = 2.5 mm, and slice number = 54. All DSI data sets in the study underwent image quality assurance for the head motion; this was done by assessing the number of images with signal dropout in the DSI data sets (Yendiki et al., 2014). Altogether, the acquired DSI data sets comprised 5508 images, which were subsequently scrutinized by calculating the signal dropout counts. DSI data sets with more than 90 signal dropout images were discarded because they considerably skewed the diffusion measures, with a percentage error of ∼6%.

Reconstruction of the Diffusion Spectrum Imaging Data

For each voxel of DSI data set, the 102 diffusion‐attenuated signals of a half sphere were projected to fill the other half of the sphere; notably, the data in the q space were considered real and symmetrical around the origin (Kuo et al., 2008). We calculated the probability density function (PDF) by invoking the Fourier relationship between the PDF and diffusion‐attenuated signals (Yeh et al., 2010), and determined the orientation distribution function (ODF) by computing the second moment of the PDF along each of the 362 radial directions in a sixfold tessellated icosahedron (Yeh et al., 2011). Generalized fractional anisotropy (GFA) was estimated as (standard deviation of the ODF)/(root mean square of the ODF) (Tuch, 2004). The GFA value is considered to be indicative of white matter integrity, as is the FA value in DTI (Gorczewski et al., 2009).

Tract‐Based Automatic Analysis

The DSI data sets were subjected to TBAA to generate connectograms following previously described procedures (Chen et al., 2015). Briefly, all the DSI data sets were registered to create a study‐specific template (SST), which was registered to the DSI template (NTU‐DSI‐122) (Hsu et al., 2015). We then transformed the predefined sampling coordinates of the 76 fiber tract bundles (Chen et al., 2015) from the DSI template into individual DSI data sets by transforming the sampling coordinates first from the DSI template to the SST, and then from the SST to individual DSI data sets. We sampled GFA along each of 76 fiber tracts by using the sampling coordinates that were transformed to each individual DSI data set, and finally obtained a 2D connectogram for each participant.

Diagnostic Analysis

The diagnostic analysis comprised training and testing phases. In the training phase, we identified a unique pattern of the connectogram that most precisely distinguished patients from healthy controls. In the testing phase, we individually applied the unique pattern to all of the participants and evaluated the diagnostic performance of the pattern considering sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV).

The image data in the training and testing phases were organized as independent data sets. Data acquired from January 2010 to May 2015 were analyzed in the training phase, which comprised 108 patients (54 males, 54 females) and 144 healthy controls (70 males, 74 females). Data acquired from May 2015 to April 2016 were analyzed in the testing phase, which comprised 60 patients (30 males, 30 females) and 60 healthy controls (30 males, 30 females). To account for the sex differences in diagnosing schizophrenia, the males and females were separately analyzed using the same procedures. The demographic data for the training and testing phases are shown in Tables 1 and 2, respectively. Finally, to determine how much accuracy was gained by employing different sex‐specific models, an additional analysis that combined the male and female data was performed.

Table 1.

Participant Demographics in the Training Phase

Male Female
Patients (N = 54) Controls (N = 70) P value Patients (N = 54) Controls (N = 74) P value
Age (years) 28.75 ± 5.78 29.11 ± 5.15 0.305a 30.34 ± 5.94 30.11 ± 5.36 0.456a
Education (years) 14.15 ± 2.23 15.82 ± 3.16 0.035a 14.37 ± 2.51 15.76 ± 2.86 0.087a
PANSS‐positive subscale score 12.72 ± 5.15 12.58 ± 5.33 0.957b
PANSS‐negative subscale score 17.19 ± 5.94 14.68 ± 6.01 0.031b
PANSS general psychopathology subscale score 25.35 ± 8.63 24.55 ± 8.23 0.452b
Duration of illness (years) 9.56 ± 7.72 8.05 ± 5.68 0.136b
Age of onset (years) 23.55 ± 7.23 24.61 ± 7.16 0.455b
Total antipsychotic dose (mg)c 311.63 ± 185.21 306.79 ± 175.43 0.963b
a

The P value between patients and controls.

b

The P value between male and female patients.

c

Daily chlorpromazine equivalent dose.

Table 2.

Participant Demographics in the Testing Phase

Male Female
Patients (N = 30) Controls (N = 30) P value Patients (N = 30) Controls (N = 30) P value
Age (years) 29.17 ± 5.78 28.66 ± 6.10 0.336a 29.54 ± 5.77 29.58 ± 5.63 0.653a
Education (years) 14.15 ± 2.23 15.82 ± 3.16 0.063a 14.37 ± 2.51 15.76 ± 2.86 0.127a
PANSS‐positive subscale score 13.13 ± 5.67 13.72 ± 5.45 0.918b
PANSS‐negative subscale score 15.21 ± 5.35 16.20 ± 6.16 0.121b
PANSS general psychopathology subscale score 26.63 ± 8.72 25.52 ± 8.14 0.631b
Duration of illness (years) 8.86 ± 5.22 7.72 ± 5.19 0.375b
Age of onset (years) 22.94 ± 7.05 23.11 ± 6.93 0.452b
Total antipsychotic dose (mg)c 303.45 ± 195.92 293.77 ± 177.68 0.926b
a

The P value between patients and controls.

b

The P value between male and female patients.

c

Daily chlorpromazine equivalent dose.

Training Phase

Figure 1 outlines the procedures used in the training phase, which consisted of six steps. The first three steps were conducted on half of the participants (Training I), and the last three steps were conducted on the remaining participants (Training II). To account for variability in the resulting unique pattern caused by such grouping, we also randomly divided the Training I and II participants and repeated the training procedures 300 times, generating 300 examinations in the training phase.

Figure 1.

Figure 1

Procedures used in the training phase. AUC, area under the curve; ROC, receiver‐operating characteristic; PLS, patient‐liked score; POC, patient‐or‐control.

For the Training I participants, we calculated the mean connectogram of both patients and controls (Step 1), and determined the differences in the obtained connectograms (Step 2). We subsequently obtained a series of masks based on the combined thresholds, which were derived from those connectogram differences (Step 3). Specifically, we calculated Cohen's effect size (ES) stepwise in the 2D connectogram (76 × 100 steps in total) as follows:

ES=|Mean(FAa)Mean(FAb)|(Std(FAa)2+Std(FAb)2)/2

where FAa and FAb indicate the FA values in the two comparison groups. In addition to the size of the difference, we also considered the length of the difference as another important factor for differentiating patients with schizophrenia from controls. Therefore, we defined the cluster size (CS) as the length of a segment in a tract that showed the difference contiguously. Specifically, the CS was the number of contiguous steps along a tract pathway that passed a certain threshold level of ES. The ES threshold levels included 21 levels equally partitioned from 0 to 1, and the CS threshold levels included 15 levels from 1 to 15. Thus, 315 (21 × 15) binary masks were determined in each examination.

For each Training II participant, a patient‐or‐control (POC) map was plotted by stepwise comparing the GFA value of the participant's connectogram with the mean GFA values of patients and controls, as obtained from step 1. If the GFA value of the participant was closer to that of the patient (i.e., |GFAparticipant − GFApatient| < |GFAparticipant − GFAcontrol|), a value of 1 was assigned to that step; if the GFA value was closer to that of the control, a value of −1 was assigned (Step 4). We applied each mask obtained in Step 3 to each participant's POC map and calculated the patient‐like‐score (PLS) by adding the values within the mask, which had been normalized by the number of steps in that mask (Step 5).

We then performed ROC curve analysis on each mask by comparing each participant PLS with individual clinical diagnostic outcomes (namely, schizophrenia or healthy). We estimated the area under the ROC curve (AUC) for each mask and plotted an AUC map that corresponded to a 21 × 15 matrix of AUC (Step 6).

We obtained 300 AUC maps after repeating the training procedures 300 times and identified the optimal threshold as the criterion of the mask that had the highest AUC value in the AUC map, averaged over 300 AUC maps. To visualize the unique pattern of the connectogram resulting from the optimal threshold, we created a heat map by accumulating the difference connectograms that were masked by the optimal threshold in the 300 examinations.

Testing Phase

We performed TBAA on each participant in the testing phase to obtain individual 2D connectograms and estimated the PLS of each participant by using the optimal mask obtained from the training phase. Here, we used the mean connectograms of all patients and controls in the training phase as a reference and performed the same comparison procedure, as described in Step 4 in the training phase. We used an optimal mask by applying the optimal threshold to the difference in the connectograms calculated from all participants in the training phase. The participants with a PLS of > 0 were considered to have schizophrenia, whereas those with a PLS of < 0 were considered healthy. By individually comparing PLS results with clinical diagnostic outcomes, we performed ROC curve analysis to evaluate the diagnostic performance and calculated indices, namely sensitivity, specificity, PPV, NPV, and accuracy.

Patterns Correlated With Clinical Variables

To elucidate the possible confounding effects of clinical variables on the optimal masks identified in the predictive analysis, the patterns of the 2D connectograms that correlated with clinical variables (including duration of illness and medication dose) were analyzed. Stepwise Pearson correlation between the GFA values and clinical variables was performed on each step of the connectograms. The steps with correlation coefficients larger than 0.2 were considered to be clinically correlated. Overlaps between the optimal masks and the clinically correlated steps were evaluated to clarify what confounding effects duration of illness and medication dose had on the optimal masks. Additionally, Pearson correlations between PLS and the Positive and Negative Syndrome Scale (PANSS) scores were evaluated for men and for women separately to investigate the relationships between the prediction scores (PLS) and clinical variables. The connectogram data used in the analysis here included those from patients in the training and testing phases.

RESULTS

Demographic Data

Tables 1 and 2 list the demographic data of the participants. Notably, there was almost no significant difference in any relevant parameters, except years of education between the male patients and controls (P = 0.035) and the PANSS‐negative subscale scores between the male and female patients in the training phase (P = 0.031).

Training Phase

Figure 2a,b shows the distributions of ES for men and women, respectively. Notably, men had more steps with high ES than did women: in men, there were 609 steps with ES > 0.8 (8% of 7600 steps), whereas in women, there were only 72 steps with ES > 0.8 (0.9% of 7600 steps).

Figure 2.

Figure 2

Distributions of Cohen's effect size (ES) for (a) men and (b) women. The top panels are the connectograms of ES and the bottom panels are the histograms of ES. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 3 shows the AUC maps for male, female, and all participants. Overall, the masks for men tended to have higher AUC values as they approached higher and lower threshold levels of ES and CS, respectively (Fig. 3a). The optimal mask was located at the combined threshold levels of ES = 0.85 and CS = 1, which yielded a mean AUC value of 0.87 (black‐edged square, Fig. 3a). The masks for women tended to have higher AUC values as they approached lower threshold levels of both ES and CS (Fig. 3b). The optimal mask was located at the combined threshold levels of ES = 0.1 and CS = 1, which yielded a mean AUC value of 0.77 (black‐edged square, Fig. 3b). When the participants were examined altogether, their masks tended to have higher AUC values as they approached lower threshold levels of both ES and CS (Fig. 3c). The optimal mask was located at the combined threshold levels of ES = 0.2 and CS = 1, which yielded a mean AUC value of 0.79 (black‐edged square, Fig. 3c).

Figure 3.

Figure 3

AUC maps for (a) males, (b) females, and (c) all participants. The AUC map was a 21 × 15 matrix, in which each element indicated the AUC value for a mask corresponding to a combined threshold level of the ES and cluster size (CS). The elements with numerical values are the elements surrounding the element with the maximal AUC value (black‐edged square). The averaged AUC value of each map was the value averaged from 300 examinations. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 4a–c reveals the optimal masks for men, women, and all the training participants, respectively. The optimal mask for men comprised 535 steps; these steps were discretely distributed, and mostly located in the association and commissural fibers. The optimal mask for women comprised 4460 steps; these steps were widespread over the whole brain. When the two masks were overlapped, 494 steps were found to overlap, constituting 92% of the mask for men and 11% of the mask for women.

Figure 4.

Figure 4

Optimal masks for (a) men (cyan) and (b) women (orange). (c) The overlaps between the two masks (red). [Color figure can be viewed at http://wileyonlinelibrary.com]

The heat maps of the optimal masks are presented in Figure 5. Those of the men showed a high occurrence of contribution from a few discrete segments (Fig. 5a), whereas those of women and of all the participants showed a widespread occurrence of contribution from all the tracts (Fig. 5b,c).

Figure 5.

Figure 5

Heat maps of the optimal masks for (a) males, (b) females, and (c) all participants. Each heat map was generated by accumulating the masks that resulted from the application of the optimal threshold levels of ES and CS to different connectograms in 300 examinations. The pixel was coded in blue if the mean generalized fractional anisotropy (GFA) value of the controls was larger than that of the patients (GFAC > GFAS), whereas it was coded in orange if the mean GFA value of the controls was smaller than that of the patients (GFAC < GFAS). The hue of the heat map indicates the number of occurrences included in the optimal mask from 300 examinations. [Color figure can be viewed at http://wileyonlinelibrary.com]

Figure 6 presents the 3D visualization of the distinct heat map patterns of male and female brains. Specifically, the optimal mask in men was concentrated in the association and commissural fibers, which connected the prefrontal and temporal lobes (Fig. 6a). By contrast, the optimal mask in women was uniformly distributed throughout the whole brain (Fig. 6b).

Figure 6.

Figure 6

3D visualization of the heat maps of the brains of the (a) male and (b) female participants. Each fiber tract bundle is represented by a track skeleton. Five discrete colors were assigned according to the number of occurrences in the heat maps. Cold‐colored segments represent those where the control GFA value was larger than the patient value (GFAC > GFAS), whereas the warm‐colored segments represent those where the control GFA value was smaller than the patient GFA value (GFAC < GFAS). The color appears bright if the number of occurrences in the segments was larger than 200, whereas it appears dark if the number was 100–200. Segments with <100 occurrences are shown in gray. [Color figure can be viewed at http://wileyonlinelibrary.com]

Testing Phase

The diagnostic indices were calculated from the testing group by using PLS = 0 as the cutoff point. For men, the overall prediction accuracy was 0.85, with a sensitivity of 0.83, specificity of 0.87, PPV of 0.86, and NPV of 0.84. For women, the overall accuracy was 0.75, with a sensitivity of 0.80, specificity of 0.70, PPV of 0.75, and NPV of 0.75. When all the participants were examined together, the overall accuracy was 0.76, with a sensitivity of 0.75, specificity of 0.76, PPV of 0.77, and NPV of 0.75. The ROC curves showed a satisfactory prediction performance of the optimal masks; specifically, the AUC value was 0.87, 0.77, and 0.79 for men, women, and all participants, respectively (Fig. 7).

Figure 7.

Figure 7

ROC curve analysis for men (black), women (dark gray), and all participants (light gray). The areas under the ROC curves are 0.87, 0.77, and 0.79 for men, women, and all participants, respectively.

Patterns Correlated With Clinical Variables

To investigate the possible confounding effects of clinical variables on the optimal masks, we calculated the Pearson correlation between clinical variables (duration of illness and medication dose) and GFA value on each step of the connectograms. The steps that were clinically correlated were 101 steps for duration of illness, and 0 steps for medication dose. Figure 8a shows the steps that were correlated with duration of illness. Notably, when overlapping these steps with the optimal masks, there were 15 overlapped steps for men (Fig. 8b) and 65 overlapped steps for women (Fig. 8c).

Figure 8.

Figure 8

(a) Steps in the connectogram that are correlated with duration of illness. The overlaps of these steps (red) with the optimal masks (cyan) are presented for (b) men and (c) women. [Color figure can be viewed at http://wileyonlinelibrary.com]

In male patients, the correlation coefficients between PLS and the PANSS positive, negative, and general psychopathology subscale scores were 0.36 (P = 0.008), 0.46 (P < 0.001), and 0.26 (P = 0.058), respectively; in female patients, these coefficients were 0.34 (P = 0.012), 0.35 (P = 0.009), and 0.15 (P = 0.279), respectively. The results indicated that male patients had higher correlations of PLS with PANSS scores than did female patients, and that PLS was most and least correlated with PANSS‐negative subscale scores and general psychopathology subscale scores, respectively.

DISCUSSION

This study proposed a method for the individualized prediction of schizophrenia on the basis of whole‐brain patterns of white matter tract integrity. Bypassing conventional machine learning methods, this study is the first to use the features resampled on major white matter tract pathways of the whole brain to predict schizophrenia at an individual level. We used TBAA to obtain a series of GFA values corresponding to the 76 white matter tracts for each participant. We then compared the individual information, namely, the 2D connectograms, with a collective database and scored the structural similarity of the connectograms to schizophrenia by capturing the patient‐like steps within masked regions. By using this approach, we successfully identified the optimal masks that most accurately predicted schizophrenia; the accuracy of the individualized predictions was 85%, 75%, and 76% for men, women, and all the participants, respectively. Such a high prediction performance suggests that the patterns of white matter tract integrity defined by the optimal mask contain representative information for schizophrenia. The unique patterns discovered in this study can therefore serve as potential imaging biomarkers for diagnosing schizophrenia.

Many studies have reported widespread differences in the whole‐brain FA values between patients with schizophrenia and healthy controls (Fujino et al., 2014; Roalf et al., 2013; Scheel et al., 2013; Walther et al., 2011). Although it is reasonable to use these different patterns as classifiers of schizophrenia at an individual level, challenges exist during implementation. First, overlaps of diffusion measures occur between patients and controls, even in the regions exhibiting significant statistical difference. Although combining the measures from multiple regions yields a higher classification performance (Zarogianni et al., 2013), it is unclear as to how many and which regions should be included. Moreover, the findings of significant differences are inconsistent among different studies. Some crucial predictors might be missed if one performs the classification by only using some regions showing statistically significant differences. The regions that fail the statistical tests may provide relevant information because these regions may be altered despite high variability. Therefore, in this study, we used patterns not necessarily related to statistical significance. A whole‐brain approach for analyzing diffusion data is favorable for obtaining such patterns, and here we selected TBAA because it provides diffusion measures of well‐defined white matter tracts in the entire brain. Additionally, the diffusion index GFA value was determined in the native space of participants and rendered in a standardized “tract space.” Such an output format is appropriate for performing individualized estimation. By comparing the similarity of the TBAA outputs with the collective database, we proved its feasibility in predicting schizophrenia on an individual basis by using the whole‐brain pattern of white matter tract integrity.

In our analyses, the optimal masks were identified at the CS threshold level of 1. Notably, this does not mean that the prediction is contributed from multiple single steps that are isolated from each other; indeed, the heat maps of the optimal masks (Fig. 5) reveal that there are multiple segments with variable lengths. The threshold level of CS = 1 suggests that contributions for predicting schizophrenia are important down to the single steps of the connectogram, which implies that the individual prediction of schizophrenia using diffusion MRI should be performed stepwise along the fiber pathway.

This study identified the optimal masks for predicting schizophrenia in chronic patients. Although we also considered sex‐associated differences and separately analyzed men and women, it is unclear whether the masks were confounded by clinical variables, such as duration of illness and medication dose. Our results showed that only duration of illness had an association with the GFA values in 101 steps of the connectogram, with 15 and 65 overlapping steps with the optimal masks for men and women, respectively, which constituted < 3% of the diagnostic masks (Fig. 8). These results suggest that the confounding effect from illness chronicity and medications on the optimal masks is negligible.

Whether the unique patterns identified in chronic patients are applicable to first‐episode patients remains unknown. Recent diffusion MRI studies on first‐episode patients have reported little difference in tract integrity between first‐episode and chronic patients (Fitzsimmons et al., 2014; Wu et al., 2015). In other words, it appears that patient‐related white matter tracts are impaired at disease onset and remain stationary throughout the course of illness. Bohlken et al. (2016) reported an association between genetic markers and brain connectivities in patients with schizophrenia and proposed that white matter integrity should be the trait marker associating genes with biological pathways that contribute to schizophrenia development. Overall, the aforementioned studies and our present analysis indicate that the unique tract integrity patterns identified in this study might predict schizophrenia early in the course of the disease.

The heat map of men substantially differs from that of women (Fig. 5). In men, the regions with a high occurrence of contribution were concentrated in discrete tract bundles and showed high ES differences between patients and controls. These tracts included the left arcuate fasciculus, left cingulum bundle (main body part), bilateral cingulum bundles (hippocampal part), bilateral fornices, bilateral uncinate fasciculi, bilateral inferior fronto‐occipital fasciculi, left inferior longitudinal fasciculus, bilateral frontostriatial tracts to the orbitofrontal cortex, and a large portion of callosal fibers. These tracts are well‐known for their microstructural derangement in schizophrenia and have been regularly reported in the literature (Fujino et al., 2014; Roalf et al., 2013; Scheel et al., 2013; Walther et al., 2011; Wu et al., 2015). By contrast, in women, the regions with a high occurrence of contribution were spread through the entire brain, involving not only the well‐known tracts but also many other tracts. The regions exhibiting a high ES were similar to those in men. However, if we only used the regions with a higher ES for classification, the performance was worse overall than were those with a lower ES. As shown in Figure 3b, the AUC was 0.68 at the combined threshold level of ES = 0.7 and CS = 1, and it increased as the ES threshold decreased. This indicates that the regions with a low ES also contribute to schizophrenia classification in women.

The difference in the heat maps may be related to the sex‐associated differences in schizophrenia, including onset age, symptom severity, treatment response, illness course, and outcome (Barajas et al., 2015; Moriarty et al., 2001; Ramsey et al., 2013). Women with schizophrenia tend to have less severe illness and more desirable global outcomes than do men (Vlassoff, 2007). Moreover, women with schizophrenia tend to show pronounced positive symptoms, such as hallucinations and delusions, whereas their male counterparts tend to manifest severe negative symptoms, such as anhedonia, asociality, and affective flattening (Leung and Chue, 2000). Women with schizophrenia are also more likely to be more socially active than are their male counterparts (Abel et al., 2010). In this study, we found that male patients had higher correlations between PLS and PANSS scores than did female patients, and that PLS was most correlated with PANSS negative subscale scores. These findings suggest that PLS partially reflects the severity of negative symptoms, which are sex‐associated. We therefore speculate that the discrete tracts shown in the heat map for men might be related to negative symptoms and be sufficient to predict schizophrenia in men.

The sex differences in the heat maps can also be understood from our ES analysis (Fig. 2). Specifically, we determined that men had more steps with high ES than did women. In men, 8% of the connectogram had ES > 0.8%, whereas in women, it was only 0.9%. Consequently, the average ES of the differences in connectograms was lower in females than in males (0.23 ± 0.18 vs 0.32 ± 0.24, respectively). Moreover, the ES maps were very different between men and women. In men, there were discrete segments with high ES that presented a pattern similar to that in their heat map. In women, most of the steps presented moderate ES with widespread distribution over the whole connectogram, again similar to the pattern in their heat map. This observation indicates that the drastic difference in the heat maps between men and women may arise from the higher variability in the GFA values in female patients than in male patients among our recruited participants.

In this article, we proposed using connectograms as features to predict schizophrenia at an individual level, and the prediction accuracy was 85%, 75%, and 76% for men, women, and all the participants, respectively. Although the performance is not as impressive as what has been reported using machine learning techniques (Castellani et al., 2012; Lu et al., 2016; Nieuwenhuis et al., 2012; Squarcina et al., 2017; Yang et al., 2010), our method is unique in two ways. First, the technique only uses difference connectograms with appropriate thresholds to predict schizophrenia. The simplicity of the method could potentially facilitate its clinical application. However, it may require a sufficiently large number of samples in the training set to achieve satisfactory performance, especially for determining subtle differences between the comparison groups. Second, the unique tract integrity patterns derived from our approach are specific tract locations in the “tract space.” These patterns can be understood in terms of the neuroanatomical functions of individual tracts. This allows us to interpret the results on the basis of previous neuroanatomical or physiological studies on schizophrenia.

This study has some limitations to acknowledge. First, we did not recruit patients with psychotic symptoms similar to schizophrenia, such as bipolar disorder, leaving uncertain specificity regarding the predictive patterns. One recent study indicated that patients with bipolar disorder and schizophrenia show a lower FA value than do healthy individuals. However, bipolar disorder involves fewer white matter regions than does schizophrenia (Skudlarski et al., 2013), and the different patterns of white matter integrity between patients with bipolar disorder and those with schizophrenia suggest that our predictive patterns might be specific for schizophrenia. Second, our patients in the testing phase were all patients with chronic schizophrenia (although, as previously discussed, the current optimal masks might also be applicable to first‐episode patients or those with ultrahigh risks). Further validation through a prospective study on this patient subgroup is necessary to confirm this applicability. Third, this study used DSI to acquire diffusion MRI data sets. Additional studies are required to compare the diagnostic performance across different diffusion acquisition schemes, such as DTI or Q‐ball imaging, or different diffusion indices.

In conclusion, the tract integrity information derived from TBAA is useful for the individualized prediction of schizophrenia. By applying the proposed analysis to chronic patients, the performance of individualized prediction was 85%, 75%, and 76% for male, female, and all the participants, respectively, in the ROC curve analysis. The proposed analysis method and resulting predictive patterns warrant further exploration of potential imaging biomarkers in refined subgroups of patients with schizophrenia or among a spectrum of acquisition variables.

FINANCIAL DISCLOSURE

All authors declare that there is no competing, financial, or potential conflicts of interests.

ACKNOWLEDGMENTS

This manuscript was edited by Wallace Academic Editing.

Y‐J Chen and C‐M Liu contributed equally to this work.

REFERENCES

  1. Abel KM, Drake R, Goldstein JM (2010): Sex differences in schizophrenia. Int Rev Psychiatry 22:417–428. [DOI] [PubMed] [Google Scholar]
  2. Alexander AL, Lee JE, Lazar M, Field AS (2007): Diffusion tensor imaging of the brain. NeuroTherapeutics 4:316–329. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Ardekani BA, Tabesh A, Sevy S, Robinson DG, Bilder RM, Szeszko PR (2011): Diffusion tensor imaging reliably differentiates patients with schizophrenia from healthy volunteers. Hum Brain Mapp 32:1–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Barajas A, Ochoa S, Obiols JE, Lalucat‐Jo L (2015): Gender differences in individuals at high‐risk of psychosis: A comprehensive literature review. Sci World J 2015:430735. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Basser PJ, Mattiello J, LeBihan D (1994): Estimation of the effective self‐diffusion tensor from the NMR spin echo. J Magn Reson Ser B 103:247–254. [DOI] [PubMed] [Google Scholar]
  6. Basser PJ, Pierpaoli C (1996): Microstructural and physiological features of tissues elucidated by quantitative‐diffusion‐tensor MRI. J Magn Reson Ser B 111:209–219. [DOI] [PubMed] [Google Scholar]
  7. Bohlken MM, Brouwer RM, Mandl RC, Van den Heuvel MP, Hedman AM, De Hert M, Cahn W, Kahn RS, Hulshoff Pol HE (2016): Structural brain connectivity as a genetic marker for schizophrenia. JAMA Psychiatry 73:11–19. [DOI] [PubMed] [Google Scholar]
  8. Caan MW, Vermeer KA, van Vliet LJ, Majoie CB, Peters BD, den Heeten GJ, Vos FM (2006): Shaving diffusion tensor images in discriminant analysis: A study into schizophrenia. Med Image Anal 10:841–849. [DOI] [PubMed] [Google Scholar]
  9. Caprihan A, Pearlson GD, Calhoun VD (2008): Application of principal component analysis to distinguish patients with schizophrenia from healthy controls based on fractional anisotropy measurements. NeuroImage 42:675–682. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Castellani U, Rossato E, Murino V, Bellani M, Rambaldelli G, Perlini C, Tomelleri L, Tansella M, Brambilla P (2012): Classification of schizophrenia using feature‐based morphometry. J Neural Transm (Vienna) 119:395–404. [DOI] [PubMed] [Google Scholar]
  11. Chen YJ, Lo YC, Hsu YC, Fan CC, Hwang TJ, Liu CM, Chien YL, Hsieh MH, Liu CC, Hwu HG, Tseng WY (2015): Automatic whole brain tract‐based analysis using predefined tracts in a diffusion spectrum imaging template and an accurate registration strategy. Hum Brain Mapp 36:3441–3458. [DOI] [PMC free article] [PubMed] [Google Scholar]
  12. Chiappelli J, Hong LE, Wijtenburg SA, Du X, Gaston F, Kochunov P, Rowland LM (2015): Alterations in frontal white matter neurochemistry and microstructure in schizophrenia: Implications for neuroinflammation. Transl Psychiatry 5:e548. [DOI] [PMC free article] [PubMed] [Google Scholar]
  13. Fitzsimmons J, Hamoda HM, Swisher T, Terry D, Rosenberger G, Seidman LJ, Goldstein J, Mesholam‐Gately R, Petryshen T, Wojcik J, Kikinis R, Kubicki M (2014): Diffusion tensor imaging study of the fornix in first episode schizophrenia and in healthy controls. Schizophrenia Res 156:157–160. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Fujino J, Takahashi H, Miyata J, Sugihara G, Kubota M, Sasamoto A, Fujiwara H, Aso T, Fukuyama H, Murai T (2014): Impaired empathic abilities and reduced white matter integrity in schizophrenia. Progr Neuropsychopharmacol Biol Psychiatry 48:117–123. [DOI] [PubMed] [Google Scholar]
  15. Gorczewski K, Mang S, Klose U (2009): Reproducibility and consistency of evaluation techniques for HARDI data. MAGMA 22:63–70. [DOI] [PubMed] [Google Scholar]
  16. Han M, Huang XF, Chen DC, Xiu MH, Hui L, Liu H, Kosten TR, Zhang XY (2012): Gender differences in cognitive function of patients with chronic schizophrenia. Progr Neuropsychopharmacol Biol Psychiatry 39:358–363. [DOI] [PubMed] [Google Scholar]
  17. Harrison PJ (1999): The neuropathology of schizophrenia. A critical review of the data and their interpretation. Brain 122: 593–624. [DOI] [PubMed] [Google Scholar]
  18. Hsu YC, Hsu CH, Tseng WY (2012): A large deformation diffeomorphic metric mapping solution for diffusion spectrum imaging datasets. NeuroImage 63:818–834. [DOI] [PubMed] [Google Scholar]
  19. Hsu YC, Lo YC, Chen YJ, Wedeen VJ, Isaac Tseng WY (2015): NTU‐DSI‐122: A diffusion spectrum imaging template with high anatomical matching to the ICBM‐152 space. Hum Brain Mapp 36:3528–3541. [DOI] [PMC free article] [PubMed] [Google Scholar]
  20. Kambeitz J, Kambeitz‐Ilankovic L, Leucht S, Wood S, Davatzikos C, Malchow B, Falkai P, Koutsouleris N (2015): Detecting neuroimaging biomarkers for schizophrenia: A meta‐analysis of multivariate pattern recognition studies. Neuropsychopharmacology 40:1742–1751. [DOI] [PMC free article] [PubMed] [Google Scholar]
  21. Kubicki M, McCarley RW, Shenton ME (2005): Evidence for white matter abnormalities in schizophrenia. Curr Opin Psychiatry 18:121–134. [DOI] [PMC free article] [PubMed] [Google Scholar]
  22. Kuo LW, Chen JH, Wedeen VJ, Tseng WY (2008): Optimization of diffusion spectrum imaging and q‐ball imaging on clinical MRI system. NeuroImage 41:7–18. [DOI] [PubMed] [Google Scholar]
  23. Leung A, Chue P (2000): Sex differences in schizophrenia, a review of the literature. Acta Psychiatr Scand Suppi 401:3–38. [DOI] [PubMed] [Google Scholar]
  24. Lu X, Yang Y, Wu F, Gao M, Xu Y, Zhang Y, Yao Y, Du X, Li C, Wu L, Zhong X, Zhou Y, Fan N, Zheng Y, Xiong D, Peng H, Escudero J, Huang B, Li X, Ning Y, Wu K (2016): Discriminative analysis of schizophrenia using support vector machine and recursive feature elimination on structural MRI images. Medicine 95:e3973. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Moriarty PJ, Lieber D, Bennett A, White L, Parrella M, Harvey PD, Davis KL (2001): Gender differences in poor outcome patients with lifelong schizophrenia. Schizophrenia Bull 27:103–113. [DOI] [PubMed] [Google Scholar]
  26. Nieuwenhuis M, van Haren NE, Hulshoff Pol HE, Cahn W, Kahn RS, Schnack HG (2012): Classification of schizophrenia patients and healthy controls from structural MRI scans in two large independent samples. NeuroImage 61:606–612. [DOI] [PubMed] [Google Scholar]
  27. Nurnberger JI, Jr. , Blehar MC, Kaufmann CA, York‐Cooler C, Simpson SG, Harkavy‐Friedman J, Severe JB, Malaspina D, Reich T (1994): Diagnostic interview for genetic studies. Rationale, unique features, and training. NIMH Genetics Initiative. Arch Gen Psychiatry 51:849–859. discussion 863‐4. [DOI] [PubMed] [Google Scholar]
  28. Oh JS, Kubicki M, Rosenberger G, Bouix S, Levitt JJ, McCarley RW, Westin CF, Shenton ME (2009): Thalamo‐frontal white matter alterations in chronic schizophrenia: A quantitative diffusion tractography study. Hum Brain Mapp 30:3812–3825. [DOI] [PMC free article] [PubMed] [Google Scholar]
  29. Pettersson‐Yeo W, Allen P, Benetti S, McGuire P, Mechelli A (2011): Dysconnectivity in schizophrenia: Where are we now?. Neurosci Biobehav Rev 35:1110–1124. [DOI] [PubMed] [Google Scholar]
  30. Ramsey JM, Schwarz E, Guest PC, van Beveren NJM, Leweke FM, Rothermundt M, Bogerts B, Steiner J, Bahn S, Burne T (2013): Distinct molecular phenotypes in male and female schizophrenia patients. PLoS One 8:e78729. [DOI] [PMC free article] [PubMed] [Google Scholar]
  31. Reese TG, Heid O, Weisskoff RM, Wedeen VJ (2003): Reduction of eddy‐current‐induced distortion in diffusion MRI using a twice‐refocused spin echo. Magn Reson Med 49:177–182. [DOI] [PubMed] [Google Scholar]
  32. Roalf DR, Ruparel K, Verma R, Elliott MA, Gur RE, Gur RC (2013): White matter organization and neurocognitive performance variability in schizophrenia. Schizophrenia Res 143:172–178. [DOI] [PMC free article] [PubMed] [Google Scholar]
  33. Scheel M, Prokscha T, Bayerl M, Gallinat J, Montag C (2013): Myelination deficits in schizophrenia: Evidence from diffusion tensor imaging. Brain Struct Funct 218:151–156. [DOI] [PubMed] [Google Scholar]
  34. Skudlarski P, Schretlen DJ, Thaker GK, Stevens MC, Keshavan MS, Sweeney JA, Tamminga CA, Clementz BA, O'Neil K, Pearlson GD (2013): Diffusion tensor imaging white matter endophenotypes in patients with schizophrenia or psychotic bipolar disorder and their relatives. Am J Psychiatry 170:886–898. [DOI] [PubMed] [Google Scholar]
  35. Squarcina L, Castellani U, Bellani M, Perlini C, Lasalvia A, Dusi N, Bonetto C, Cristofalo D, Tosato S, Rambaldelli G, Alessandrini F, Zoccatelli G, Pozzi‐Mucelli R, Lamonaca D, Ceccato E, Pileggi F, Mazzi F, Santonastaso P, Ruggeri M, Brambilla P (2017): Classification of first‐episode psychosis in a large cohort of patients using support vector machine and multiple kernel learning techniques. NeuroImage 145:238–245. [DOI] [PubMed] [Google Scholar]
  36. Stephan KE, Friston KJ, Frith CD (2009): Dysconnection in schizophrenia: From abnormal synaptic plasticity to failures of self‐monitoring. Schizophrenia Bull 35:509–527. [DOI] [PMC free article] [PubMed] [Google Scholar]
  37. Tuch DS (2004): Q‐ball imaging. Magn Reson Med 52:1358–1372. [DOI] [PubMed] [Google Scholar]
  38. Vlassoff C (2007): Gender differences in determinants and consequences of health and illness. Journal of health, population, and Nutrition 25:47–61. [PMC free article] [PubMed] [Google Scholar]
  39. Walther S, Federspiel A, Horn H, Razavi N, Wiest R, Dierks T, Strik W, Muller TJ (2011): Alterations of white matter integrity related to motor activity in schizophrenia. Neurobiol Dis 42:276–283. [DOI] [PubMed] [Google Scholar]
  40. Wolfers T, Buitelaar JK, Beckmann CF, Franke B, Marquand AF (2015): From estimating activation locality to predicting disorder: A review of pattern recognition for neuroimaging‐based psychiatric diagnostics. Neurosci Biobehav Rev 57:328–349. [DOI] [PubMed] [Google Scholar]
  41. World Health Organization (1992): The ICD‐10 Classification of Mental and Behavioural Disorders: Clinical Descriptions and Diagnostic Guidelines, Vol. xii Geneva: World Health Organization; 362 p. [Google Scholar]
  42. Wu CH, Hwang TJ, Chen YJ, Hsu YC, Lo YC, Liu CM, Hwu HG, Liu CC, Hsieh MH, Chien YL, Chen CM, Isaac Tseng WY (2015): Primary and secondary alterations of white matter connectivity in schizophrenia: A study on first‐episode and chronic patients using whole‐brain tractography‐based analysis. Schizophrenia Res 169:54–61. [DOI] [PubMed] [Google Scholar]
  43. Yang H, Liu J, Sui J, Pearlson G, Calhoun VD (2010): A hybrid machine learning method for fusing fMRI and genetic data: Combining both improves classification of schizophrenia. Front Hum Neurosci 4:192. [DOI] [PMC free article] [PubMed] [Google Scholar]
  44. Yeh FC, Wedeen VJ, Tseng WY (2010): Generalized q‐sampling imaging. IEEE Trans Med Imag 29:1626–1635. [DOI] [PubMed] [Google Scholar]
  45. Yeh FC, Wedeen VJ, Tseng WY (2011): Estimation of fiber orientation and spin density distribution by diffusion deconvolution. NeuroImage 55:1054–1062. [DOI] [PubMed] [Google Scholar]
  46. Yendiki A, Koldewyn K, Kakunoori S, Kanwisher N, Fischl B (2014): Spurious group differences due to head motion in a diffusion MRI study. NeuroImage 88:79–90. [DOI] [PMC free article] [PubMed] [Google Scholar]
  47. Zarogianni E, Moorhead TW, Lawrie SM (2013): Towards the identification of imaging biomarkers in schizophrenia, using multivariate pattern classification at a single‐subject level. NeuroImage Clin 3:279–289. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Human Brain Mapping are provided here courtesy of Wiley

RESOURCES