The Effect of Preprocessing Pipelines in Subject Classification and Detection of Abnormal Resting State Functional Network Connectivity using Group ICA

Abstract

Resting state functional network connectivity (rsFNC) derived from functional magnetic resonance (fMRI) imaging is emerging as a possible biomarker to identify several brain disorders. Recently it has been pointed out that methods used to preprocess head motion variance might not fully remove all unwanted effects in the data. Proposed processing pipelines locate the treatment of head motion effects either close to the beginning or as one of the final steps. In this work, we assess several preprocessing pipelines applied in group independent component analysis (gICA) methods to study the rsFNC of the brain. The evaluation method utilizes patient/control classification performance based on linear support vector machines and leave-one-out cross validation. In addition, we explored group tests and correlation with severity measures in the patient population. We also tested the effect of removing high frequencies via filtering. Two real data cohorts were used: one consisting of 48 mTBI and one composed of 21 smokers, both with their corresponding matched controls. A simulation procedure was designed to test the classification power of each pipeline. Results show that data preprocessing can change the classification performance. In real data, regressing motion variance before gICA produced clearer group differences and stronger correlation with nicotine dependence.

1 Introduction

Reliable identification of mental illnesses and brain related diseases is critical to apply adequate treatment and improve medical outcomes. To this day, many diagnosis tools rely on subject interviews (Fagerstrom et al., 1990; Saunders et al., 1993) and some have been difficult to validate (Balestreri et al., 2004; Borg et al., 2004; Ruff et al., 2009). On the other hand, techniques such as the assessment of resting state functional network connectivity (rsFNC) of the brain, derived from functional magnetic resonance imaging (fMRI), are emerging as possible alternatives with clinical applications (Fox et al., 2010; Lee et al., 2013b). Changes in rsFNC patients have been observed in many studies including traumatic brain injury (Mayer et al., 2011; Vakhtin et al., 2013), addiction (Chanraud et al., 2011; Claus et al., 2013; Schmidt et al., 2015), Alzheimer’s disease (Sheline and Raichle, 2013), bipolar disorder and schizophrenia (Chai et al., 2011; Das et al., 2014; Yu et al., 2011). In spite of the success achieved, there have been few systematic approaches comparing classification results as a function of differences in data processing. Most classification studies use a specific pipeline and there has been no consensus on how to select the best processing pipeline or on whether there is an optimal approach.

There are many choices to make when preparing data for group independent component analysis (gICA). Appropriate data preprocessing can remove spurious variance unrelated to neurological signals, decrease the presence of false positives and increase the possibility of observing neural effects. One important characteristic that seems to change in different studies is the order in which preprocessing steps are applied. Of special concern are the steps where head motion correction is applied (Mowinckel et al., 2012; Satterthwaite et al., 2012; Van Dijk et al., 2012). Some algorithms such as ICA (Calhoun and Adali, 2012) tend to be more robust, but are not totally immune to motion (Damaraju et al., 2014). Motion preprocessing includes the removal of motion variance via regression of realignment parameters and minimization of the effects of sudden head movements, which produces spikes in the signal. A recent proposal for a methodological baseline in gICA and rsFNC suggests processing head motion correction after gICA (Allen et al., 2011). In some cases, the despiking step is performed before gICA (Mayer et al., 2014). Recent analysis of head motion variance in seed-based methods suggests the correction of head movement early in the preprocessing (Power et al., 2014). So far there has not been a consensus, or a comparison, among these options when gICA methods are utilized, resulting in different versions of preprocessing pipelines. In general, the effects of different preprocessing choices on single subject classification are understudied and more evidence is needed to determine valid recommendations and best practices.

In the present work, we hypothesize that the order of steps taken to deal with motion artifacts will affect the final results obtained from statistical group difference tests, correlation with behavioral assessments and single subject classification. We assess several alternative pipelines that permute despiking, motion variance regression and gICA steps in different orders. The hypothesis is tested using simulated and real data from two different cohorts. The analysis focuses on cross validated subject classification and patient-control group differences.

Classification and cross validation techniques have been applied in the past to assess the applicability of functional connectivity as a biomarker (Liu et al., 2015; Pariyadath et al., 2014; Zeng et al., 2014). In addition to group differences, the present work uses a cross validation approach to analyze the differences among pipelines. The motivation for investigating these differences in preprocessing options is based on recent studies that warn about small but significant rsFNC bias due to head motion (Power et al., 2012; Van Dijk et al., 2012). We carefully considered the suggestion that rsFNC bias could be minimized by processing head motion variance before using the gICA (Damaraju et al., 2014), rather than after gICA as has been done previously (Allen et al., 2011; Mayer et al., 2014; Rocca et al., 2013; Vakhtin et al., 2013; Yuan et al., 2015). Frequency filtering is another process that can greatly change the results. Although high frequencies have been commonly related to nuisance biological signals (Allen et al., 2011; Cordes et al., 2001), other studies also suggest that important information is contained in high frequencies (Calhoun et al., 2011; Chen and Glover, 2015; Garrity et al., 2007; Van Someren et al., 2011; Yaesoubi et al., 2015). A frequency analysis of resting state signals using this TE dependency indicates that BOLD signals can have important content up to 0.5 Hz (Chen and Glover, 2015). Based on these results, we include filters with different bandwidths in the analysis, assuming that important information for classification purposes could be found in frequencies above 0.15 Hz. However, observations are compared against results obtained after filtering high frequencies to validate the outcomes.

2 Materials and methods

This study uses numerical simulation to test the performance obtained from using different preprocessing pipelines and then repeats the analysis using real data. Three different sample cohorts were selected. One cohort features mild traumatic brain injury (mTBI) patients and a set of healthy controls. In order to have a large enough number of samples, numerical simulations utilized a set of healthy subjects plus the healthy controls from the mTBI cohort as seed data. Afterwards, the analysis was repeated using the whole mTBI cohort. Because the control samples were used twice, it was necessary to replicate the results in a different and independent cohort to confirm the observed effects. The second cohort is smaller than the first and features nicotine dependent versus non-dependent subjects. The classification for this second cohort tries to differentiate smokers from non-smokers, but it also allows us to examine how preprocessing affects estimating the relationship between rsFNC and nicotine dependence.

2.1 The mTBI Cohort

A total of 96 subjects, 48 mTBI patients (24 females) plus 48 age (within 3 years) and sex matched healthy controls (HC), were selected for this study. In addition, subjects were matched on head movement using the mean frame-wise displacement (FWD) measure (Power et al., 2014) such that mTBI and HC do not differ more than 0.5 mm and the group difference was not statistically significant (p > 0.30). The 48 mTBI patients (mean age 27.8 ± 9.2) were recruited from local emergency rooms. Subjects classified as mTBI had a Glasgow Coma Scale between 13 and 15 at first contact with medical staff, no more than 30 minutes loss of consciousness (if present), and no more than 24 hours post-traumatic amnesia in case it happened. This inclusion criterion is based on the American Congress of Rehabilitation Medicine. HC and mTBI subjects were excluded if there was a prior history of neurological disease, major psychiatric disturbance, and additional closed head injuries with more than 5 minutes of lost consciousness, additional closed head injury within the past year, learning disorder, ADHD or a history of substance abuse/dependence including alcohol. All participants provided informed consent in accord to institutional guidelines at the University of New Mexico.

2.2 The Simulation Cohort

A set of 76 healthy subjects from another study was combined with the 48 HC from the mTBI cohort (both collected with the same parameters on the same scanner) resulting in a total of 124 subjects. The age of this cohort ranged from 18 to 65 (mean 30.8 ± 11.1) years. Of the 124 subjects 58 were females and 66 were males. Subjects from the extra 76 subjects did not report injury to the brain, brain-related medical problems, substance abuse, bipolar or psychotic disorders. Samples were distributed among two groups labeled as simulated patients (SP) and simulated controls (SC) with matched age (up to 3 years) and gender.

2.3 The Smokers Cohort

A total of 42 subjects, 21 smokers (10 females) between 18 and 51 years old (31.5 ± 10.5) plus 21 age (within 3 years) and sex matched non-smoking controls, were included. Mean FWD between matched samples is no larger than 0.5 mm and the group difference is not significant (p > 0.45). All smokers were instructed to abstain from smoking for three hours before the scan. Subjects were excluded if there was injury to the brain, brain-related medical problems, or bipolar or psychotic disorders. Smokers (SMK) reported having more than one cigarette per day over a period of 60 days before the scan. Levels of substance dependence for non-control subjects were measured using the Fagerstrom Test for Nicotine Dependence (FTND) (Fagerstrom et al., 1990). The mean FTND score for smokers is 10 ± 1.7. Smokers with signs of alcohol dependence were excluded based on an AUDIT score (Saunders et al., 1993) larger than 7. The non-smoking controls (CTR) group consisted of healthy individuals with no history of substance abuse/dependence assessed using the Structured Clinical Interview for DSM-IV-TR Axis I Disorders, Research Version, Patient Edition (SCID-I/P) (First et al., 2002). Control subjects with current abuse or dependence of alcohol were excluded. AUDIT scores were not collected for the CTR group. All control subjects reported themselves as non-smokers and scored zero for the FTND.

2.4 Imaging

All images were collected on a 3 Tesla Siemens Trio scanner. A five minute resting state run was completed by each participant using a single-shot, gradient-echo echo planar pulse sequence (TR = 2000 ms; TE = 29 ms; flip angle = 75º; FOV = 240 mm; matrix size = 64 x 64). The first five images were eliminated to account for T1 equilibrium effects. A total of 145 images were selected for further analysis. Foam padding and paper tape was used to restrict motion within the scanner. Presentation software (Neurobehavioral Systems) was used for stimulus presentation and synchronization of stimuli with the MRI scanners. Subjects were instructed to stare at a foveally presented fixation cross (visual angle = 1.02º) for approximately five minutes and to minimize head movement. Thirty-three contiguous, axial 4.55-mm thick slices were selected in the mTBI cohort to provide whole-brain coverage (voxel size: 3.75 x 3.75 x 4.55 mm, 1.05 mm gap). In the smokers cohort each volume consisted of 33 axial slices (64 × 64 matrix, 3.75 × 3.75 mm², 3.5 mm thickness, 1 mm gap).

2.5 Preprocessing Pipelines

After examining the different variations of preprocessing that can be found in the literature (Allen et al., 2011; Mayer et al., 2014; Power et al., 2012), we considered four different pipelines in the current study. The primary differences among pipelines varied the order in which spike and motion artifact variances were processed. The four pipelines were evaluated via training/test cross-validation procedure explained in a further section. This selection procedure is performed to avoid biasing the classification results by hand picking the best pipeline. Figure 1 presents a depiction of each pipeline and a detail description of the preprocessing steps used is provided next.

Figure 1.

Preprocessing pipelines tested in this study. Before estimating rsFNC, a set of preprocessing steps must be applied to the raw fMRI data. The initial steps labeled as STRCoN correspond to slice-timing correction, rigid body realignment for motion, co-registration, spatial normalization, and transformation to the MNI standard space. The step labeled MotReg corresponds with removal of motion variance through regression. Step SpkReg indicates orthogonalization with respect to spike regressors. When spike and motion regressions were necessary, they were performed in one single regression step called SpkMotReg. The label Smooth-gICA indicates that smoothing and group ICA were performed. The final step on each pipeline is Filter, where a pass-band filter is applied. Four different pipelines named PA, PB, PC and PD were considered with different order of steps as displayed in this figure.

All of the pipelines were preprocessed using statistical parametric mapping 5 (SPM; http://www.fil.ion.ucl.ac.uk/spm) (Friston, 2003) including slice-timing correction, realignment, co-registration, spatial normalization and transformation to the Montreal Neurological Institute (MNI) standard space. These preprocessing steps will be designated as “STRCoN” for notation purposes. The despiking step, designate as “SpkReg”, consisted of the orthogonalization with respect to spike regressors, a procedure that was implemented in house using linear regression. Each spike is represented by an independent regressor valued one at the spike time point and zero everywhere else. The DVARS method (Power et al., 2012) was used to find spike regressors where the RMS exceeded 3 standard deviations. In the step designated as “MotReg”, time courses were orthogonalized with respect to i) linear, quadratic and cubic trends; ii) the six realignment parameters; and iii) realignment parameter derivatives. In two of the pipelines, the steps MotReg and SpkReg are performed together using one regression analysis that includes all regressors. We will denote this step as “SpkMotReg”. Smoothing and group independent component analysis (gICA) are performed one after the other in all four pipelines. The notation “Smooth-gICA” is used to indicate the application of these two steps. An FWHM Gaussian kernel of 6 mm was used for the smoothing step. Data from all subjects were subject to a gICA (Calhoun et al., 2001; Calhoun and Adali, 2012) using the GIFT software (http://mialab.mrn.org/software/gift/) to obtain a set of functionally independent resting state networks (RSN). The number of components was determined to be 70 using a modified version of ICASSO (Himberg et al., 2004; Ma et al., 2011) such that the overall R-index is close to the minimum and the quality index of any given RSN is above 0.7. This setup was considered a good consistency trade-off between RSN quality and number of components considering the differences among all four pipelines. The final step “Filter” refers to a fifth-order Butterworth band-pass filter. Because of spike orthogonalization from either SpkReg or SpkMotReg, spike time points are zero valued. In order to apply filtering, spike time-points were interpolated using a spline then set again to zero after filtering.

Four different pipelines were implemented using the steps described above. Pipeline A (PA) applies spike and motion regressors using “SpkMotReg” before gICA in the order STRCoN-SpkMotReg-Smooth-gICA-Filter. In pipeline B (PB), only motion parameters are processed before gICA with steps succession STRCoN-MotReg-Smooth-gICA-SpkReg-Filter. In pipeline C (PC) only despiking is performed before gICA with step ordering STRCoN-SpkReg-Smooth-gICA-MotReg-Filter. Pipeline D (PD) applies “SpkMotReg” after gICA using the order STRCoN-Smooth-gICA-SpkMotReg-Filter. In addition, fifteen different data sets were calculated for each pipeline using filters with low cutoff frequency of 0.01 Hz and high cutoff frequencies between 0.10 Hz and 0.24 Hz separated by 0.01Hz steps. For each of the three experiments (simulation, mTBI and smokers), sixty different data sets were generated based on the four pipelines and fifteen frequency filters.

2.6 Functional Network Connectivity

Spatial maps were z-transformed and thresholded at |z|>3.5 to identify brain areas of relevance in each RSN. Artifactual RSNs were detected and discarded based on their dynamic range and low to high frequency power ratio following the method proposed by Allen et al. (Allen et al., 2011). RSNs were also manually inspected and not included if their spatial maps had considerable overlap with cerebrospinal fluid or white matter areas. RSNs were classified into broader categories or discarded if the main activation was located in areas of white matter or cerebrospinal fluid. In the case of the mTBI cohort, a subset of 53 non-artifactual RSNs that could be replicated on all four pipelines was selected. In the case of the smokers cohort, only 43 RSNs were selected following the same procedure. RSNs were matched across pipelines by visual inspection and a spatial correlation of at least 0.5. Choosing a subset of RSNs with spatial content that can be match across all four permutations of steps SpkReg and MotReg (including SpkMotReg) minimized the differences among pipelines. These RSNs were organized in eight groups: subcortical (SBC), auditory (AUD), sensorimotor (SEN), cerebellum (CER), visual (VIS), attentional (ATT), default mode network (DMN), and cognitive control (CC). The CER group contains RSNs with spatial content that overlaps with visual and cerebellum regions, but their peak activation was detected within the cerebellum. Fifteen time-course sets were calculated for each case using band-pass filters with a low cutoff frequency of 0.01 Hz and high cutoff frequencies between 0.10 Hz and 0.24 Hz separated by 0.01Hz steps. The rsFNC matrix of each subject and time-course set was determined by measuring the correlation coefficient between the time courses of each one of the possible RSN pairs. The mTBI cohort has a total of 1378 (53*52/2) rsFNCs and the smokers cohort a total of 903 (43*42/2). For all RSNs and all subjects, the time points where spikes were detected were excluded from the calculation of correlation.

2.7 Simulations

Simulation frameworks for rsFNC can be overwhelmingly complicated. Examples where ground truth and noise becomes separated are not readily available. After trying different simulation scenarios, we concluded that the end result of applying gICA to artificially synthesized signals may either create unrelated components or be too simplistic. However, it is necessary to test the effectiveness of our pipeline assessment approach by controlling the rsFNC without excluding real-data characteristics. To this end, we have determined that manipulating real fMRI images, by adding an artificially generated signal in such a way that it will not undermine the effects of neuronal and noise-nuisance signals, will be the best strategy. This approach is practical only if the power of added signals is not too strong. In our model, time courses TC for each RSN are a combination of BOLD and noise such that TC₁(t) = BOLD₁(t) + NOISE₁(t) and TC₂(t) = BOLD₂(t) + NOISE₂(t). Ideally the correlation between BOLD signals corr[BOLD₁, BOLD₂] is zero or close enough to zero, but in practice corr[TC₁, TC ₂] is the available measurement. The simulation strategy is to add a signal S(t), with the resulting mixture TC_new (t) = TC(t) + S(t) = BOLD(t) + NOISE(t) + S(t) to tip off the correlation between two RSNs that originally had close to a zero correlation. Characterization of signal power is better defined as a simple power ratio (PR) between the variances of S(t) and TC(t), since a signal-to-noise ratio would require knowledge of the original BOLD signal. The procedure used to implement the simulation is described in this section.

Data for the SC controls were not modified in any way during simulation. Data for the SP samples were manipulated to increase the correlation between the time courses of chosen RSNs. The same artificially generated signal with relatively small amplitude, but algorithmically controlled time shifts, was implanted in each one of time courses from the chosen RSNs in the SP samples. Simulated connectivity was introduced after registration and spatial normalization. Up to this point, all four pipelines followed the same preprocessing steps. Time courses for the two selected RSNs were then extracted from the 4D fMRI image using space time regression (STR). The signals extracted with STR were used without filtering, despiking, smoothing or nuisance correction. The artificially created signals were back-reconstructed into the fMRI data with a random spatial shift. The shift was implemented to simulate subject-wise differences of gICA spatial maps observed in real data. Spatial shifts for each coordinate were drawn from a zero mean normal distribution with a standard deviation of 9 mm. The artificially created signals were generated by adding sinusoidal waves with frequencies between 0.01 Hz and 0.24 Hz in increments of 0.01 Hz. The phases of each sinusoidal wave were chosen at random using a uniform distribution U[0, 2π]. One of the reasons for not limiting to low frequency ranges was the fact that filtering had not been applied at this point. The other reason was based on observations in the literature stating that BOLD signals have significant frequency content even at frequencies higher than 0.15Hz (Kundu et al., 2012). The steps used for the complete simulation procedure are described in the bullet list below.

• Select a population of healthy controls and run gICA.

• Select two non-artifact RSNs (RSN₁ and RSN₂) with time-courses exhibiting a small correlation.

• Divide the population into SP and SC groups such that age and sex are matched.

• Create two artificial signals S₁(t) and S₂(t) with a known correlation. These signals can be sums of sines of required frequencies, but random phase shifts.

• Separate registered and normalized fMRI images from the SP group and only for these images proceed as follows:

  • Use STR to estimate the time courses TC₁(t) and TC₂(t) from RSN₁ and RSN₂.

  • Properly change the amplitude of the signals S₁(t) and S₂(t) to achieve required PR.

  • Create two RSNs (sRSN₁ and sRSN₂) by spatially shifting RSN₁ and RSN₂ by a distance and direction drawn at random. This can be implementing by randomly drawing a different number of voxels to shift for each of the three coordinates: X, Y, and Z.

  • Reconstruct the fMRI image using unshifted data TC₁(t), TC₂(t), RSN₁ and RSN₂ plus the shifted RSNs and artificial signals S₁(t), S₂(t), sRSN₁ and sRSN₂.

• Run SC and SP together through the respective preprocessing pipeline.

After adding TC₁(t) with S₁(t) and TC₂(t) with S₂(t), it can be inferred that measured correlation between final mixed time-courses is not the same as correlation between the artificial signals. Measured correlations changed according to the artificial signal power as displayed in Figure 2. The simulation method seeks to merely increment the correlation between time courses, rather than replacing the signal. For this purpose, the power ratio selected between artificial signal and measured time course was low. The time course had eight times more power than the artificially added signal (a PR of −9dB) in order to avoid overshadowing the characteristics of the STR signal. Two correlation levels (0.1 and 0.2) were enforced algorithmically. For a total of 145 time points, it is necessary to have a correlation of 0.16 or higher in order to be significant at the p<0.05. Correlation levels 0.1 and 0.2 lie at both sides of the significance threshold and will help in understanding the difference between strong and weak connectivity.

Figure 2.

Plot used to parameterize the simulation. This shows the relationship between final Measured Correlation, Power-Ratio (PR) and correlation between artificial signals. The plot uses two different time-courses: TC1 and TC2. These time-courses were extracted from fMRI data for two different RSNs using space-time-regression. The two time courses exhibit low correlation value. Two signals, S1 and S2, are algorithmically generated to have an artificial correlation Art.Corr[S1,S2] with different values 1.0, 0.5, 0.2 and 0.1. Signals and time courses are mixed and the measured correlation corr[TC1+S1, TC2+S2] is plotted for different, PR = (var[S]/var[TC]). After 0dB, Art.Corr dominates the measured correlation, but below −12dB its influence is minimal. The simulation was performed with PR = −9dB, aiming at minimizing changes in the characteristics of original time courses TC1 and TC2, but allowing a significant correlation level of 0.2. In our data, significant correlations (p<0.05) are those surpassing the threshold of 0.16.

2.8 Pipeline Assessment

We designed a selection process to allow for the classification algorithm to pick appropriate pipeline and frequency content. Classification performance measures that allowed for a more thorough analysis were also obtained through this selection process. This section describes the applied pipeline/frequency selection.

In each cohort, the features used consisted of the time course correlations of the rsFNC matrices. The rsFNC features were orthogonalized with respect to age and gender before classification. A linear support vector machine (SVM) based on sequential minimal optimization with soft margin parameter C = 1 was utilized to classify subjects. The performance of the SVM classifier was tested using a leave-one-out cross-validation (LOOCV). Classification accuracy was assessed using the area under the curve (AUC) measure. An additional selection loop was nested for each one of the LOOCV iterations to select the best pipeline and bandwidth. All 60 combinations of 4 pipelines and 15 frequency bandwidths were subject to this extra selection loop. After omitting the one sample, the remaining samples were fed to 60 LOOCVs corresponding with each pipeline-filter combination. The AUC values of all 60 secondary LOOCVs were saved for further analysis. On each loop, the pipeline-filter combination with the highest AUC was then chosen to classify the omitted sample. Pipeline performance was assessed using the mean AUC value of each pipeline over all saved AUC measures. The same procedure was applied to frequency bandwidths. Permutation tests were used to evaluate significance of the pair-wise AUC differences.

2.9 Group Tests, Correlation with FTND and RSN Spatial Differences

For each cohort, rsFNC differences between groups were assessed with two sample t-tests and corrected using false discover rate (FDR) multi-comparison correction. In the mTBI cohort, the significant rsFNC differences in each pipeline were further analyzed and compared with the other pipelines. In the case of the smokers cohort, linear regression with the covariates sex, age and FTND was applied, but only to the 21 subjects in the SMK sample group. Regression results were used to determine the relationship between rsFNC and nicotine dependence. Finally, peak coordinates for subject specific RSNs were used to test spatial map differences among pipelines. In the case of simulation data, the RSNs displayed in Figure 3 correspond to an aggregation among all subjects and does not show subject specific differences. The Euclidean distance between the peak coordinates of spatial maps from each subject and the mean spatial map was used to assess spatial differences among subjects. This procedure was applied for all selected RSNs in the mTBI and smokers cohorts.

Figure 3.

Spatial content of the uncorrelated RSNs used in the simulation. The mean correlation between these two RSNs was low (ideally zero) in the HC samples. Simulated connectivity was achieved using these t-maps and space time regression to extract, modify and reintroduce artificial phase information. In this figure, images were cropped using t-value thresholds between 20 and 30 and the color bars indicate these levels.

3 Results

3.1 Simulations

Spatial content for the two gICA components selected for simulation is displayed in Figure 3. These RSNs were selected after running all HC samples from the mTBI cohort through all four pipelines and verifying that RSN correlations were not significantly different from zero (p<0.05) on any pipeline. The time courses of these two RSNs are the same ones used to obtain Figure 2. Group differences of head movement were not significant (p > 0.1) when assessed using mean FWD. The PR between artificial signal and time courses was kept at −9 dB. RSN correlation was manipulated using phase changes. Three simulations were then performed using this data. The first simulation separated the HC group into SP and SC, but did not modify the data before running the four pipelines. This first simulation constitutes the comparison baseline. The other two simulations enforced artificial correlations of 0.1 and 0.2 as described in the methods section. The three simulations were fed to all four pipelines. We estimated the amount of residual variance due to head movement using the correlation between rsFNC and mean FWD. Table I shows these results where significance was assessed using permutation tests. Only PC and PD had significant correlations with FWD in the simulation data.

Table I.

Correlations of rsFNC with FWD for all Pipelines

Cohort	PA	PB	PC	PD
Simulations (Sim.)
Sim. Correlation=0.0	0.16	0.17	*0.24	*0.26
Sim. Correlation=0.1	0.09	0.14	*0.33	*0.30
Sim. Correlation=0.2	0.07	0.10	*0.28	*0.19
mTBI ATT1 vs. ATT7	0.02	0.08	0.03	−0.02
mTBI CER4 vs. SEN2	0.13	0.10	−0.06	−0.02
Smokers	−0.14	−0.04	−0.06	−0.01

^*Significant at p<0.05 uncorrected

After running the LOOCV on the simulated data, we compared the AUC measurements obtained from each LOOCV loop among pipelines and among frequency bandwidths. These results are presented in Figure 4. The baseline case shows the AUC at no better than chance. PA and PB showed significantly larger AUC than PC and PD when added the simulated correlations. Frequency results show close to chance AUC for all frequencies at baseline. For the data with simulated correlations of 0.1 and 0.2, the AUC measure increases as higher frequencies are included. The largest AUC was measured when using the bandwidth [0.01 0.24] Hz.

Figure 4.

Pipeline and Frequency performances for simulated data. Simulated increments of connectivity were achieved by artificially adding time coherent signals where the correlation significance threshold (p<0.05) is 0.16. The strongest connectivity case with correlation of 0.2 demonstrates the case of a significant correlation. The case with correlation of 0.1 is an artificially simulated no-significant correlation. The baseline case show results for the case where no signal was added. Results shown are averages of either pipeline or frequency dimensions. Pipelines PA and PB exhibit better performance than PC and PD for cases where signals were introduced. Frequency results are different from chance only for the largest simulated connectivity of 0.2. The best frequency bandwidth in the 0.20 case was the range [0.01 0.22] Hz. There were no significant differences among bandwidths with high cut-off 0.20, 0.21, 0.22,0.23 and 0.24 Hz on all three bar plots. Otherwise, most of the AUC differences were significant in the frequency dimension. The number of significant differences was too high to fit in the figure.

Group difference analysis revealed more pipeline features as shown in Figure 5. The t-values follow the same increasing trend observed for the AUC graph in Figure 4 providing evidence for the observed AUC differences. Figure 5 also displays differences of peak spatial maps among pipelines for each of the two RSNs. PA and PB were significantly less affected by subject spatial content than PC and PD in both RSNs. In summary, results from PC and PD show larger effects related to head movement variance and smaller classification performance than PA and PB.

Figure 5.

t-values and mean spatial maps deviations for simulated data. As the case of No-Signal shows, without artificial manipulation the t-value is not significant for any pipeline. However, the RSN pair in this simulation was chosen to exhibit this characteristic. The two pipelines where MotReg (or SpkMotReg which includes MotReg) was performed before gICA (see PA and PB in Figure 1) were sensitive to the artificially simulated correlation. In the pipelines where MotReg was performed after gICA (see PC and PD in Figure 1), these were only sensitive if the simulated correlation was strong enough (0.20). This figure also shows mean deviations of subject-wise spatial maps. This deviation was calculated using the Euclidean distance from the peak coordinates of the mean spatial map (averaged over all subjects) to the peak coordinates of each subject spatial map. Significantly higher deviations were detected for PC and PD in ATT6. Deviations in ATT7 were not significantly different, but a trend of higher deviation can be seen in PC and PD.

3.2 The mTBI Cohort

Spatial content for the 53 RSNs considered in the mTBI cohort are displayed in Figure 6. These RSNs were found in all pipelines. Classification performance results can be found in Table II. The classification AUC reached a value of 80.2 % after optimally selecting pipeline and bandwidth for each LOOCV loop. In Figure 7, we provide histograms of selected pipelines and bandwidths. PA was selected in 85 out of 96 LOOCV loops. PB was chosen 11 times, but PC and PD were never chosen. The most frequently chosen bandwidth was 0.22 Hz (29 out of 96 times). The patterns observed in the histograms closely follow the mean AUC results, also displayed in Figure 7. PA and the bandwidth with cutoff 0.22 Hz achieved the highest mean AUC. Although histograms and mean AUC suggests that classification performance is best after using PA, there were no significant differences among the pipeline AUCs after a permutation test assessment. In the same fashion, bandwidths did not show significant differences of AUC.

Figure 6.

RSNs used in the analysis. This set obtained using gICA was selected based on the frequency content and visual inspection. Some RSNs in the CER group overlap with visual areas, but with a large peak region in the cerebellum and high temporal correlation with other cerebellar components. A total of 53 RSNs were identified in the mTBI sample and for all four pipelines. In the smokers sample, only 43 of these components could be matched. RSNs marked with an asterisk (*) were missing in the smokers sample. The match was confirmed by visual inspection and using a spatial correlation larger than 0.5.

Table II.

Classification Results for mTBI and smokers cohorts

Cohort	Number of Features	Sensitivity (%)	Specificity (%)	AUC (%)
mTBI	1378	83.3	77.1	80.2
Smokers	903	95.2	95.2	95.2

Figure 7.

Pipeline and Frequency Performance for the mTBI Sample. On each cross validation loop, the LOOCV optimization selected the pipeline and bandwidth combination with the best AUC measure. The top row displays the mean AUC performances obtained from all pipeline and bandwidth combinations applied by the LOOCV optimization. Permutation tests indicated no statistically significant differences. The bottom row shows the histograms of pipeline and bandwidth selection. Although AUC differences were not significant, the pipeline histogram follows the trend observed in AUC. PA is the most frequently selected pipeline with a higher mean AUC. Frequency bandwidths show a more uncertain distribution.

Mean rsFNC and t-value matrices are displayed in Supplementary Figure 1. Group differences of rsFNC were found in PA and PB where patients have increased connectivity for the CER4-SEN2 and the ATT1-ATT7 rsFNCs. The spatial content of CER4 is located in the middle part of the cerebellar vermis. SEN2 encompasses both the supplementary motor area (SMA) and the paracentral lobule areas. ATT1 embraces the precuneus and ATT7 the left angular gyrus. More details of these two rsFNCs are presented in Figure 8, where changes of t-value and p-value for different bandwidths are displayed. The significant t-values of PA and PB are very similar and exhibit robustness of p-value variability against frequency content. These two rsFNCs were not significant in PC or PD regardless of the selected bandwidth with the distinctive characteristic that corresponding p-values show larger variability. Although it is possible the results displayed in Figure 8 could be affected by head movement, none of the pipelines exhibited a significant correlation with mean FWD as shown in Table I. PA and PB exhibited the largest sensitivity to differences in rsFNC. In addition, bandwidth selection did not affect the significance of t-tests and produces little t-value variability. Spatial map differences were not significant among pipelines for this cohort.

Figure 8.

Spatial content of the significant rsFNC differences in the mTBI group. These two rsFNC are significant only in PA and PB, which are the pipelines where motion variance is removed before gICA. The t-values seemed unaffected by frequency. On the other hand, FDR corrected p-values show large variability with the bandwidth selection in PC and PD. Activation differences in the precuneus and angular gyrus areas (Olsen et al., 2014; Smits et al., 2009) suggest attentional deficits related to these areas. Abnormal rsFNC have also been found in the cerebellum (Nathan et al., 2014) and in other cases linked to post concussive complaints (Stevens et al., 2012).

3.3 The Smokers Cohort

Figure 9 displays the pipeline and frequency assessments for the smokers cohort. PA and the bandwidth [0.01 0.24] Hz exhibit the highest AUC for this cohort. Table I also shows head movement contamination results for each pipeline. No FWD differences could be found among pipelines after a permutation test. Final classification results are presented in Table II with higher AUC (95.2 %) than the mTBI cohort. Two sample t-tests were performed using this bandwidth resulting in several significant group differences. Mean rsFNC and t-value matrices for the smokers cohort are displayed in Supplementary Figure 2. Significant sample t-tests are provided in the Supplementary Table I. Of all the rsFNC showing group differences, only one has a significant correlation coefficient with FTND. The linear regression model also shows a significant relationship with FTND scores. When testing with the bandwidth [0.01 0.15] Hz, only PA remained significant. Further tests revealed that the FTND relationship with rsFNC was significant in PA no matter what choice of bandwidth was selected. The RSN pair is highlighted by a circle in Supplementary Figure 2 and the spatial content illustrated in Figure 10. One RSN (VIS 5) shows peak activation in the fusiform gyrus and the other one (SBC 2) in the putamen. We can see in Figure 10 that PC achieves the largest regression coefficient, but its significance is also one of the least robust to bandwidth selection. PB exhibits a similar lack of robustness and has the smallest regression coefficient. PA and PD has similar FTND coefficient, but PA achieves the smallest p-value of all pipelines at any bandwidth. RSN spatial differences were not significant among pipelines for this cohort.

Figure 9.

Pipeline and Frequency Performance for the Smokers Sample. Differences in mean AUC are not significant for either pipelines or frequency bandwidth. The preferences for PA and high frequency bandwidth displayed by the histograms are similar to that observed in the mTBI cohort.

Figure 10.

RSN with significant group difference corrected using FDR. In addition, this RSN pair exhibit an increase in rsFNC correlated with FTND. The spatial content of these networks suggests a resting state interaction between reward and visuospatial brain networks. The two plots on the right show changes against bandwidth selection of FTND regression coefficients and corresponding p-values for each pipeline. Although PC exhibits the largest effect size, the significance of the regression coefficient is dependent on the bandwidth. PB has the lowest effect size and also shows high instability with respect to significance. The effect sizes of PA and PD are very similar. The p-values in PA are the lowest and always significant in showing the highest level of robustness against selected bandwidth.

4 Discussion

Simulations provided the strongest evidence obtained in this work. Results from smokers and mTBI cohorts allowed us to confirm that simulations reflect what is observed in real data. Outcomes indicate that data preprocessing affects the average classification performance and the detection of rsFNC differences. Pipelines where head motion was corrected before gICA were preferred by the implemented pipeline selection. While it is known that linear regression might not completely remove non-linear effects of head movement (Satterthwaite et al., 2012), it is important to consider other factors. One possible explanation is rooted in the fact that regressions applied before gICA will work in a voxel-wise manner, but after gICA the regression operates on time courses obtained from aggregations over many voxels. Reconstructed time courses are the result of an operation where information and data have been reduced. Although gICA is in principle a linear process, the combination of multiple data reduction steps suggested by Calhoun et al. (Calhoun et al., 2001; Erhardt et al., 2011) leads to additional nonlinear effects. Consider a matrix X holding data to be fed into gICA. Motion variance regression, such as the one applied in PA, can be put in matrix form X = S + C where S is the matrix of regression residuals and C the matrix of regressed motion covariates. A data reduction operator R_n, such as principal component analysis, is then applied in the gICA pipeline where a set of n components (n>0) is removed. The data reduction operator leads to the inequality R_nX ≠ R_nS + R_nC indicating the existence of non-linearities. The differences between performing gICA on motion corrected R_nS or motion uncorrected R_nX data cannot be simplified nor fully described by linearity assumptions. A complete mathematical description of motion variance regression in combination with gICA can be highly complicated and is outside the scope of this paper. Lacking a more theoretical framework, we reduced preprocessing options to several different black boxes and compared the relationship between input and output of each one. The results from these black boxes indicate that processing head motion variance before or after gICA lead to different results in both temporal and spatial analyses. Better classification performance and larger rsFNC group differences may be linked to a reduced influence of head movement if motion parameters are processed before gICA. Reduced spatial map variability may also be related to the removal of head motion variance before gICA. In addition, results suggest that the position of spike processing in the pipeline is not critical. In general, the two pipelines where head motion was processed before gICA exhibit the best performance whether spikes were processed before or after gICA.

In the simulation, the two levels of rsFNC were chosen to test weak and strong rsFNCs. Weak rsFNC was set for a correlation higher than zero, but not strong enough to reach a statistical significance with a p-value larger than 0.05. The setup for strong rsFNC enforced a statistically significant correlation with a p-value smaller than 0.05. The threshold for a significant correlation is based on the number of time courses available to calculate the correlation. Simulation results suggest that some preprocessing methods may obscure weak rsFNCs. In Figure 4, only the results obtained after preprocessing with PA or PB delivered classification accuracy larger than 55% when the simulated rsFNC is 0.10. Although these classification performances are not higher than chance, results suggest that PA and PB would be a better choice for weak connectivity than PC and PD. The outcomes for the baseline case came out as anticipated. All pipelines delivered a classification close to chance in the baseline case, indicating the lack of difference between the two populations as it was preset in the simulation. The strong rsFNC case resulted in a classification larger than chance (> 65%) for pipelines PA and PB, but with a significantly lower performance for PC and PD. Nevertheless, all pipelines exhibited an AUC larger than 55% coinciding with the introduction of an artificial signal that simulates larger connectivity. While these results were obtained after artificially manipulating the data, which is an obligated step in our simulation procedure, the strict cross validation method in our pipeline assessment counteracted the effects of possible bias.

Differences among pipelines also influenced the estimation of spatial maps. Results displayed in Figure 5 show larger differences among the spatial maps of pipelines where motion parameters were not processed before gICA. Although the difference among pipelines is less than a voxel, motion variance produced some significant increments on spatial maps variability. The large correlation with FWD observed for PC and PD in Table I and the spatial maps results suggests that motion variance contamination can have a significant presence if not regressed in the early stages of the preprocessing.

More information about the differences among pipelines can be grasped from the t-value displayed in Figure 5. PA and PB shows an increasing trend of group difference that follows the enforced simulated connectivity, closer than the results obtained from PC and PD. The t-values of PC and PD were relatively low and lacked significance except for the strong rsFNC case, suggesting a larger difficulty in assessing existing group differences for the weak rsFNC case. The comparison of t-test results among pipelines suggests that head motion should be addressed before smoothing and gICA to account for a better detection of small rsFNCs. This order of preprocessing steps agrees with observations presented in ROI studies, where it is known that despiking and removal of motion variance early in the preprocessing pipeline can reduce bias in rsFNC when applying ROI methods (Power et al., 2014). Although gICA and ROI approaches are different techniques, a recent study that considered only healthy controls suggests that (similar to the case of seed methods) despiking before the gICA step may reduce bias in the final rsFNC measures (Damaraju et al., 2014). Different from the mentioned study, we did not consider pipelines without despiking or without motion correction because we are assuming that both steps must be carried out in practical applications. Another important characteristic of our study is the focus on the analysis of patient-control differences. Previous studies on preprocessing steps used only healthy controls with a different criteria to form sample subgroups (Damaraju et al., 2014; Power et al., 2014).

In the real data, the classification performance obtained for the mTBI cohort achieved an accuracy of 80.2%, which is substantially higher than chance. In the smokers cohort, the performance was higher than for the mTBI cohort. These results are promising and suggest that fMRI could be highly informative in detecting mTBI or nicotine dependence once an experiment design targeting diagnosis is implemented. Mean classification performance was not different among pipelines or frequencies, but the trend to select PA and include high frequencies coincides with the simulation results. With respect to the rsFNC group differences and nicotine dependence measure, PA results show higher sensitivity and robustness against frequency content. This sensitivity can be explained in part by differences of p-value spectrums among pipelines and the dependence of FDR multi-comparison correction on these spectrums. Even when ground truth is not known in real data, the sensitive and robust behavior of PA observed in simulations, mTBI and smokers cohorts supports correcting for head movement before smoothing and gICA.

The current analysis included information from many parts of the brain to assess the behavior of different preprocessing and the power to detect group differences. Of all 1378 considered RSN pairs, few exhibited an rsFNC difference between HCs and mTBIs. This fact seems antagonistic with respect to results reported earlier by Vakhtin et al. (Vakhtin et al., 2013), where several rsFNC group differences were found through the brain; however, in that study the number of test comparisons was much lower. In contrast to the Vakhtin results, the study performed by Mayer et al. (Mayer et al., 2014) sought to achieve significance after multi-comparison correction primarily because of the larger number of RSNs considered. Although other processing differences can be mentioned, that study processed motion variance after gICA, coinciding with the lack of significant group differences also observed in the present analysis. Only by processing motion variance before gICA we could observe rsFNC changes for the mTBI cohort. The significance of rsFNC differences depended little with bandwidth in these two examples. In addition, no evidence of significant head movement noise was found. Based on these two results, we believe that head movement and other nuisance signals were not major players in the observed rsFNC differences. In favor of the revisited data preprocessing applied in this report is the fact that we followed novel recommendations in the field (Damaraju et al., 2014; Power et al., 2014) that aimed at decreasing the effect of movement artifacts. In this manner, we expected to reduce the impact of motion on our statistical analysis.

Abnormal activation of the precuneus related to working memory and attention have been associated with post-concussive syndrome (Smits et al., 2009). Sustained attention impairments in patients with brain injury have been related to increase activation in the precuneus (Bonnelle et al., 2011). Abnormal activation has also been reported in the left angular gyrus during verbal encoding (Olsen et al., 2014; Strangman et al., 2009). Based on the previous literature, the brain areas in ATT1-ATT7 with significant rsFNC difference might be involved in attentional dysfunction of the mTBI population. By processing motion variance before gICA, we could find a significant change in the rsFNC between the cerebellum and the SMA/paracentral lobule RSNs. After confirmation that frequency does not affect significance, several factors that differ from previous work can be mentioned. The most important difference is the small amount or the complete lack of cerebellum RSNs in previous studies. In this vein, Nathan et al. (Nathan et al., 2014) found increases in rsFNC linked to the cerebellum (as well as DMN and SMA) given that more cerebellar areas were included. Another study found functional connectivity of the cerebellum among those correlated with post concussive complaints (Stevens et al., 2012).

Pipeline assessment in the smokers cohort barely made use of PB, PC and PD as the pipeline usage histogram illustrates in Figure 9. The high rate of PA selection suggests that we could ignore the other three pipelines and use PA for all LOOCV loops. Results from the mTBI cohort suggest that the influence of sex, age and other possible sources of bias may be minimized by the enforcement of matched samples and cross validation. The high classification accuracy obtained in the smokers group could be caused by the selection of samples with high nicotine dependence. Observed accuracy in this cohort may not be a strange outcome since high classification performance has been previously observed in a similar population setup by another study that used a dual regression technique and a lower number of RSNs (Pariyadath et al., 2014). Different from the Pariyadath study, where maximum accuracy reached 78.6%, the number of RSNs in the present analysis is larger and this may account for the observed higher accuracy of 95.2%. Another difference resides in the method used to find biological causes of rsFNC variations, which in our analysis is based on group differences and their correlation with FTND (in the training data). Although FDR was applied only to group differences (p-values from t-tests), the correlation of nicotine dependence with the rsFNC between putamen and fusiform gyrus was independent of the CTR group, thus providing evidence of a strong relationship as useful information for the classification algorithm. The putamen is part of a dopaminergic network involved in anticipation of reward and related emotions. Increases in the functional connectivity of this reward brain area have been associated with substance use disorders including cocaine (Konova et al., 2015) and heroine (Schmidt et al., 2015). Along with other areas involved in planning, attention and visual processing, higher activation of the putamen in smoking abstainers is related to increased salience of craving cues (McClernon et al., 2009). The role of putamen as a hub of information and its involvement in drug seeking behavior (Konova et al., 2015) suggests that craving may produce functional changes affecting the putamen’s synchrony with other parts of the brain. The information flow does not only involve reward pathways, such as the mesocorticolimbic system, but recruits coherent involvement of the visuospatial system increasing attention to smoking cues (Due et al., 2014). The connection between reward and visual networks (in relation to craving) increases the activation of relevant brain areas including the putamen and fusiform gyrus (Ely et al., 2015). In general, evidence from previous studies supports the existence of a relationship between dopaminergic and visuospatial functions in nicotine dependence. Results in this work indicate that only through the use of an appropriate preprocessing is it possible to observe significant changes (after multi-comparison correction) in the synchronic function of the pair putamen-fusiform gyrus.

One important limitation of our study was the available frequency range. BOLD signal content has been found at high frequencies (Chen and Glover, 2015) and up to 0.8 Hz (Boubela et al., 2013; Lee et al., 2013a). Our study was limited in this respect to 0.25 Hz. Future work should explore different frequency characteristics in relation to the initial independent variable. Another important parameter with different choices is despiking, but in this work we did not explore all available options. Some studies suggest to remove spike time points by replacing the spike point with an interpolated value extracted from surrounding (no spike) times values (Allen et al., 2011). Other authors suggest excluding the spike time course from the analysis (Grouiller et al., 2011). We choose to remove spiky time courses, attempting to reduce as much as possible the influence of spikes. An interpolation process used to replace spiky values preserved time continuity among the samples and minimized spike influence in non-spiky time courses after filtering. The spiky time point was excluded from the final correlation. We also need to mention that spikes were detected using DVARS (Power et al., 2014), but other options can be adopted including large deviations of FWD (Power et al., 2012). Although we considered spatial map variability, the Euclidean distance used to measure spatial deviations leaves out many important parameters. Items that were not considered include location of the spatial map, distance between RSNs, shape and area of the spatial map, and multiple brain areas. These parameters require a study focused on the relationship between spatial and temporal differences among subjects. Finally, we need to consider that all classification procedures were performed solely for pipeline assessment and discarded particularities important for subject diagnosis. The most notorious problem is that many non-artifact RSNs were discarded because they could not be replicated in all four gICA. Discarded RSNs could have provided useful information for classification purposes. Future studies that focus in diagnosis can reintroduce variations from some of the RSN discarded here to obtain even high accuracies.

5 Conclusion

Results presented in this work show relationships among classification performance, group differences, correlation with non-categorical measures, frequency content and pipeline choices. The optimization of pipeline and frequency bandwidth for the purposes of classification tends to select the pipeline where motion variance is regressed out before gICA (PA and PB in Figure 1) and bandwidths with relatively high frequency content. The rsFNC group differences in the mTBI cohort and correlation with FTND in smokers add supporting evidence to indicate that PA is a good choice for preprocessing. Simulations suggest that PA and PB are more sensitive at detecting rsFNC differences when the signal is small. In real data, the influence of this sensitivity allowed for the detection of higher classification performance and significant group differences. Although simulation results indicate that including higher frequencies might be convenient, care must be taken to avoid frequencies containing known nuisance signals. Given that bandwidth choice seems to vary between 0.10 Hz and 0.15 Hz in the literature (Allen et al., 2011), robustness against frequency indicates that results obtained from using any of these two filter bandwidths should not be rejected on a frequency basis without a more careful assessment.

Supplementary Material

2

Supplementary Figure 1. rsFNC matrices for the mTBI sample and corresponding t-value matrices. The bandwidth selected in this figure is the [0.01 0.15] Hz. Two rsFNC differences were significant in PA and PB. The triangles point to ATT1-ATT7 and the circles to CER4-SEN2 rsFNCs. The significance of these t-values varies little with bandwidth. No significant t-value was observed for PC or PD.

3

Supplementary Figure 2. rsFNC matrices for the Smokers sample with t-value matrix and significant correlation with FTND. The t-matrices indicate patterns of increased and decreased rsFNC. Significant t-values are provided in Supplementary Table I. For this cohort, we are focusing on the rsFNC difference that shows a relationship with nicotine dependence. The RSN pair VIS5 vs. SBC2 have an rsFNC significantly correlated with FTND, which is highlighted by a dark circle.

Highlights.

• We analyze different preprocessing pipelines for group independent component analysis (gICA).

• We evaluate each pipeline using classification algorithms, leave-one-out cross validation and statistical methods such as regression, correlation and t-test.

• The data pinpoint a strategy that best seems to prepare fMRI data for gICA.