Repeatability and reliability of GABA measurements with magnetic resonance spectroscopy in healthy young adults

Abstract

Purpose:

Gamma-Aminobutyric acid (GABA) abnormalities have been implicated across neuropsychiatric disorders. Despite substantial interest in probing GABA in vivo, human imaging studies relying on MRS have generally been hindered by technical challenges, including GABA’s relatively low concentration and spectral overlap with other metabolites. While past studies have demonstrated moderate-to-strong test-retest repeatability and reliability of GABA within certain brain regions, many of these studies have been limited by small sample sizes.

Methods:

GABA+ (macromolecular-contaminated) test-retest reliability and repeatability were assessed via a MEGA-PRESS MRS sequence in the rostral anterior cingulate cortex (rACC; n=21) and dorsolateral prefrontal cortex (dlPFC; n=20) in healthy young adults. Data were collected on a 3T scanner (Siemens Prisma) and GABA+ results were reported in reference to both total creatine (GABA+/tCr) and water (GABA+/water).

Results:

Results showed strong test-retest repeatability (coefficients of variation; mean GABA+/tCr CV = 4.6%; mean GABA+/water CV = 4.0%) and reliability (intraclass correlation coefficients; GABA+/tCr ICC = 0.77; GABA+/water ICC = 0.87) in the dlPFC. The rACC demonstrated acceptable (but comparatively lower) repeatability (mean GABA+/tCr CV = 8.0%; mean GABA+/water CV = 7.5%), yet low-moderate reliability (GABA+/tCr ICC = 0.40; GABA+/water ICC = 0.44).

Conclusion:

The present study demonstrated excellent GABA+ MRS repeatability and reliability in the dlPFC. The rACC demonstrated inferior results, possibly due to a combination of shimming impedance and measurement error. These data suggest that MEGA-PRESS can be utilized to reliably distinguish participants based on dlPFC GABA+ levels, while mixed results in the rACC merit further investigation.

1. Introduction

Gamma-Aminobutyric acid (GABA) has been implicated in the pathogenesis of a range of psychiatric disorders including major depressive disorder (MDD) and schizophrenia.^1–4 Prior studies using MRS have identified decreased GABA in the occipital cortex⁴ (OCC), anterior cingulate cortex² (ACC) and dorsolateral prefrontal cortex³ (dlPFC) in individuals with MDD, in line with reports of reductions in GABAergic interneurons in the dlPFC emerging from post-mortem studies.⁵ Likewise, post-mortem and animal studies indicate possible GABAergic dysfunction in schizophrenia, although in vivo studies remain equivocal.^1,4,6 Despite evidence of GABAergic abnormalities across these disorders, the precise mechanisms of GABA dysfunction remain unclear. Continued investigation of GABA in vivo is therefore necessary to elucidate its possible contribution to the etiology and maintenance of psychiatric illness.

MRS has become widely used for GABA quantification, as its non-invasive design paired with recent technological advances make it a particularly attractive option.^4,7 Nevertheless, MRS quantification of GABA has been challenged by the metabolite’s relatively low concentration, considerable spectral overlap with other metabolites, and macromolecular contamination.^7,8 Given these difficulties, studies testing the reliability and agreement of MRS-derived GABA levels are critical to ensuring appropriate interpretation of anomalies that may be found in the context of pathology.

As summarized in Table 1, past research has demonstrated moderate to strong test-retest repeatability and reliability of GABA MRS in the cingulate cortex and OCC.^8–21 Several additional studies have examined other areas, including motor regions,^21–24 Broca’s area,⁹ and the dlPFC,^22,25,26 among others.^27–29 However, many previous studies have been limited by small sample sizes. Accordingly, replication studies are necessary, particularly as the homogeneity of the magnetic field has been shown to vary with between-subjects factors such as sinus shape and size, thereby impacting shimming and MRS data quality.^30,31

TABLE 1.

Prior GABA MEGA-PRESS MRS Repeatability and Reliability Studies at 3T

Reference	Brain Region	N	Statistical Methodology	Cr-Referenced GABA Findings	Water-Referenced GABA Findings
Baeshen et al. (2020)	PCC	18	Wilcoxon signed-rank tests	—
			Pearson’s r		Z = −1.85
			ICC		r = 0.54
			CV
					ICC = 0.5
			BA Plots
					CV = 8.8%
Ferland et al. (2019)	Left sensorimotor cortex (SMC)	10
			Pearson’s r	r = 0.65	r = 0.82
			ICC	ICC = 0.67	ICC = 0.81
			CV	CV = 14%	CV = 10%
Mikkelsen et al. (2019)	Medial parietal lobe	284*	CV	—	GABA+ CV = 16.9%
					GABA’ CV = 28.8%
Brix et al. (2017)	ACC	21	CV	—	ACC: CV = 6–14%
	Left Broca’s area				Broca’s area: CV = 4–7%
Mikkelsen et al. (2017)	Medial parietal lobe	272*	CV	GABA+ CV = 12.0%	—
				GABA’ CV = 27.6%
Yasen et al. (2017)	dlPFC	13	ICC	dlPFC: ICC = 0.97; CV = 27.9%	—
	Primary motor cortex		CV	Motor: ICC = 0.93; CV = 21.0%
Greenhouse et al. (2016)	Lateral PFC (lPFC)	28		lPFC: r = 0.75, CV = 4.6%	—
	SMC		Pearson’s r	SMC: r = 0.64, CV = 3.9%
	Dorsal premotor cortex (dPMC)		CV	dPMC: r = 0.63, CV = 3.9%
	OCC			OCC: r = 0.52, CV = 5.3%
Mikkelsen et al. (2016)		15		—	OCC: GABA+ ICC = 0.67; CV = 4.0%
	OCC		ICC		OCC: GABA’ ICC = 0.72; CV = 8.6%
	ACC		CV		ACC: GABA+ ICC = 0.16; CV = 14.8%
					ACC: GABA’ ICC = 0.41; CV = 12.6%
Shungu et al. (2016)	dlPFC	6	Pearson’s r	r = 0.98	r = 0.98
			ICC	ICC = 0.97	ICC = 0.98
			CV	CV = 1.72%	CV = 1.25%
Long et al. (2015)	Cerebellar dentate	5	CV	Left, corrected: CV = 6.1–13.4%	Left, corrected: CV = 5.3–11.9%
				Right, corrected: CV = 5.0–8.0%	Right, corrected: CV = 5.0–8.5%
Gaetz et al. (2014)	ROIs in these areas:	5	CV		—
				Motor: CV = 9%
	Motor			Visual: CV = 11%
	Visual			Auditory: CV = 9%
	Auditory
Near et al. (2014)	OCC	17	Pearson’s r	r = 0.53	—
			ICC	ICC = 0.52
			CV	CV = 4.3%
Harada et al. (2011)		8	ICC	Overall ICC = 0.72	—
	Lentiform nuclei
	Left frontal lobe
	ACC
Geramita et al. (2011)	ACC	10	ICC	ACC: ICC = 0.50; CV = 6.5%	ACC: ICC = 0.79; CV = 5.3%
	Right Frontal White Matter (rFWM)		CV	rFWM: ICC = −0.35; CV = 8.2%	rFWM: ICC = 0.48; CV = 8.7%
O’Gorman et al. (2011)	dlPFC	14	CV	—	CV = 7%
Bogner et al. (2010)	OCC	11	CV	Processed with integration: CV = 17.0%	Processed with integration: CV = 17.8%
				Processed with fitting: CV = 13.3%	Processed with fitting CV = 15.0%
Evans et al. (2010)	OCC	8	CV	—	OCC: CV = 6.5%
	Precentral gyrus				Precentral gyrus: CV = 8.8%

^*These papers were both based on one large dataset from a study across 24–25 different research sites.

Note. For the purposes of this table, “water-referenced GABA findings” include studies utilizing an internal water reference and/or water scaling factor; PCC = posterior cingulate cortex; ICC = intraclass correlation coefficient; CV = coefficient of variation; BA = Bland-Altman; ACC = anterior cingulate cortex; dlPFC = dorsolateral prefrontal cortex; GABA+ = GABA collected with macromolecular contaminants; GABA’ = GABA collected with macromolecular suppression; PFC = prefrontal cortex; OCC = occipital cortex; ROI = region of interest.

MRS test-retest studies have also varied in their use of statistical methods and terminology. For example, many past studies used the terms “repeatability,” “reliability,” “reproducibility,” and “agreement” seemingly interchangeably, leading to confusion about the exact constructs and theoretical underpinnings in question. In the present study, we use “repeatability” and “agreement” to refer to the degree of similarity between multiple measurements taken from a subject under identical conditions.³² In contrast, we use “reliability” to refer to the capacity of the methodology to distinguish among subjects.^32,33 Notably, many test-retest studies have reported both repeatability metrics such as coefficients of variation (CVs) and reliability metrics such as intraclass correlation coefficients (ICCs) with minimal discussion of the fundamental differences between these types of measures. We believe further clarity regarding the respective implications of each metric is needed within the MRS literature.

In this context, the main goal of the present study was to evaluate rostral ACC (rACC) and dlPFC GABA test-retest repeatability and reliability with MRS in a larger sample of healthy young adults, with careful distinction and discussion of the agreement and reliability of GABA measurements.

2 |. METHODS

2.1 |. Sample

Participants (N = 35) were control subjects with no history of psychopathology drawn from a larger study investigating sex differences and stress in young adults with MDD. Absence of current or past psychopathology was ascertained using a semi-structured clinical interview (Structured Clinical Interview for the DSM-5, SCID-5),³⁴ which was performed by a PhD- or MA-level clinician. All participants were recruited from the greater Boston community and were between the ages of 18 and 25 (mean (M) = 21.2 years, standard deviation (SD) = 2.4 years), split evenly by sex (51% female). The self-identified racial makeup of the sample was 51% White, 29% Asian, 9% Black, and 9% biracial, with 3% (one participant) declining to answer. Eleven percent identified as Hispanic or Latinx.

All participants were right-handed with no significant medical history or use of psychotropic medications. Given the well-established impact of alcohol use on GABA concentrations, participants with greater than five lifetime alcohol-related blackouts or an alcohol use disorder were excluded.^35–38 An initial screening visit was conducted wherein written informed consent was obtained in compliance with the requirements of the Partners Human Research Committee. Eligible participants completed a 2-hour MR scan within a month of the screening session. As menstrual cycle has been shown to impact GABA concentrations in the ACC, all females were scanned during their follicular phase.³⁹

2.2 |. MRI acquisition

Structural and functional images as well as MRS were acquired via a Siemens 3T Prisma scanner (Siemens AG, Germany) operating at 123 MHz for proton imaging and spectroscopy, using a 64-channel, phased-array head coil for reception and a body coil for transmission. A set of T1-weighted high-resolution 3D structural images were collected with a multi-echo MPRAGE sequence (TR = 2530 ms; TE = 1.69, 3.55, 5.41, 7.27 ms; slice thickness = 1mm; total number of slices = 176; flip angle = 7.0°; field of view = 256mm; voxel dimensions = 1×1×1mm).

The T1-weighted structural images were used to place a voxel in the rACC (17.5 ml; 35×20×25mm³) and left dlPFC (18.75 ml; 25×30×25mm³) for MRS data collection (Figure 1). Data were collected from both voxels for all participants, with 26 participants having test-retest data collected for one voxel and nine participants for both (depending on scan time constraints). Test-retest data (in the same scanning session) were acquired for 24 participants in the rACC and 20 in the dlPFC. MRS data were collected immediately after a high-resolution localizer (namely the 2D high-resolution images repartitioned from the MPRAGE images) and before any other scans, such that no frequency drift was seen due to gradient heating.

FIGURE 1.

Images illustrating the voxel placement for the (A) left dorsolateral prefrontal cortex (dlPFC) and (B) rostral anterior cingulate cortex (rACC). Voxel placement is presented in sagittal, coronal, and axial views on a single subject for each region.

Proton GABA+ (macromolecular-contaminated) measurement employed a MEshcher-GArwood Point RESolved spectroscopy (MEGA-PRESS) sequence obtained from the University of Minnesota with the acquisition frequency sitting at 3.0ppm and frequency-selective editing pulses, each with a duration of 17ms,⁴⁰ alternatively at 1.9 ppm (on) and 7.5 ppm (off) interleaved with the averages.^7,40–42 MEGA-PRESS is an established MRS acquisition protocol for GABA detection that has demonstrated superior GABA test-retest reliability compared with other sequences.^8,43 Shimming of the magnetic field within the prescribed voxel was performed using a vendor-provided 3D shimming routine designated for the human brain region followed by manual adjustment as needed (performed by the same MRS physicist for all participants (CZ)). The full width at half maximum (FWHM) was measured from both the Siemens console (rACC: M = 17.47 Hz, SD = 1.92 Hz; dlPFC: M = 15.95 Hz, SD = 2.14 Hz) and the unsuppressed water peak (rACC: M = 7.82 Hz, SD = 1.21 Hz; dlPFC: M = 7.69 Hz, SD = 1.27 Hz). A VAPOR (VAriable pulse Power and Optimized Relaxation delays) module was utilized to achieve water suppression.⁴⁴ Following shimming, carrier frequency was adjusted, flip angles and water suppression were optimized, and the MEGA-PRESS spectra were collected at TE = 68ms, TR = 3000 ms, spectral bandwidth = 1.2 kHz, 2048 data points, readout duration = 1706 ms, total number of signal averages = 192, total scan duration = 10 min with applied directions of the slice-selecting gradients identical across subjects (sagittal, R->L; coronal, A->P; transversal, F->H). The test and retest data acquisitions took place back‐to‐back, with no delay or re‐acquisition of anatomical images between data collections. Therefore, correction for the voxel tissue composition was not necessary as the voxel locations were identical between each participant’s test and retest scans. After MEGA-PRESS data collection, unsuppressed water signal positioned at the location of the GABA 3ppm resonance was collected with an offset frequency of −1.7ppm for eddy current correction and quantification purposes, after accounting for the error of chemical shift replacement (≤2%).

2.3 |. GABA+ quantification

The MRS data were exported in .IMA format and processed using FID-A.⁴⁵ To quantify neural GABA+ concentrations, the 96 edit-on and 96 edit-off FIDs were corrected for possible frequency and phase drift, Gaussian filtered (2 Hz) and Fourier transformed prior to grouping on and off spectra and taking the corresponding edit-on and -off spectra differences. The grouped edit-off and difference spectra, in conjunction with corresponding unsuppressed water signals, were imported into Linear Combination Model⁴⁶ (LCModel, version 6.3–1N) to fit the following metabolites: total creatine (tCr), total choline (tCho), glutamate (Glu), Glx (Glu + glutamine; Gln), myo‐inositol, and NAA from the edit-off spectrum, and GABA+ from the difference spectrum (Figure 2). The parameter sptype=‘mega-press-3’ was used for GABA+ quantification, and the baseline was stiff and flat following the LCModel default settings. The difference spectra were fitted with an in-house simulated basis set of metabolites using GAMMA.⁴⁷ No basis was utilized for the co-edited 3 ppm macromolecule signal, as the overlap with GABA could result in unreliability in fitting.

FIGURE 2.

GABA+-edited (difference) spectrum showing metabolite fitting lines as estimated with LCModel, depicting the GABA+-edited spectrum (dark blue), fitting line (orange), total NAA (tNAA; purple), GABA+ (light blue), glutamate+glutamine (Glx; green), and residuals (grey).

GABA+ concentrations are reported as GABA+/tCr (a ratio of GABA+ to total Creatine) and GABA+/water (a ratio of GABA+ to water multiplied by a scaling factor, reported in mM). LCModel fitting of the MRS data was assessed for quality based on Cramer-Rao Lower Bound (CRLB) values of <15% and signal-to-noise ratios of >20, with one participant excluded. Additionally, spectra were visually assessed prior to analyses by MR physicists (XC and FD) for severe baseline distortion, excluding two additional participants (final rACC n = 21 and dlPFC n = 20).

2.4 |. Statistical analyses

Statistical analyses were performed using SPSS Statistics, version 24.0. To assess closeness of agreement, within-subject CVs were calculated by first computing the standard deviation of the paired measurements divided by the mean for each subject, and then averaging across participants for each voxel:

$$\text{CV}=\frac{SD}{\overline{x}}\times 100=\frac{\sqrt{{\sum }_{i=1}^{i=2}{\left(x_i-\overline{x}\right)}^2}}{\overline{x}}\times 100,$$ [1]

where x_i is the value of the ith scan for each participant and SD is the sample standard deviation for each participant. CVs are the most commonly reported metric in the MRS test-retest literature,^8,48 and provide an assessment of within-subject measurement agreement that is independent of the range of values in the sample.

Next, Bland-Altman (BA) plots were created to visually examine measurement repeatability.^49–51 BA plots provide a useful complement to agreement metrics such as CVs, allowing for easy identification of outliers and systematic trends in measurement error. Following Bland and Altman,^49,50 the difference between the paired measurements was plotted against the mean for each participant (BlandAltmanLeh package in R, version 3.5.3).⁵² One-sample t-tests were used to check the assumption that this mean difference was not significantly different from zero.^49,50

To calculate limits of agreement (between which 95% of the datapoints are expected to lie) Bland and Altman^49,50 recommend using repeatability coefficients for test-retest studies. These limits assume an interscan mean difference of zero, which is most appropriate for studies using the same measurement technique on the same subjects, provided it is reasonable to assume no systematic mean difference in observations over time. Thus, repeatability coefficients were calculated as the t critical value (for the upper and lower 2.5% tails) for each voxel multiplied by the standard deviation of the interscan differences, with an assumed mean difference set at zero^49,50 as follows:

$$\text{Repeatability}\text{coefficient}=\pm t_{0.975}\sqrt{\frac{{\sum }_{j=1}^{j=n}{\left(x_{1j}-x_{2j}\right)}^2}{n-1}},$$ [2]

where j indexes the individual participants, and n = total number of participants for each voxel. The repeatability limits of agreement and a zero line of no difference were added to the plots to aid in the assessment of the magnitude of measurement error and identify outliers.

To investigate reliability, single-rating, absolute-agreement, two-way mixed-effects ICCs and their 95% confidence intervals were calculated.⁵³ ICCs assess how reliably an instrument distinguishes between subjects⁵⁴ and are calculated as the between-subject variance divided by the total variance. They are widely used by different fields of study to measure reliability.^8,53–55

3 |. RESULTS

Within the dlPFC, the mean CV for GABA+/tCr and GABA+/water were 4.6% and 4.0%. GABA+/tCr and GABA+/water produced ICCs of 0.77 and 0.87, respectively (Table 2). Both the GABA+/tCr and GABA+/water BA plots showed 95% of participants within the repeatability coefficients (one outlier each; Figure 3), as is expected with unbiased, normally distributed measurement error.⁴⁹ For both GABA+/tCr and GABA+/water, the mean interscan differences were not significantly different from zero (mean [95% CI] for GABA+/tCr: −0.0004 [−0.0072, 0.0063], P = 0.89; for GABA+/water: 0.001 [−0.045, 0.047], P = 0.96).

Table 2.

GABA+/tCr and GABA+/water Test-Retest Results

Voxel & Ratio		n		CV			ICC
				Mean	Range		Value	95% CI
dlPFC GABA+/tCr		20		4.60%	0.03–11.68%		0.77	0.51, 0.90
dlPFC GABA+/water		20		3.97%	0.24–11.02%		0.87	0.70, 0.95
rACC GABA+/tCr		21		8.05%	1.27–21.99%		0.40	0.01, 0.70
rACC GABA+/water		21		7.49%	0.15–22.68%		0.44	0.05, 0.72

Note. tCr = total creatine; ICC = intraclass correlation coefficient; CV = coefficient of variation; dlPFC = dorsolateral prefrontal cortex; rACC = rostral anterior cingulate cortex

FIGURE 3.

Bland-Altman plots for (A) dlPFC: GABA+/water, (B) dlPFC: GABA+/tCR, (C) rACC: GABA+/water, and (D) rACC: GABA+/tCR. The plots depict the mean of scan 1 and scan 2 on the x-axis and the interscan difference on the y-axis. The solid line represents the zero line of no difference. The dotted lines depict limits of agreement, calculated with repeatability coefficients (the critical t value multiplied by the standard deviation of the interscan differences).

Within the rACC, the mean CV for GABA+/tCr and GABA+/water were 8.0% and 7.5%. The rACC showed lower ICC values, with GABA+/tCr and GABA+/water ICCs of 0.40 and 0.44. respectively. Within the BA plots, 95% of measurements fell between the repeatability coefficients for GABA+/tCr (one outlier), while the GABA+/water plot showed 91% of measurements between the repeatability coefficients (two outliers). The mean interscan differences were again not significantly different from zero (mean [95% CI] for GABA+/tCr: 0.009 [−0.002, 0.022], P = 0.10; for GABA+/water: 0.07 [−0.01, 0.16], P = 0.08).

4 |. DISCUSSION

The present study demonstrated greater GABA+ MRS test-retest repeatability and reliability within the dlPFC as compared to the rACC. These strong findings in the dlPFC replicated results of past studies in a larger cohort.^18,22,25,26 Although reliability thresholds for ICCs are not well-established within the broader literature, dlPFC ICCs were well above the 0.75 threshold characterized as “excellent” by both Baeshen et al.⁸ and Cicchetti et al.⁵⁶ and “good” by Koo and Li,⁵⁵ suggesting that MEGA-PRESS can be used to reliably distinguish between the dlPFC GABA+ levels of healthy subjects. Measurement agreement in the dlPFC was also strong. Thresholds for interpreting CVs are rarely discussed in the MRS literature, but the current averaged CVs were far below those previously used (e.g., 20% used by Baeshen et al.⁸). Additionally, BA plots revealed few outliers with no systematic trends in measurement error, further indicating strong agreement.

Overall, the rACC demonstrated lower agreement and reliability than the dlPFC, possibly related to the nearby sinus cavity affecting shim quality. Indeed, the mean rACC FWHM calculated from both the Siemens console and the final data (peak of the unsuppressed water scan) was higher than in the dlPFC, though this difference was not significant when calculated from the final data (P > 0.7). Measures of repeatability (i.e., averaged CVs and BA plots) yielded stronger results than measures of reliability (i.e., ICCs) in the rACC, highlighting the importance of reporting both metrics. Averaged CVs demonstrated moderate agreement, and BA plots revealed few outliers and no bias in measurement error. Meanwhile, rACC ICCs were in the low-moderate range.^8,55,56

To further contextualize the principal measures of agreement and reliability, it is useful to examine the mathematical structure of the CV and ICC. The CV depends exclusively on the within-subject variance, while the ICC is dependent on both the between- and within-subject variances. Thus, as demonstrated by Bland and Altman,⁵⁷ for a given level of measurement repeatability (e.g., CV), there can be marked variability in the ICC depending on the range of values within the chosen sample. In the present study of healthy young adults, the somewhat truncated range of observed rACC GABA+ concentrations may therefore partially explain the poor ICCs. Furthermore, it should be considered that the dependence of the CV and ICC on the within-subject variance is not symmetric. Specifically, the ICC decreases as the within-subject variance increases, whereas the CV increases in proportion to the square root of the within-subject variance. Thus, increases in CV that reflect a modest increase on the square root scale can have a much larger effect on decreasing the ICC. In the context of the present study, the impact of greater within-subject variance shown in the rACC would therefore be magnified in the ICC calculations as compared to those of the CV.

The results of this study should be considered with respect to its limitations, including the use of only two consecutive scans to assess test-retest repeatability and the inclusion of only healthy participants. Keeping the participant in situ between scans obviated the need to reposition the voxel for the second scan, thereby reducing measurement error and likely increasing agreement. Thus, between-group comparisons of GABA concentrations may be subject to greater noise related to voxel placement than demonstrated by these results. In addition, the study sample included only young adults without a history of psychopathology, who may show a narrower range of GABA levels compared with the broader population. As discussed above, this limited variability likely lowered the estimate of reliability based on the ICC.

Future research with a larger sample of psychiatric cases and healthy controls is needed to more thoroughly assess the test-retest reliability of rACC GABA across different population groups. Investigation is also needed to improve techniques for macromolecular suppression with MEGA-PRESS. GABA+ concentrations with MEGA-PRESS in the present study and others are likely overestimated due to macromolecular contamination, and current techniques for suppression of this contamination pose challenges.^7,8 Macromolecules may also vary with factors such as age or brain region, potentially increasing the difficulty of interpreting GABA findings.^58–60 Thus, improving techniques for macromolecular suppression remains a critical area of future work.

Despite these limitations, the present study provides evidence of excellent GABA+ test-retest repeatability and reliability in the dlPFC, and moderate repeatability in the rACC in a larger healthy sample. These results provide important context for future studies of clinical populations and further methodological work.

DATA AVAILABILITY STATEMENT

The data presented here are available at the NIMH Data Archive (https://nda.nih.gov/edit_collection.html?id=2485).