Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2014 Aug 18.
Published in final edited form as: Neuroreport. 2013 Aug 7;24(11):626–630. doi: 10.1097/WNR.0b013e3283637845

Between Site Reliability of Startle Prepulse Inhibition Across Two Early Psychosis Consortia

Kristin S Cadenhead 1,2,✉, Jean Addington 3, Tyrone D Cannon 4, Barbara A Cornblatt 5, Camilo de la Fuente-Sandoval 6, Dan H Mathalon 7, Diana O Perkins 8, Larry J Seidman 9, Ming Tsuang 1, Elaine F Walker 10, Scott W Woods 4, Peter Bachman 11, Ayse Belger 8, Ricardo E Carrión 5, Franc CL Donkers 8, Erica Duncan 10,12, Jason Johannesen 4, Pablo León-Ortiz 6, Gregory Light 1,2, Alejandra Mondragón 6, Margaret Niznikiewicz 9, Jason Nunag 1, Brian J Roach 7, Rodolfo Solís-Vivanco 6; the North American Prodromal Longitudinal Studies Consortium
PMCID: PMC4135529  NIHMSID: NIHMS610313  PMID: 23799460

Abstract

Prepulse inhibition (PPI) and reactivity of the acoustic startle response are widely used biobehavioral markers in psychopathology research. Previous studies have demonstrated that PPI and startle reactivity exhibit substantial within-site stability; between-site stability, however, has not been established. In two separate consortia investigating biomarkers of early psychosis, traveling subjects studies were performed as part of quality assurance procedures in order to assess the fidelity of data across sites. In the North American Prodromal Longitudinal Studies (NAPLS) Consortium, 8 normal subjects traveled to each of the 8 NAPLS sites and were tested twice at each site on the startle PPI paradigm. In preparation for a binational study, 10 healthy subjects were assessed twice in both San Diego and Mexico City. Intraclass correlations between and within sites were significant for PPI and startle response parameters, confirming the reliability of startle measures across sites in both consortia. There were between site differences in startle magnitude in the NAPLS study that did not appear to be related to methods or equipment. In planning multi-site studies, it is essential to institute quality assurance procedures early and establish between site reliability to assure comparable data across sites.

Keywords: Endophenotype, Reliability, Startle, Prepulse Inhibition

Introduction

Prepulse inhibition (PPI) and reactivity of the acoustic startle response are widely used translational biomarkers in psychopathological research. PPI is an index of sensorimotor gating and is used in animal and human studies to understand brain disorders such as schizophrenia and Tourette's Disorder that are characterized by gating impairments in the neural substrates that underlie sensory information processing 1. In the PPI paradigm, weak lead stimuli inhibit the startle response to intense, abrupt stimuli (acoustic, visual, tactile) 2. PPI is typically reduced in individuals with schizophrenia 3, stable with repeated within site testing 4-10, heritable 11,12, and associated with genes of relevance to psychosis13,14, suggesting its utility as an endophenotype and as a vulnerability marker for psychosis risk 15.

An increasing emphasis in schizophrenia research has been in the area of early detection and intervention. The use of biobehavioral markers such as PPI in the study of the prodromal phase of psychosis provides a means of not only identifying individuals at greatest risk for psychosis but also understanding neurodevelopmental abnormalities early in the course of illness that can contribute to better informed treatment 16. Although it is possible to use empirically derived criteria for a prodromal psychosis syndrome 17 to identify individuals at increased risk of psychotic illness, the 2 year psychotic conversion rate is between 15-35% 18, making it difficult to recruit a sufficient number of subjects at any one site. Therefore, multisite studies are essential to attain sufficient statistical power to investigate the prodromal phase of illness.

For biomarkers such as PPI to be useful in multisite studies that are needed to increase statistical power, facilitate the identification of disease risk, increase the odds of finding uncommon genetic variation or identification of relevant subgroups however, the measures need to be stable with repeated assessment and reliable across sites 19. Because differences in testing conditions and procedures across sites can introduce uncontrolled variance in experimental measures, it is essential to understand potential site differences and control variation across sites as much as possible. Although multisite studies have investigated PPI 19, to our knowledge, there are no published reports of between site reliability of startle measures using normal subjects traveling between sites. This study investigated the within- and between-site reliability of PPI and startle reactivity in two consortia designed to identify vulnerability markers in early psychosis: The NAPLS (North American Prodromal Longitudinal Studies) Consortium and a UCMEXUS (University of California Institute for Mexico and the United States).

Materials and Methods

Participants

Participants included: 1) 8 healthy subjects recruited from each of the 8 NAPLS sites (Emory, Harvard, University of Calgary, UCLA, UCSD, UNC, Yale, Zucker Hillside) (age 19-30, 4 males and 4 females) and 2) 10 healthy subjects (ages 28-38, 4 males and 6 females), recruited from UCSD and the National Institute of Neurology and Neurosurgery (INNN) in Mexico City. All 9 institutions received approval from their individual ethics committees for the study. Subjects provided written informed consent after the procedures were fully explained. Subjects were excluded if they had the following: any concomitant medical or neurological illness, current substance abuse or dependence (excluding nicotine), any Axis I disorders (per Structured Interview for DSM-IV) or positive family history of psychosis.

Acoustic startle paradigm

Equipment and procedures were identical at the 8 NAPLS sites as well as between UCSD and INNN. Manuals with equipment setup, testing procedures and instructions to subjects were developed in English and Spanish (for INNN). A meeting was held in Boston October 2009 to train all NAPLS sites; UCSD staff visited the Mexico site in September 2009 to train INNN using the same procedures 20.

Subjects were screened for hearing impairment (>45 dB 1000 Hz). Smokers were allowed to smoke up to 30 min prior to startle testing to avoid nicotine withdrawal or intoxication. A customized Startle-stimulus generating system (Grace Design Model m902 Amplifier and Neurobehavioral Systems Presentation software) developed by the UCSD site was used for all sites. The sound was calibrated at all sites using a Quest 210 Sound Level Meter and a custom-made PPI calibration session to ensure 70dB for background noise and 115dB for extended length startle bursts at each of the sites. Neurophysiologic recordings at NAPLS sites were performed using identical Biosemi systems and recording software (Biosemi, Amsterdam, Netherlands). For the UCMEXUS study, data were recorded using NeuroScan equipment and software (NuAmps Digital EEG Amplifier, Neuro Scan Labs, Sterling, VA). Electrodes (Ag/AgCl) were placed below and at the outer canthus of the right eye with resistances less than 10 kΩ 20. Startle stimuli were presented binaurally through identical headphones (TDH-39P) at all sites. A 70 dB [A] broadband background noise was used with a pulse (115 dB [A], 40 ms noise burst) presented either alone or following (30, 60 or 120 ms interstimulus interval; ISI) a prepulse (86 dB [A], 20 ms noise burst). The paradigm began with a 5-minute acclimation period, then five pulse alone stimuli followed by 30 trials consisting six trials each of the three prepulse conditions and 12 pulse alone stimuli presented in a fixed, pseudorandom order. The paradigm ended with five more pulse alone stimuli for a total of 40 trials. EMG activity for both consortia was analyzed at UCSD using Brain Vision Analyzer (Brain Vision LLC, Morrisville, NC) and high-pass filtered at 28Hz at 12dB/Oct. Waveforms were smoothed using a 40Hz 24dB/Oct low-pass filter. All trials were manually inspected for artifacts. Startle data were analyzed using wave-form averaging for each of the four different trial types within each block, after applying baseline correction and rectification of the data. The magnitude of the peak startle response (highest point relative to baseline between 30 and 120 ms after onset of startle stimulus) was determined. All subjects demonstrated a robust startle response to the first block of startle stimuli but subjects who demonstrated a relative lack of startle stimulus elicited eye blink to the second block of startle stimuli in any test session were excluded per established methods 20. The following startle measures were examined: 1) reactivity, or the mean magnitude of response to pulse alone stimuli, and 2) prepulse inhibition (PPI), the percentage of change in startle magnitude to prepulse +pulse versus pulse-alone trials ((pulse –prepulse + pulse)/pulse)*100). The stability of the startle measures between and within sites was assessed using intraclass correlations (ICC; Random, Consistency Model) and repeated measures analysis of variance (ANOVA) design. All subjects were tested twice at each site and traveled to the other sites within 3 months for NAPLS and within 1 year (Mean 5.5 months) for UCMEXUS. The order of testing was balanced across sites in both studies with a specified order for NAPLS subjects starting with the home site and within the 9 subjects who were included in the UCMEXUS study (5 at UCSD first, 4 at INNN first).

Results

As shown in Table 1, within site ICCs of startle and PPI variables were significant across all reactivity and PPI conditions in both NAPLS and UCMEXUS (Table 1). Between-site analyses were similarly performed comparing time 1 to time 1 and time 2 to time 2 across sites. All but the 30 ms PPI (p=0.056) condition for UCMEXUS were significant. Finally, within- and between-site ICCs were calculated using both sessions at each of the sites and all were significant. In repeated measures ANOVA of PPI (NAPLS: 8 Sites X 2 sessions X 3 ISIs; UCMEXUS: 2 Sites X 2 sessions X 3 ISIs) there were no statistical site (NAPLS: F[7,108]=0.45, ns; UCMEXUS: F[1,9]=0.51, ns) or session (NAPLS: F[1,108]=0.72, ns; UCMEXUS: F[1,9]=0.83, ns) main or interaction effects supporting the within- and between-site reliability. In contrast, a repeated measures ANOVA of startle reactivity (NAPLS: 8 sites X 2 times X 3 blocks; UCMEXUS: 2 sites X 2 times X 3 blocks) revealed a significant site effect (F[7,28]=2.46, p<0.05) for the NAPLS study due to one site having greater startle amplitude relative to the other sites (see Figure 1), but no session or interaction effects. When NAPLS site 4 was removed from the analysis, the significant site effect was no longer present (F[6,30]=1.97, ns). The site main effect for UCMEXUS (F[1,9]=0.48, ns) was non-significant as were session and interaction effects.

Table 1.

Intraclass Correlations (ICC – Random/Consistency Model) of startle reactivity and Prepulse Inhibition (PPI) within and between sites in the NAPLS and UCMEXUS studies.

NAPLS Within Site ICC NAPLS Between Sites ICC NAPLS Within and Between Site ICC UCMEXUS Within Site ICC UCMEXUS Between Sites ICC UCMEXUS Within and Between Site ICC
Startle Reactivity
Block 1 0.43*** 0.46*** 0.51*** 0.85*** 0.43* 0.60***
Block 2 0.80*** 0.76*** 0.79*** 0.88*** 0.52** 0.65***
%PPI
PPI 30 ms 0.67*** 0.50*** 0.57*** 0.63* 0.31 0.38**
PPI 60 ms 0.60*** 0.57*** 0.48*** 0.67** 0.80*** 0.52***
PPI 120 ms 0.68*** 0.73*** 0.78*** 0.59** 0.80*** 0.69***
Open in a new tab
***

p < 0.001

**

p<0.01

*

p<0.05

Figure 1.

Figure 1

Open in a new tab

Site differences in startle reactivity were evident in the NAPLS consortium. Data represents estimated marginal means that collapsed two test sessions at each site.

Discussion

This is the first report of between-site reliability of PPI and startle reactivity measured with traveling subjects. The present findings replicate previous studies that demonstrate within-site stability of startle measures in normal and schizophrenia spectrum subjects 4-10 and extend these findings to demonstrate measurement comparability across laboratories in two separate multisite studies using 2 different types of equipment for neurophysiologic recording. It is likely that the standardization of equipment, protocols, training, analysis and quality assurance procedures across sites contributed to the observed consistency of startle data.

Although startle reactivity was stable both within- and between-sites, significant site differences were observed in the NAPLS study driven by larger startle amplitude at one of the sites, prompting a review of equipment settings, stimulus calibration, ambient acoustic noise, electrical noise, placement of electrodes, subject instructions, and testing environment across sites. A decibel meter from UCSD was mailed to the site in question (site 4) to assure the loudness of the startle stimuli was accurate and consistent across sites. No methodological or equipment differences were identified. Individual subject data revealed that three subjects had larger startle responses at site 4 (Figure 2), accounting for the site differences. One subject with a large startle response was first exposed to the startle stimuli at site 4, perhaps accounting for the larger response. Since each subject began their travels at their home site, it is unlikely that order effects account for the observed differences. Thus, despite institution of careful quality assurance procedures, identical subjects, methodology and equipment, site differences still occur and need to be examined and controlled for in biomarker studies. Future analyses of NAPLS consortium data will continue to examine site differences in reactivity and site will be used as a between subjects factor.

Figure 2.

Figure 2

Open in a new tab

Individual traveling subjects data from the NAPLS consortium demonstrates that 3 subjects accounted for the observed site differences. Data represents estimated marginal means that collapse both test sessions and 3 blocks of startle magnitude in response to pulse alone stimuli.

A limitation of the 2 studies is the relatively small sample size in each (NAPLS included 8 subjects tested at 8 sites and UCMEXUS included 10 subjects tested at 2 sites). The sample sizes, however, are consistent with the few traveling subjects studies performed to establish reliability of neuroimaging measures across sites 22,23. Although future between-site biomarker reliability studies should ideally use more subjects, sending multiple subjects to different cities and countries for multiple testing sessions obviously presents financial and logistical challenges.

Conclusion

In planning multi-site biomarker studies, it is essential to institute standardized quality assurance procedures prior to data collection of targeted research samples. The use of identical equipment, training and similar testing environments appears to be useful to minimize sources of cross-site variance in electrophysiological studies. The observed reliability of startle measures across laboratories provides support for the utility of these measures as biomarkers and endophenotypes in large multisite studies. Investigation and statistical control of potential site differences is essential in any multisite biomarker study.

Acknowledgements

The authors would like to thank Kathleen Shafer, Daniel Roman, Clara Robles, Sylvana Stefano, Rafael Favila, Aaron Peterson, Sona Sandhu, Anna Evans, Kathleen Monforton, Brenton Roman, Joshua Kenney for their technical assistance with the project.

Source of Funding and Conflicts of Interest: This research was supported by a Collaborative Grant from UC MEXUS –CONACYT (Cadenhead and de la Fuente-Sandoval, Co-PIs), National Institutes of Health (NIH)-supported grants U01 MH081944 to KSC; U01 MH082022 to SWW, U01MH081984 to JA, U01 MH081902 to TDC, UO1 MH081857 to BAC, U01MH082004 to DOP, U01 MH081928 to LJS, and U01MH081988 to EFW. Conflicts of Interest: Kristin Cadenhead, Elaine Walker, Larry Seidman, Tyrone Cannon, Ming Tsuang, Alejandra Mondragón, Rodolfo Solis-Vivanco, Pablo León-Ortiz and Ricardo Carrión - none declared. Barbara Cornblatt has been a consultant for Hoffman La Roche and received royalties for the CPT-IP. Jean Addington has been a consultant for Hoffman La Roche. Diana Perkins is on the Advisory Board for Sunovion DSMB, Genentech CNS, Genentech Mosaic Registry and a Consultant for Telesage. Scott Woods has been a Consultant for Merck. Camilo de la Fuente-Sandoval has served as consultant and/or speaker for IMS Health, Carnot Laboratories, Eli Lilly and Janssen.

References

  • 1.Braff DL. Prepulse inhibition of the startle reflex: a window on the brain in schizophrenia. Curr Top Behav Neurosci. 4:349–71. doi: 10.1007/7854_2010_61. [DOI] [PubMed] [Google Scholar]
  • 2.Graham FK. Presidential Address, 1974. The more or less startling effects of weak prestimulation. Psychophysiology. 1975;12(3):238–248. doi: 10.1111/j.1469-8986.1975.tb01284.x. [DOI] [PubMed] [Google Scholar]
  • 3.Braff DL, Grillon C, Geyer MA. Gating and habituation of the startle reflex in schizophrenic patients. Archives of General Psychiatry. 1992;49(3):206–215. doi: 10.1001/archpsyc.1992.01820030038005. [DOI] [PubMed] [Google Scholar]
  • 4.Cadenhead KS, Carasso B, Swerdlow NR, Geyer MA, Braff DL. Prepulse inhibition and habituation of the startle response are stable neurobiological measures in a normal male population. Biological Psychiatry. 1999;45:360–364. doi: 10.1016/s0006-3223(98)00294-7. [DOI] [PubMed] [Google Scholar]
  • 5.Abel K, Waikar M, Pedro B, Hemsley D, Geyer M. Repeated testing of prepulse inhibition and habituation of the startle reflex: a study in healthy human controls. J Psychopharmacol. 1998;12(4):330–7. doi: 10.1177/026988119801200402. [DOI] [PubMed] [Google Scholar]
  • 6.Schwarzkopf SB, McCoy L, Smith DA, Boutros NN. Test-retest reliability of prepulse inhibition of the acoustic startle response. Biol Psychiatry. 1993;34(12):896–900. doi: 10.1016/0006-3223(93)90059-m. [DOI] [PubMed] [Google Scholar]
  • 7.Ludewig K, Geyer MA, Etzensberger M, Vollenweider FX. Stability of the acoustic startle reflex, prepulse inhibition, and habituation in schizophrenia. Schizophr Res. 2002;55(1-2):129–37. doi: 10.1016/s0920-9964(01)00198-0. [DOI] [PubMed] [Google Scholar]
  • 8.Quednow BB, Kuhn KU, Beckmann K, Westheide J, Maier W, Wagner M. Attenuation of the prepulse inhibition of the acoustic startle response within and between sessions. Biol Psychol. 2006;71(3):256–63. doi: 10.1016/j.biopsycho.2005.05.001. [DOI] [PubMed] [Google Scholar]
  • 9.Flaten MA. Test-retest reliability of the somatosensory blink reflex and its inhibition. Int J Psychophysiol. 2002;45(3):261–5. doi: 10.1016/s0167-8760(02)00034-x. [DOI] [PubMed] [Google Scholar]
  • 10.Light GA, Swerdlow NR, Rissling AJ, Radant A, Sugar CA, Sprock J, Pela M, Geyer MA, Braff DL. Characterization of neurophysiologic and neurocognitive biomarkers for use in genomic and clinical outcome studies of schizophrenia. PLoS One. 2012;7(7):e39434. doi: 10.1371/journal.pone.0039434. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 11.Greenwood TA, Braff DL, Light GA, Cadenhead KS, Calkins ME, Dobie DJ, Freedman R, Green MF, Gur RE, Gur RC, Mintz J, Nuechterlein KH, Olincy A, Radant AD, Seidman LJ, Siever LJ, Silverman JM, Stone WS, Swerdlow NR, Tsuang DW, Tsuang MT, Turetsky BI, Schork NJ. Initial heritability analyses of endophenotypic measures for schizophrenia: the consortium on the genetics of schizophrenia. Arch Gen Psychiatry. 2007;64(11):1242–50. doi: 10.1001/archpsyc.64.11.1242. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 12.Hasenkamp W, Epstein MP, Green A, Wilcox L, Boshoven W, Lewison B, Duncan E. Heritability of acoustic startle magnitude, prepulse inhibition, and startle latency in schizophrenia and control families. Psychiatry Res. 2010;178(2):236–43. doi: 10.1016/j.psychres.2009.11.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Greenwood TA, Lazzeroni LC, Murray SS, Cadenhead KS, Calkins ME, Dobie DJ, Green MF, Gur RE, Gur RC, Hardiman G, Kelsoe JR, Leonard S, Light GA, Nuechterlein KH, Olincy A, Radant AD, Schork NJ, Seidman LJ, Siever LJ, Silverman JM, Stone WS, Swerdlow NR, Tsuang DW, Tsuang MT, Turetsky BI, Freedman R, Braff DL. Analysis of 94 Candidate Genes and 12 Endophenotypes for Schizophrenia From the Consortium on the Genetics of Schizophrenia. Am J Psychiatry. 2011 doi: 10.1176/appi.ajp.2011.10050723. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Greenwood TA, Light GA, Swerdlow NR, Radant AD, Braff DL. Association analysis of 94 candidate genes and schizophrenia-related endophenotypes. PLoS One. 2012;7(1):e29630. doi: 10.1371/journal.pone.0029630. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Gottesman, Gould TD. The endophenotype concept in psychiatry: etymology and strategic intentions. Am J Psychiatry. 2003;160(4):636–45. doi: 10.1176/appi.ajp.160.4.636. [DOI] [PubMed] [Google Scholar]
  • 16.Cadenhead KS. Vulnerability markers in the schizophrenia spectrum: implications for phenomenology, genetics, and the identification of the schizophrenia prodrome. Psychiatr Clin North Am. 2002;25(4):837–53. doi: 10.1016/s0193-953x(02)00021-7. [DOI] [PubMed] [Google Scholar]
  • 17.Miller TJ, McGlashan TH, Rosen JL, Cadenhead K, Cannon T, Ventura J, McFarlane W, Perkins DO, Pearlson GD, Woods SW. Prodromal assessment with the structured interview for prodromal syndromes and the scale of prodromal symptoms: predictive validity, interrater reliability, and training to reliability. Schizophr Bull. 2003;29(4):703–15. doi: 10.1093/oxfordjournals.schbul.a007040. [DOI] [PubMed] [Google Scholar]
  • 18.Cannon TD, Cadenhead K, Cornblatt B, Woods SW, Addington J, Walker E, Seidman LJ, Perkins D, Tsuang M, McGlashan T, Heinssen R. Prediction of psychosis in youth at high clinical risk: a multisite longitudinal study in North America. Arch Gen Psychiatry. 2008;65(1):28–37. doi: 10.1001/archgenpsychiatry.2007.3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Swerdlow NR, Sprock J, Light GA, Cadenhead K, Calkins ME, Dobie DJ, Freedman R, Green MF, Greenwood TA, Gur RE, Mintz J, Olincy A, Nuechterlein KH, Radant AD, Schork NJ, Seidman LJ, Siever LJ, Silverman JM, Stone WS, Tsuang DW, Tsuang MT, Turetsky BI, Braff DL. Multi-site studies of acoustic startle and prepulse inhibition in humans: Initial experience and methodological considerations based on studies by the Consortium on the Genetics of Schizophrenia. Schizophr Res. 2007 doi: 10.1016/j.schres.2007.01.012. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Cadenhead KS. Startle reactivity and prepulse inhibition in prodromal and early psychosis: effects of age, antipsychotics, tobacco and cannabis in a vulnerable population. Psychiatry Res. 2011;188(2):208–16. doi: 10.1016/j.psychres.2011.04.011. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Cadenhead KS, Carasso BS, Swerdlow NR, Geyer MA, Braff DL. Prepulse inhibition and habituation of the startle response are stable neurobiological measures in a normal male population. Biol Psychiatry. 1999;45(3):360–4. doi: 10.1016/s0006-3223(98)00294-7. [DOI] [PubMed] [Google Scholar]
  • 22.Friedman L, Stern H, Brown GG, Mathalon DH, Turner J, Glover GH, Gollub RL, Lauriello J, Lim KO, Cannon T, Greve DN, Bockholt HJ, Belger A, Mueller B, Doty MJ, He J, Wells W, Smyth P, Pieper S, Kim S, Kubicki M, Vangel M, Potkin SG. Test-retest and between-site reliability in a multicenter fMRI study. Hum Brain Mapp. 2008;29(8):958–72. doi: 10.1002/hbm.20440. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Brown GG, Mathalon DH, Stern H, Ford J, Mueller B, Greve DN, McCarthy G, Voyvodic J, Glover G, Diaz M, Yetter E, Ozyurt IB, Jorgensen KW, Wible CG, Turner JA, Thompson WK, Potkin SG. Multisite reliability of cognitive BOLD data. Neuroimage. 2011;54(3):2163–75. doi: 10.1016/j.neuroimage.2010.09.076. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES