Timbre-independent extraction of pitch in newborn infants

Abstract

The ability to separate pitch from other spectral sound features, such as timbre, is an important prerequisite of veridical auditory perception underlying speech acquisition and music cognition. The current study investigated whether or not newborn infants generalize pitch across different timbres. Perceived resonator size is an aspect of timbre that informs the listener about the size of the sound source, a cue that may be important already at birth. Therefore, detection of infrequent pitch changes was tested by recording event-related brain potentials in healthy newborn infants to frequent standard and infrequent pitch-deviant sounds while the perceived resonator size of all sounds was randomly varied. The elicitation ofanearly negative andalater positive discriminative responsebydeviant sounds demonstrated that the neonate auditory system represents pitch separately from timbre, thus showing advanced pitch processing capabilities.

Pitch is a perceived auditory feature ordering sounds on a scale from low to high. Pure tones resonate at a single frequency, and the classical findings of psychophysics showed that for such tones, perceived pitch is determined by the tone frequency (see, e.g., Zwicker & Fastl, 1990). However, adults perceive pitch even when hearing nonharmonic sounds, and they can compare pitch across spectrally and temporally very different sounds (e.g., telling whether a sound produced by a flute was higher orlower than a sound produced by a violin). This suggests that the auditory system analyzes sound structure in a complex way to arrive at perceiving and representing pitch. Some theorists suggest that learning plays an important role in pitch perception (Terhardt, 1974). Therefore, it is important to test whether or not nontrivial forms of pitch analysis are functional in newborn infants, especially because pitch processing is an important prerequisite of music and speech perception and is also crucial for representing auditory objects (Kubovy & Van Valkenburg, 2001). The current study was designed to test whether neonates can equate pitch across sounds with different timbres.

Sensitivity to spectral sound features appears very early during ontogenesis. Behavioral studies have shown that the fetal auditory system is functional to a degree already at ~20 weeks of gestational age (Birnholz & Benacerraf, 1983; Hepper & Shahidullah, 1994). Reliable event-related magnetic responses were obtained to pure tones from fetuses from the 27th week of gestational age onward with the latency of the evoked response decreasing with age (Holst et al., 2005; Draganova et al., 2005). Fetal responses related to spectrum-based sound discrimination have been obtained for pitch-deviant complex tones from ~30 weeks of gestational age (e.g., Huotilainen et al., 2005).

In the current study, pitch processing in newborn infants was tested by recording electric brain responses to sounds (auditory event-related brain potentials [ERPs]). In adults, a component of the human auditory ERP has been shown to be sensitive to perceived pitch, as opposed to the raw acoustic parameters underlying pitch perception, such as the frequencies present in a sound. Using the missing fundamental phenomenon (when removing the fundamental frequency from a complex tone does not change its perceived pitch; see de Boer, 1976), Winkler and colleagues (1995) presented healthy young adults with infrequent complex tones of 600-Hz fundamental frequency (pitch deviants) among nine different complex tones having 300 Hz as the missing fundamental (standards). Thus neither the standard tones nor the deviants contained harmonics below 600 Hz. Pitch-deviant tones elicited an MMNm response, the magnetic counterpart of the mismatch negativity (MMN) ERP component, which is an electrophysiological signal reflecting that the violation of an acoustic regularity was detected in the brain (for a recent review, see Näätänen, Paavilainen, Rinne, & Alho, 2007). Because each frequency present in the deviant was also present in five of the standard tones, these results indicated that the MMN response is based on perceived pitch rather than the spectral composition of the complex tones (Winkler et al, 1995; for a review of other compatible results, see Näätänen & Winkler, 1999). Because the sequence of standard (300-Hz missing fundamental) tones was composed of nine complex tones differing in their harmonic structure, MMN elicited by pitch change also showed that pitch repetition was detected even when other spectral parameters varied in the sound sequence (Näätänen & Winkler, 1999; Winkler et al., 1995). Thus it is feasible to use MMN to test whether pitch-repetition regularity is extracted from a sequence composed of spectrally different sounds.

Similarly to adults, occasional changes in the frequency of pure and complex tones have been found to elicit an ERP component in full- and pre-term newborn infants, and these neonatal electromagnetic responses have been regarded as analogues of the adult MMN and MMNm (Alho, Saino, Sajaniemi, Reinikainen, & Näätänen, 1990; Čeponienė et al., 2002; Draganova et al., 2005). However, the sensitivity of detecting frequency changes in pure sinusoidal tones was found to be substantially lower in neonates than in adults (Novitski, Huotilainen, Tervaniemi, Näätänen, & Fellman, 2007), a result in line with behavioral evidence obtained for 1-month old infants (Wormith, Pankhurst, & Moffit, 1975). More reliable discrimination of spectral changes was shown with complex harmonic tones (Čeponienë et al., 2002), environmental sounds (Sambeth, Huotilainen, Kushnerenko, Fellman, & Pihko, 2006), noise (Kushnerenko et al., 2007), and speech sounds (Kujala et al., 2004), suggesting that changes in sounds activating wider regions of the auditory cortex are more likely to be detected. Using the MMN method, the operation of higher-level auditory abilities was also demonstrated in newborn infants, such as the segregation of concurrent streams of sound (Winkler et al., 2003) and auditory temporal grouping (Stefanics et al., 2007). Very importantly for the current study, Carral and colleagues (2005) demonstrated that the neonate auditory system extracts invariant features from variable sound sequences, providing a basis for testing the processing of higher-order auditory cues.

Timbre is a complex sound property partly based on spectral as well as dynamic cues and having several perceptually distinct components (e.g., McAdams, Winsberg, Donnadieu, De Soete, & Krimphoff, 1995; Saldanha & Corso, 1964). In adults, it has been found that occasional timbre-deviant sounds elicit the MMN response (Goydke, Altenmüller, Möller, & Münte, 2004; Tervaniemi, Winkler, & Näätänen, 1997). Perceived resonator size is an aspect of timbre, which allows adults to estimate the size of the sound source with reasonable accuracy irrespective of the pitch of the emitted sound (Smith, Patterson, Turner, Kawahara, & Irino, 2005, van Dinther & Patterson 2006). Vestergaard, Shtyrov, Pulvermüller, and Patterson (2007) and Vestergaard and colleagues (2008) showed that infrequent perceived resonator size changes elicit MMN both in adults, who did not attend the sounds, and in sleeping newborn infants, suggesting the existence of a neural subsystem for automatic speaker-size processing in the human auditory system.

In the current experiment, timbre variation was implemented by varying the resonator size of an instrument (French horn) sound, using such levels of the resonator-size feature, which are discriminated by the neonate auditory system (Vestergaard et al., 2008). Sounds with two different pitches (distinguishable by neonates; see Novitski et al., 2007) were presented to sleeping newborn infants in an oddball design with resonator size varying independently of pitch. If the neonate auditory system extracts pitch information independently of the resonator size, then occasional pitch deviants will be detected, resulting in the elicitation of the corresponding electrophysiological signals (i.e., pitch invari-ance will be detected despite variance in timbre; cf. Carral et al., 2005). If, however, pitch and timbre (resonator size) are not processed separately by newborn infants, then pitch deviants will not be detected and no electrophysiological sign of the differential processing of the “standard” and “deviant” sounds will be observed. That is, if pitch repetition is not detected and, because no individual sound appears sufficiently often in the sequence to allow the formation of holistic sound-repetition regularity, then the pitch-deviant sounds do not violate any detected regularity in the sequence and, therefore, will not trigger differential processing in the neonate auditory system.

Methods

Participants

ERP responses were recorded from 12 (5 female) healthy full-term newborn infants on day 2 or 3 postpartum. Their gesta-tional age was 37–40 weeks, birth weight 2730–4330 g, and Apgar score 9–10. Data from 3 of the 12 subjects (2 boys) were discarded because of excessive artifacts. The recordings were carried out in the hospital ward of the First Department of Obstetrics and Gynecology, Semmelweis University, Budapest, Hungary, after informed consent was obtained from one or both parents. The mother of the infant was present during the recording. The study was approved by the Ethics Committee of the Semmelweis University as well as by the Institutional Review Board of the Institute for Psychology, Hungarian Academy of Sciences.

Stimuli and Procedure

Sequences were composed of instrumental sounds with two different pitches, 139 Hz (C#3) and 175 Hz (F3). The sounds, played on a French horn, were taken from the database by Goto, Hashiguchi, Nishimura, and Oka (2003). For both pitches, seven resonator-size variants were created. Different resonator sizes were simulated by setting the resonance scale of the sounds with a vocoder (Kawahara & Irino, 2004). The just noticeable difference (JND) for simulated resonator size for French horn is approximately 6%–9% for adult listeners (van Dinther & Patterson, 2006). For the current study, resonator size was varied over 22% (ca. 6 JNDs), corresponding to the scale factors between 1/1.22 and 1.22. The seven resonator sizes were equally spaced on the logarithmic scale between −22% and +22%, producing a step size of approximately 1 JND for the horn resonance scale (values: 0.82, 0.88, 0.94, 1.00, 1.07, 1.14, 1.22, where 1.00 denotes the base sound). To improve the signal-to-noise ratio of the early ERP components, sound onsets were made more abrupt by removing the 60-ms-long initial period from the original sound and imposing a 10-ms-long raised-cosine ramp on the new sound onset. The total sound duration was 545 ms for both notes. Sounds were presented binaurally using the E-Prime stimulus presentation software (Psychology Software Tools, Inc., Pittsburgh, PA) via ER-3A loudspeakers (EtymStic Research, Inc., Elk Grove Village, IL) connected via sound tubes to self-adhesive ear-couplers (Natus Medical, Inc., San Carlos, CA) placed over the babies' ears. We have previously shown that occasionally changing the resonator size of the above-described sounds by 22% elicits ERP correlates of deviance detection in sleeping neonates (Vestergaard et al., 2008).

Sounds were presented in an oddball sequence (750 ms stimulus onset asynchrony [SOA]) with 87.5% of them (standards) having C#3 and 12.5% (deviants) F3 for pitch. The seven resonator-size variants were equiprobably distributed, separately within the standard and the deviant stimuli (14.3% each). Two stimulus blocks of 600 sounds were delivered (a total of 1,200 sounds). In control stimulus blocks delivered after the main stimulus sequences, the base frequencies for standard and deviant tones were exchanged, allowing comparison between responses elicited by identical (F3) sounds when they served as deviants in the main experimental stimulus blocks and when they were delivered frequently within the control block.

EEG Recording

EEG was recorded from the F3, F4, C3, Cz, and C4 scalp electrodes (according to the International 10–20 System) and from electrodes placed over the left and right mastoids, with the common reference attached to the tip of the nose. The ground electrode was placed on the forehead. Eye movements were monitored by recording the electrooculogram (EOG) between two electrodes, one placed below the left eye and another above the right eye. EEG was recorded at 24-bit resolution and a sampling rate of 250 Hz by a direct-coupled amplifier (V-Amp, Brain Products, Munich, Germany). The signals were on-line low-pass filtered at 110 Hz.

Data Analysis

EEG was filtered off-line between 1.5 and 16 Hz. For each stimulus, an epoch of 600 ms duration including a 100-ms pre-stimulus period was extracted from the continuous EEG record. Epochs with a voltage change below 0.1 μV or above 70 μV on any EEG or EOG channel were rejected from further analysis. Responses were analyzed for the central line of electrodes showing the best signal-to-noise ratio. Epochs were baseline corrected for the 100-ms prestimulus period and averaged separately for standards and deviants and for the experimental and the control condition. The mean number of artifact-free deviant trials per infant was 136 with over 600 identical-stimulus standard responses collected in the control condition.

For amplitude measurements, two 20-ms-long time windows were selected from the grand-averaged deviant-minus-standard difference waveform (Figure 1) at the Cz electrode. One window was centered on the negative peak in the 78–98-ms latency range and the other on the positive peak in the 198–218-ms latency range. The effects of stimulus type were analyzed with two-way repeated-measures analyses of variance (ANOVA of Stimulus type [deviant vs. control] × Electrode [C3 vs. Cz vs. C4]), separately for the two latency ranges. Greenhouse–Geisser correction of the degrees of freedom was applied where appropriate and ε values as well as η² effect sizes are given in Results. ANOVA results were further specified by Fisher's LSD post hoc tests.

Figure 1.

Group-averaged ERPs elicited by deviant (dotted line) and identical control (dashed line) sounds of 175-Hz base frequency (musical note F3) at three central electrode locations (C3, Cz, and C4). Deviant-minus-control difference waveforms are plotted in black (solid line). Stimulus onset is at the crossing of the axes. Time windows for amplitude measurements (78–98 and 198–218 ms) are shaded gray.

Although a previous study found no significant effect of sleep stages on the MMN parameters measured in neonates (Martynova, Kirjavainen, & Cheour, 2003), other studies suggested the possibility of a sleep-stage effect (Friederici, Friedrich, & Weber, 2002). Therefore, the ratio of quiet and active sleep was assessed separately for each stimulus block, following the criteria suggested by Anders, Emde, and Parmelee (1971), using EEG and EOG measures and observations of the infant's behavior. Periods of wakefulness and indeterminate stages were excluded from this analysis. Overall, the infants were more often in active (58% of the time) than in quiet sleep (22% of the time) during the recordings. This result is in line with the similar observations of Novitski and colleagues (2007). The ratio of quiet and active sleep did not significantly differ between the test and the control stimulus blocks. Therefore, comparing responses between the test and the control stimulus blocks does not significantly affect the results.

Results

Figure 1 shows grand-averaged ERPs elicited by deviant and identical control sounds at the three central electrode locations (C3, Cz, C4) together with the corresponding deviant-minus-control difference waveforms. The time windows for the amplitude measurements are highlighted. The ANOVA for the negative peak in the 78–98-ms latency range yielded a significant main effect of Stimulus type, F(1,8) = 6.23, p<.05, η² = .44, and Electrode, F(2,16) = 6.26, p<.02, η² = .44 ε = .89, and a significant Stimulus type × Electrode interaction, F(2,16) = 7.54, p<.02, η² = .49 ε = .76. The interaction was explained by significant differences found in the post hoc test between deviant-and control-stimulus responses at Cz (p<.01) and a tendency at C4 (p = .071) but not at C3 and that the deviant response at Cz was significantly more negative (at least p<.05) than either deviant or standard responses measured at the two lateral electrodes.

The ANOVA for the positive peak in the 198–218-ms latency range yielded only a significant Stimulus-type × Electrode interaction, F(2,16) = 3.67, p<.05, η² .31, ε = .86. Post hoc tests showed a significant difference between the deviant- and control-sound response on Cz(p<.02), but not on either of the lateral leads and the deviant response at Cz was significantly more positive (at least p<.05) than the standard responses measured at the two lateral electrodes as well as the deviant response measured at C4.

Discussion

Significant differences were found between the ERP responses elicited by deviant and control sounds of identical pitch while resonator size was varied. Our results showed that, despite variation in timbre (resonator size), the neonate auditory system extracted pitch invariance from the sound sequences and detected sounds that deviated from the common pitch.

In all but one of the previous studies of auditory change detection in newborn infants, the standard stimulus was constant throughout the stimulus blocks. Therefore, these studies did not shed light on whether newborn infants process the various spectrum-related perceived auditory features separately or in a global, undistinguished manner. Carral et al.'s (2005) results gave the first indication that some of the spectral features are separately processed. These authors found that variance in one spectral feature (absolute tone frequency) did not interfere with the detection of another spectral feature (pitch-change direction within a tone pair). Their results can be interpreted as showing that absolute frequency and the direction of frequency change within a perceptual unit are extracted and represented separately within the neonate auditory system. In the current experiment, extraction of pitch invariance was tested in the face of variation in a timbral feature (resonator size). The distribution of pitch and timbre levels was distinctly different: One (pitch) allowed detection of a repetition regularity whereas the other (timbre) showed a homogeneous distribution across the feature levels appearing within the sequence. Elicitation of a discriminative response by pitch deviants suggests that pitch repetition was detected under these circumstances. Thus the current results suggest that in newborn infants, a pitchlike auditory feature was analyzed and represented separately from the cues perceived as resonator size by adults. This should not be taken to mean that, in the absence of attention, the neonate auditory system does not conjoin auditory features. Rare combinations of sound features, which separately appear frequently within the stimulus sequence, have been shown to elicit discriminative ERP responses in newborn infants (Ruusuvirta, Huotilainen, Fellman, & Näätänen, 2003, 2004). Although the current results definitely do not rule out the possibility that some processes of pitch processing are learned, they suggest that representation for at least a proto-pitch feature already exists at birth. It can be argued that the fetus is already exposed to sounds within the uterus and thus even the current finding may reflect learned abilities. However, although exposure to sounds within the uterus may be necessary for normal development of the human auditory system, this cannot be regarded as the kind of learning hypothesized by theories of pitch perception (Terhardt, 1974).

The negative response foundinthe current experiment peaked rather early (at 88 ms in the group average). This is probably due to the use of complex, natural stimulus material, which provides multiple congruent cues, thus allowing early determination of deviance. Similar early discriminative negative responses have been previously shown in neonates (Kushnerenko et al., 2007; peak latency: 75 ms) and preschool-age children (Räikkönen, Birkás, Horváth, Gervai, & Winkler, 2003; peak latency: 66 ms) to rare environmental sounds and in adults (Jacobsen, Schröger, & Alter, 2004; peak latency: ca. 105 ms) to occasional deviant vowels. Previous studies showed that the early negative difference in neonates is mainly elicited by large spectral changes (Kushnerenko, Čeponienë, Balan, Fellman, & Näätänen, 2002; Kushnerenko et al., 2007; Morr, Shafer, Kreuzer, & Kurtzberg, 2002). In adults, similar changes usually elicit high-amplitude N1 responses (e.g., Alho et al., 1998; Escera, Alho, Winkler, & Näätänen, 1998). However, the N1 response appears only later in development (Pang & Taylor, 2000; Ponton, Eggermont, Kwong, & Don, 2000; Sharma, Kraus, N., McGee, & Nicol, 1997). Furthermore, unlike the adult N1, the early neonatal negativity is elicited reliably only by deviant stimuli. This suggests that the early neonatal negativity may be more similar to the adult MMN response. However, in contrast to the adult MMN, the neonatal ERP response to acoustic deviation often appears as a positive difference in the 100–300-ms poststimulus interval (e.g., Winkler et al., 2003). This positive response, which in some studies (such as the current one) follows the early negative ERP difference (whereas in others appears alone), was shown to be sensitive to acoustic energy (Kushnerenko et al., 2007). This feature of the neonate positive difference response contrasts those of the somewhat similar P3a response in adults (Friedman, Cycowicz, & Gaeta, 2001; Polich, 2007). Furthermore, the neonatal positive difference can be elicited even when no P3a response would be expected in adults (e.g., in response to rare feature conjunctions; see Ruusuvirta et al., 2003). This feature of the neonatal positive discriminative response is similar to the adult MMN response. Thus, although the morphology of the current as well as some previously observed neonatal ERP responses to infrequent deviant sounds may resemble the sequence of N1/MMN and P3a in adults (e.g., Kushnerenko et al., 2007), the underlying processes are probably different. Both the early negativity and the somewhat later positivity in neonates show some of the features of the MMN response in adults, but they are far more sensitiveto primary surface stimulus features (such as spectral width and acoustic energy) than the adult MMN. In this, they resemble more the adult N1. It is possible that during maturation of the human auditory system, responses to energy change and to deviation from detected regular features of the preceding sound sequence become better separated, forming the basis of passive attention (James, 1890). This development may be reflected by the specialization of ERP responses, the emergence of the well-known N1, MMN, P3a sequence of ERP components, which characterizes the adult response to unexpected salient changes in the acoustic input.

In conclusion, the current results demonstrated the existence of advanced pitch-related processing in newborn infants, and they suggest that separate representation of a pitchlike feature exists at birth. Pitch processing is a necessary prerequisite of normal speech development, including the perception of prosody and emotional contents, speaker identification, and music perception. Therefore, future screening methods for various auditory-related deficits, such as, for example, dyslexia might be based on the testing of pitch discrimination at a very early age (Fellman & Huotilainen, 2006).