Computerized home video detection for motherese may help to study impaired interaction between infants who become autistic and their parents

Abstract

Autism is a well‐defined clinical syndrome after the second year of life, but information on autism in the first two years of life is still lacking. The study of home videos has described children with autism during the first year of life as not displaying the rigid pattern typical of later symptoms. Therefore, developmental/environmental factors are claimed in addition to genetic/biological ones to explain the onset of autism during maturation. Here we describe (1) a developmental hypothesis focusing on the possible implication of motherese impoverishment during the course of parent–infant interactions as a possible co‐factor; (2) the methodological approach we used to develop a computerized algorithm to detect motherese in home videos; (3) the best configuration performance of the detector in extracting motherese from home video sequences (accuracy = 82% on speaker‐independent versus 87.5% on speaker‐dependent) that we should use to test this hypothesis. Copyright © 2011 John Wiley & Sons, Ltd.

Introduction

Autism is a severe psychiatric syndrome characterized by the presence of abnormalities in reciprocal social interactions, abnormal patterns of communication, and a restricted, stereotyped, repetitive repertoire of behaviours, interests, and activities. Although it is a well‐defined clinical syndrome after the second, and especially after the third, year of life, information on autism in the first two years of life is still lacking (Short and Schopler, 1988; Stone et al., 1994; Sullivan et al., 1990). Home movies (that is, films recorded by parents during the first years of life, before diagnosis) and direct observation of infants at high risk due to having an autistic sibling are the two most important sources of information for overcoming this problem. They have both described children with autism disorder (AD) during the first year of life (and also in the first part of the second year) as not displaying the rigid patterns described in older children. In particular, the children can gaze at people, turn toward voices and express interests in communication as typically developing infants do. It is of seminal importance to have more insight into these social competencies and in which situations they preferentially emerge in infants who are developing autism. In this paper, we focus on a verbal stimulus called motherese (a special kind of speech that is directed towards infants) that has been shown to be of crucial importance in language acquisition (Kuhl, 2004), and question whether its course during parent/child interaction might be affected by the autistic child.

First clinical manifestations of autism

Long before a diagnosis is established, parents are often aware of the differences exhibited by their autistic children (Werner et al., 2000). Parents' descriptions include extremes of temperament and behaviour (ranging from marked irritability to alarming passivity), poor eye contact, and lack of responsiveness to parents' voices or attempts to play and interact (e.g., Dahlgren and Gillberg, 1989; De Giacomo and Fombonne, 1998; Ohta et al., 1987).

Data from prospective and longitudinal studies focusing on high‐risk infants (siblings of AD children) have also yielded important insights into what constitutes the initial behavioural signs of autism. Studies have reported various behavioural markers that, in part, predict a subsequent diagnosis of autism in a sibling sample. These markers include atypicalities in eye contact, visual tracking, social smiling, and reactivity (Zwaigenbaum et al., 2005), lack of orienting to the child's own name (Nadig et al., 2007; Zwaigenbaum et al., 2005), prolonged latency to disengage visual attention (Swettenham et al., 1998; Bryson et al., 2007; Zwaigenbaum et al., 2005), deficits in joint attention skills (Mundy et al., 1990; Baron‐Cohen et al., 1996; Sullivan et al., 2007), lack of pointing to objects (Baron‐Cohen et al., 1996), lack of imitation (Baron‐Cohen et al., 1996; Zwaigenbaum et al., 2005), delayed expressive and receptive language (Sullivan et al., 2007; Landa and Garrett‐Mayer, 2006), and a lack of interest or pleasure in self‐initiated contact with others (Bryson et al., 2007; Baron‐Cohen et al., 1996; Zwaigenbaum et al., 2005).

Several studies focusing on early home videos have also revealed atypical developmental tendencies in infants later diagnosed with autism spectrum disorders (ASDs). In the first year, autistic infants tend to lack social interest, to interact poorly (Maestro et al., 2002; Zakian et al., 2000; Adrien et al., 1993), to look less at others (Clifford and Dissanayake, 2007; Osterling and Dawson, 1994; Osterling et al., 2002; Maestro et al., 2002, 2005b, 2006) and to show less communicative skills (e.g., vocalizations to people). Compared with delayed infants, the characteristics of a reduced response to the infant's own name, a reduced quality of affect and a tendency to look less at others appear more specific to autistic children (Clifford and Dissanayake, 2007; Baranek, 1999; Osterling et al., 2002). In the second year, pre‐autistic signs (lack of social behaviour and joint attention) become more obvious and signs particular to autism more numerous: lower eye contact quality, gaze aversion, lower social peer interest, less positive affect, less nestling, less protodeclarative showing, less anticipatory postures and less conventional games (Clifford and Dissanayake, 2007; Clifford et al., 2007); eye‐contact quality and affect quality have been found to be the best predictors of a later diagnosis of autism (Clifford et al., 2007; Maestro et al., 2001). Whether these early signs impact the interactive process between an infant and his/her parents and, as a consequence, whether they influence the development of the infant himself, remain two challenging issues.

Early language acquisition and social interactions

To explore these points, we will now briefly review what we know about language learning, social interactions, and ways to integrate the disadvantages of autistic children in both domains within the course of development. Studies of language acquisition are too numerous for an extensive review; however, we wish to highlight some specific points regarding the developmental course of language acquisition in children developed by Kuhl (2007, 2004, 2003, 2000) and Goldstein et al., (2003). (1) From birth, infants possess the ability to discriminate all languages at the phonetic level and to recognize their prosodic features. (2) They develop a strategy of learning based on input language signs and characteristics and explore language statistical properties, leading to “so‐called” probabilistic learning (Milgram and Atlan, 1983). (3) The language experience will commit the perceptual system at the neural level, increasing native‐language speech perception and decreasing foreign‐language speech perception with a “magnet effect” (Kuhl, 2000). However, simple exposure does not explain language learning: in both speech production and speech perception, the presence of a human being interacting with a child has a strong influence on learning (Goldstein et al., 2003).

Regardless of the influence of the neurobiological and genetic factors on autism (Cohen et al., 2005), one should keep in mind that infants' survival and development depend on social interaction with a caregiver who serves the infant's needs for an emotional attachment. In cases of early severe deprivation, the consequences for infant development are sometimes impressive. Rutter's studies of children adopted into the United Kingdom following early severe deprivation in Romania showed that dramatic damage stemmed from institutional deprivation, and its heterogeneity in outcome includes cognitive deficit and attachment problems, but also quasi‐autistic patterns (e.g., Rutter et al., 2007a, 2007b; Croft et al., 2007).

The quality of social interaction depends on a reciprocal process, an active dialogue between parent and child based on the infant's early competencies and the mother's (or father's) capacity for tuning. Micro‐analysis of film records examining rhythms and patterns in mother–infant face‐to‐face interaction in the neonatal period (Brazelton et al., 1975) and early communication (Condon and Sander, 1974; Stern et al., 1975) have emphasized the importance of synchrony and co‐modality of these early interactions for infants' development.

Combining early language acquisition and social interaction

Interestingly, researchers in language acquisition and researchers in early social interactions have encountered an important peculiarity that affects both the language and social development of infants: the way adults speak to infants. A special kind of speech that is directed towards infants, often called “motherese” (“parentese”), characterized by higher pitch, slower tempo, and exaggerated intonation contours (Fernald, 1985; Grieser and Kuhl, 1988), seems to play an important role in both social interaction and language development. Studies have revealed that this particular prosody may be responsible for attracting infants' attention, conveying emotional affect, and conveying language‐specific phonological information (Karzon, 1985; Hirsh‐Pasek et al., 1987; Kemler Nelson et al., 1989; Fernald, 1985; Fernald and Kuhl, 1987). (1) When given a choice, infants show a preference for mothers' infant‐directed speech (so‐called motherese) as opposed to adult‐directed speech (Pegg et al., 1992; Cooper and Aslin, 1990; Werker and McLeod, 1989; Fernald, 1985; Glenn and Cunningham, 1983). This particular prosody helps to engage and maintain the limited attention of the baby. (2) Motherese contains particularly good phonetic exemplars – sounds that are clearer, longer, and more distinct from one another (e.g., vowel hyperarticulation) – when compared to adult‐directed speech (Kuhl et al., 1997; Burnham et al., 2002).

In addition, the baby's reactions amplify the contours of prosody curves in the mother's voice (Burnham et al., 2002). Mothers' infant‐directed speech was found to depend on the quality of infants' responsiveness, which suggests that infants are actively involved in early interactions (Braarud and Stormark, 2008).

Is it possible to implicate motherese impoverishment in the pathogenesis of autism?

In our view, the well‐known autistic impairments in language, cognition and social development, as well as the tendencies toward self‐absorption, perseveration and self stimulation (Volkmar and Pauls, 2003), may be downstream effects of primary difficulties in the ability to engage in interactions involving emotional signals, motor gestures and communicative acts directed to others. This view is also supported by testimony of adult individuals with autism who expressed themselves (Chamak et al., 2008). If learning depends on a normal social interest in people and the signals they produce, children with autism, who lack a social interest, may be at a cumulative disadvantage in language learning. Their poor response to parental engagement may impoverish both parental engagement and motherese production over time. As a consequence, this impoverishment will reinforce social withdrawal and language acquisition delay.

In a previous exploratory study, we observed home movie sequences in which a withdrawn infant who will later develop autism may suddenly appear joyful when the parent implements a vocal expression using motherese. During such interactions, infants and toddlers with autism exhibit social focal attention: their faces light up, unexpected interactive skills appear, and real protodialogues expand (Laznik et al., 2005). However, we were unable to find any study comparing interest in motherese speech versus other human speech in children with autism. Even if we know that autistic children can process some aspects of human voices (Groen et al., 2008), however, studies have shown that autistic individuals display no specific cortical activation in response to human voices (e.g., Zilbovicius et al., 2006). Further, five‐year‐old autistic children do not show the expected preference for their mother's speech when given it as a choice with the noise of superimposed voices (Klin, 1991). Similarly, 32‐ to 52‐month‐old children with an ASD have a significant listening preference for electronic non‐speech motherese analogue signals in comparison to motherese signals (Kuhl et al., 2005). Given that home movies offer a unique opportunity to follow infant development and parent–infant interactions, we planned to use a multi‐disciplinary approach with a child psychiatrist, developmental psychologist, psycholinguist and engineer to test the following research questions: (1) Are infants who later develop autism initially equipped to respond to motherese specifically? In other words, is this competency stable and correlated with positive interactions? (2) As motherese amplification is bidirectional, does the parental quantity of motherese decrease overtime?

Aims of the present paper

In this paper, we describe (1) a computerized algorithm to detect motherese developed for this research; (2) its characteristics in extracting motherese from home video sequences. Given that micro‐analysis of home video is highly time consuming, we consider the production of this algorithm as the first step of our multi‐disciplinary research project. Indeed, even if motherese is clearly defined in terms of acoustic properties, the modelling and detection is expected to be difficult, as is the case with the majority of emotional speech, because the characterization of spontaneous and affective speech in terms of features is still an open question since several parameters have been proposed in the literature (Schuller et al., 2007). As a starting‐point following the acoustic properties of motherese, we characterized verbal interactions by the extraction of supra‐segmental features (prosody). Given that home movies are not recorded by professionals and often contain adverse conditions (e.g., with regard to noise, the camera, or microphones), however, acoustic segmentation of home movies shows that segmental features play a major role in robustness (Schuller et al., 2007). Consequently, the utterances are characterized by both segmental (Mel Frequency Cepstrum Coefficients, MFCCs) and supra‐segmental (e.g., statistics with regard to fundamental frequency, energy, and duration) features. These features, along with the known emotional labels of a training set of emotional utterances, are provided as input to a machine learning algorithm. Here, we compared the performance of two learning algorithms, GMM (Gaussian Mixture Model) and k‐nn (k‐nearest neighbours), on segmental and non‐segmental features alone using the ROC (receiver operating characteristic) analysis method. To improve detection, we also investigated several combinations or fusion schemes (segmental/supra‐segmental; GMM/k‐nn) to select the one that is optimal in comparison to manual segmentation of motherese sequences from home videos.

Method

Database

The speech corpus used in this experiment is a collection of natural and spontaneous interactions. This corpus contains expressions of non‐linguistic communication (affective intent) conveyed by a parent to a preverbal child. The corpus consists of recordings of Italian mothers and fathers addressing their infants. We decided to focus on the analysis of home movies, as it enables a longitudinal study (months or years) and gives information about early behaviours of autistic infants long before the diagnosis is made by clinicians. All sequences were extracted from the Pisa home video database, which includes home movies from the first 18 months of life for three groups of children aged four to seven years (Maestro et al., 2005a, 2005b). Children are matched for gender and socio‐economic status. The first group was composed of 15 children (Male/Female 10:5) with a diagnosis of autism based on the Autism Diagnostic Interview‐Revised (ADI‐R) (Rutter et al., 2003). All cases with Childhood Autism Rating Scale (CARS) total‐scores (Schopler et al., 1988) below 30 were excluded. The second group was composed of 12 children (Male/Female 7:5) with a diagnosis of Mental Retardation (MR) who had a non‐autistic CARS total score under 25. Children with MR or AD secondary to neuropsychiatric syndromes (e.g., Fragile‐X, Rett, or Down Syndrome) or evident neurological deficits or physical anomalies were excluded. Participants in the AD or MR group were administered the Griffiths Mental Developmental Scale or Wechsler Intelligent Scale in order to determine intellectual functioning. For both AD [mean IQ = 59.26; standard deviation (SD) = 8.49] and MR (mean IQ = 56.82; SD = 8.16) groups, the composite IQ score was below 70. A third group was composed of 15 typical developing (TD) children (Male/Female 9:6) recruited from children attending a local kindergarten.

The large size of this corpus, however, makes it inconvenient for human review. For the development of the first algorithm, we focused on one home video of superior acoustic quality that totalled three hours of the first year of life of an infant who later developed autism (at six years: ADI‐R communication score = 9; ADI‐R social interest score = 12; ADI‐R repeated interest score = 5; CARS total score = 36; IQ = 71). The verbal interactions of the infant's mother were carefully annotated by two blind psycholinguists into two categories: infant‐directed speech (motherese) and adult‐directed speech (normal) [Cohen's kappa = 0.82; 95% confidence interval (CI) = 0.75–0.90]. From this manual annotation, we extracted 100 utterances for each category. The utterances are typically between 0.5 seconds and 4 seconds in length. To conduct a secondary performance characteristic study, we collected an independent set of utterances (50 motherese and 50 normal speech) extracted from 10 randomly selected home movies (five from the AD group and five from the TD group).

System description

Automatic speech sounds segmentation requires several steps: feature extraction, classification and decision fusion are conducted sequentially. These steps can be divided into two major processing phases, namely the training phase and the test or classification phase. Figure 1 shows a schematic overview, which is described in more detail in the following paragraphs. The first step, namely feature extraction, shown in Figure 1 is needed in both training and test. The second step, called classification, aims at classifying the whole utterances. The last step, which is called decision fusion, is necessary for classification and combines the different streams to form one common output.

Figure 1.

Motherese classification system used in the algorithm.

Feature extraction

After the segmentation of the database into utterances, the next step is to extract features, resulting in a representation of the speech signal as input for the motherese classification system. In this paper we evaluate two approaches, respectively termed segmental and supra‐segmental features. The first is characterized by the MFCCs, while the second is characterized by statistical measures of both the fundamental frequency (F0) and the short‐time energy. A 20 ms window is used, and the overlapping between adjacent frames is halved. A parameterized vector of order 16 was computed. The supra‐segmental features are characterized by three statistics (mean, variance and range) on both F0 and short‐time energy, resulting in a six‐dimensional vector. One should note that the duration of the acoustic events is not directly characterized as a feature but is taken into account during the classification process by a weighting factor. The feature vectors are normalized (zero mean, unit standard deviation).

Classification

In this study, two different classifiers – the k‐nn classifier and the GMM‐based classifier – were investigated. The k‐nn is a distanced based classifier often used in pattern recognition. GMM is a statistical signal model trained by sequences of feature vectors that are representative of the input signal (Reynolds, 1995).

A posteriori probabilities estimation

The GMM is adopted to represent the distribution of the features. Under the assumption that the feature vector sequence x = {x ₁, x ₂, . . ., x_n} is an independent identical distribution sequence, the estimated distribution of the d‐dimensional feature vector x is a weighted sum of M component Gaussian densities g(μ,Σ), each parameterized by a mean vector μ_i and covariance matrix Σ_i; the mixture density for the model C _m is defined as:

(1)

Each component density is a d‐variate Gaussian function:

(2)

The mixture weights ω_i satisfy the following constraint:

(3)

The feature vector x is then modelled by the following a posteriori probability:

(4)

where P(C _m) is the prior probability for class C _m, we assume equal prior probabilities. We use the expectation maximization (EM) algorithm for the mixtures to obtain maximum likelihood. The k‐nn classifier is a non‐parametric technique which classifies the input vector with the label of the majority k‐nn (prototypes) (Duda et al., 2000). In order to keep a common framework with the statistical classifier (GMM), we estimate the a posteriori probability that a given feature vector x belongs to class C _m using k‐nn estimation (Duda et al., 2000):

(5)

where k _m denotes the number of prototypes that belong to the class C _m among the k‐nn.

Segmental and supra‐segmental based characterization

Segmental features (i.e. MFCC) are extracted from all the frames of an utterance U_x independently to the voiced or unvoiced parts. One should note that the nature of the segments can also be exploited (vowels/consonants) (Ringeval and Chetouani, 2008). A posteriori probabilities are then estimated by both GMM and k‐nn classifiers and are respectively termed P _gmm,seg(C _m|U_x) and P _knn,seg(C _m|U_x).

The classification of supra‐segmental features follows the segment‐based approach (SBA) (Shami and Verhelst, 2007). An utterance U_x is segmented into N voiced segments (F_xi) obtained by F0 extraction (see earlier). Local estimation of a posteriori probabilities is carried out for each segment. The utterance classification combines the N local estimations.

(6)

The duration of the segments is introduced as weights of the a posteriori probabilities: importance of the voiced segment (length(F_xi)). The estimation is also carried out by the two classifiers, resulting in supra‐segmental characterizations: P _gmm, supra(C _m|U _x) and P _knn, supra(C _m|U _x).

Fusion

The segmental and supra‐segmental characterizations provide different temporal information, and combining them should improve the accuracy of the detector. Many decision techniques can be employed (Kuncheva, 2004), but we investigated a simple weighted sum of likelihoods from the different classifiers:

C_l = λ. log(P_seg(C_m|U_x)) + (1 − λ). log(P sup ra(C_m|U_x)) (7)

with l = 1 (motherese) or 2 (normal directed speech); λ denotes the weighting coefficient. For the GMM classifier, the likelihoods can be easily computed from the a posteriori probabilities [P _gmm,seg(C _m|U_x), P _gmm,supra(C _m|U_x)] (Reynolds, 1995). However, the k‐nn estimation can produce a null a posteriori probability incompatible with the computation of the likelihood. We used a solution recently tested by Kim et al. (2007), in which the a posteriori probability is used instead of the log probability of the k‐nn:

(8)

Consequently, for the k‐nn classifier we used Equation 7, while for the GMM the likelihood is conventionally computed. We investigated cross combinations (Table 1).

Table 1.

Table of combinations of classifiers (GMM/k‐nn) × features (segmental/supra‐segmental)

Combination 1	P knn,seg	P knn,supra
Combination 2	P gmm,seg	P gmm,supra
Combination 3	P knn,seg	P gmm,supra
Combination 4	P gmm,seg	P knn,supra
Combination 5	P gmm,seg	P knn,seg
Combination 6	P gmm,supra	P knn,supra

Note: GMM, Gaussian Mixture Model; k‐nn, k‐nearest neighbours; seg, segmental; supra, supra‐segmental.

Results

Classifier configuration

First, to find the optimal structure of our classifier, we had to adjust the parameters: the number of Gaussians (M) for the GMM classifier and the number of neighbours (K) for the k‐nn classifier. We searched for the optimal configuration in terms of accuracy. Table 2 shows the best configuration of GMM and k‐nn with segmental and supra‐segmental features and further shows that the GMM classifier trained with prosody features outperformed the other classifiers in terms of accuracy.

Table 2.

Accuracy of optimal configurations for GMM and k‐nn classifiers according to segmental and supra‐segmental features

	Segmental	Supra‐segmental
k‐nn	72.5% (K = 11)	61% (K = 7)
GMM	78% (M = 15)	82% (M = 16)

Note: GMM, Gaussian Mixture Model; k‐nn, k‐nearest neighbours; M, number of Gaussians for the GMM classifier; K, number of neighbours for the k‐nn classifier.

Fusion of best system

In this study the performance of the classifier was quoted using class sensitivities, predictivities and overall accuracy, and the optimal classifiers were determined by employing ROC graphs to show the trade‐off between the hit and false positive rates. A ROC curve represents the trade‐off between the false acceptance rate (FAR) and the false rejection rate as the classifier output threshold value is varied. Two quantitative measures of verification performance, the equal error rate (EER) and the area under the ROC curve (AUC), were calculated. All calculations were conducted with Matlab (version 6). For best configuration data, results were given with the 95% CIs that were estimated using Cornuéjols et al. (2002) method. It should be noted that while EER represents the performance of a classifier at only one operating threshold, the AUC represents the overall performance of the classifier over the entire range of thresholds. Hence, we employed AUC and not EER to compare the verification performance of two classifiers and their combination. However, the result shown in Table 2 motivated an investigation of the fusion of both features and classifiers following the statistical approach described in the previous section. Improvement by the combination of features and classifiers is known to be efficient (Kuncheva, 2004). However, caution should be used, because the fusion of best configurations does not always give better results; the efficiency will depend on the nature of the errors produced by the classifiers (independent versus dependent) (Kuncheva, 2004). Table 1 and the previous section show that six different fusion schemes could be investigated [Combination 1 (Comb₁) to Combination 6 (Comb₆)]. For each of them, we optimized parameter classifiers (K value for k‐nn and M value for GMM, respectively) and weighting λ (Equation 6) parameters of the fusion. In Figure 2, we can see that for the k‐nn classifier, the best scores (0.8113/0.812) were obtained with an important contribution of the segmental features (λ = 0.8), which is in agreement with the results obtained without the fusion (Table 2). The best GMM results (0.932/0.932) are obtained with a weighting factor equal to 0.6, revealing a balance between the two features. Table 3 summarizes the best results in terms of accuracy as well as the positive predictive value (PPV) and negative predictive value (NPV) for each classifier fusion (top section) and cross‐classifier fusion (bottom section). Table 3 shows that different combinations of segmental and supra‐segmental features can reach similar accuracies with different PPVs. With regard to the k‐nn method, the best scores (accuracy = 74%; PPV = 77.91%) were obtained with an important contribution of the segmental features (λ = 0.9). The best GMM results (accuracy = 87.5%; PPV = 88.47%) were obtained with a weighting factor equal to 0.4 (i.e. a balance between the two features). We also investigated cross‐classifier fusion (Table 3, bottom). The accuracy of this method can be lower than that for the single classifiers (Table 2). However, significant improvements (90.7%) in the PPV were reached for Combination 3: P _knn,seg × P _gmm,supra combination (λ = 0.6). This result reveals the importance of evaluation metrics for fusion, which is dependent on the task in this study (motherese detection).

Figure 2.

ROC curves for Combination 1 (Comb₁) and Combination 2 (Comb₂). Combination 1 = P _knn,seg × P _knn,supra; Combination 2 = P _gmm,seg × P _gmm,supra; λ = weighting coefficient used in the equation fusion for each combination.

Table 3.

Accuracy, positive predictive value (PPV), and negative predictive value (NPV) of fusion configurations according to classifiers and segmental and supra‐segmental feature parameters

		Accuracy	PPV	NPV	Λ
Using only one classifier
Combination 1 k‐nn, seg × k‐nn, supra	k‐nn, seg (K = 11)	74%	77.91%	71.05%	0.9
	k‐nn, supra (K = 1)
	k‐nn, seg (K = 7)	72.5%	74.42%	71.84%	0.8
	k‐nn, supra (K = 11)
Combination 2 GMM, seg × GMM, supra	GMM, seg (M = 12)	87.5%	88.47%	86.41%	0.4
	GMM,supra (M = 15)
	GMM, seg (M = 16)	86.5%	84.85%	88.42%	0.5
	GMM, supra (M = 16)
Combining k‐nn and GMM classifiers
Combination 3 K‐nn, seg × GMM, supra	k‐nn, seg (K = 11)	85.5%	89.41%	83.81%	0.7
	GMM, supra (M = 12)
	K‐nn, seg (K = 5)	85%	90.7%	82.69%	0.6
	GMM, supra (M = 12)
Combination 4 GMM, seg × k‐nn, supra	GMM, seg (M = 15)	80.5%	79.61%	81.44%	0.2
	k‐nn, supra (K = 11)
	GMM, seg (M = 16)	79.5%	77.57%	81.72	0.3
	k‐nn, supra (K = 1)

Note: PPV (positive predictive value) = TruePositive/(TruePositive + FalsePositive). PPV evaluates the proportion of correctly detected motherese utterances out of all utterances labelled as motherese. PPV can be viewed as the reliability of positive predictions induced by the classifier. NPV (negative predictive value) gives similar information about the detection of normal speech.

In sum, the best fusion configuration used only the GMM classifier for both segmental and supra‐segmental features (M = 12 and M = 15, respectively, and λ = 0.4). Performance of this fusion corresponds to Combination 2 from Table 3, and were as follow: accuracy = 87.5% (95%CI = 82.91–92.08%); PPV = 88.47% (95%CI = 83.03–95.18%); NPV = 86.41% (95%CI = 79.4–92.88%). We also investigated the performance of the detector with these fusion parameters in detecting motherese versus normal speech within a second set of utterances extracted from 10 randomly‐selected home videos with 12 independent speakers. Performances under speaker‐independent conditions were as follows: accuracy = 82% (95%CI = 73.87–89.58%); PPV = 86.36% (95%CI = 66.52–89.48%); NPV = 77.55% (95%CI = 76.73–96.6%).

Discussion

Motherese detector validity

Our goal was to develop a motherese detector by investigating different features and classifiers. Using classification techniques that are often used in speech and speaker recognition (GMM and k‐nn), we have developed a motherese detection system and tested it on mode dependent of speaker. The fusion of features and classifiers was also investigated. We obtained results from which we can draw several interesting conclusions. First, our results show that segmental features alone contain much useful information for discrimination between motherese and adult direct speech, since they outperform supra‐segmental features. Thus, we can conclude that segmental features can be used alone. However, according to our detection results, prosodic features are also very promising. Based on the previous two conclusions, we combined classifiers that use segmental features with classifiers that use supra‐segmental features and found that this combination improves the performance of our motherese detector considerably.

Thus, we can conclude that a classifier based on segmental features alone can be used to discriminate between motherese and normal‐directed‐speech, but a significant improvement can be achieved in most cases when this classifier is fused with another that is based on prosodic features. Furthermore, fusing scores from k‐nn and GMM is also very fruitful.

Limitations and strengths

For our motherese classification experiments, we used only speech excerpts that were already segmented (based on human transcription). In other words, detection of the onset and offset of motherese was not investigated in this study but can be addressed in a follow‐up study. Detection of the onset and offset of motherese (motherese segmentation) can be seen as a separate problem that gives rise to other interesting questions, such as how to define the beginning and end of motherese and what kind of evaluation measures to use. These are problems that can typically be addressed within a Hidden Markov Model framework.

In its best configuration, the novel detector had positive and negative predictive values reaching 90%; this level of prediction is suitable for further studies of home videos. When we explored the motherese detector's performance on sequences blindly validated by two psycholinguists (speaker‐independent), the performance of the detector remained very good (accuracy = 82%). As we hypothesize the detector to be language independent, however, we still need to explore the motherese detector's performance on sequences randomly extracted from different languages. As noted, although motherese is itself a highly variable and complex signal (Fernald and Kuhl, 1987), it would be interesting to investigate to what extent motherese is dependent on the speaker, culture, etc.

Finally, the aim of this study was to develop a motherese detector to enable emotion classification. Our plan for emotion recognition is based on the fact that emotion is expressed in speech by audible paralinguistic features or events, which can be seen as the “building blocks” or “emotion features” of a particular emotion. Adding visual information could further help to improve emotion detection. Our plan is thus to perform emotion classification via the detection of audible paralinguistic events. In this study, we have developed a motherese detector to provide a first step in this plan towards a future emotion classifier.

Possible applications to the field of autism

Despite the limitations listed earlier, we are now systematically exploring in home videos from the Pisa database (Maestro et al., 2005b) the natural courses of motherese and parent–infant interaction and their co‐occurrence. The pattern of parent–infant interactions has been extensively explored in infants later diagnosed with autism (for reviews see Saint‐Georges et al., 2010; Palomo et al., 2006). However, whether motherese and positive interactions co‐occur has not been investigated. This will be tested on randomized sequences extracted from the database using group comparisons matched for age and sex: autism versus typically developing; autism versus children with intellectual disability; early onset autism versus late onset autism. These analyses are made possible by the availability of several cases matched for age and sex with normal and delayed developing controls (Maestro et al., 2005b). We hypothesize that motherese should be correlated with positive interactions both in autism and non‐autism, but to a lesser extent in autism.

The second analysis will focus on a longitudinal view of motherese using a subset of cases with several videos at different ages. We hypothesize that in early onset autism, motherese will decrease over time in terms of relative frequency due to lack of interactive feedback from the child (Muratori and Maestro, 2007). This deficit in stimulation would have developmental consequences, which would lead in particular to the specific social and language impairments found in older autistic children (Zilbovicius et al., 2006). Indeed, motherese prosody is related to the mother's emotions and seems an authentic marker of the quality of parent–child interactions. In their infants' absence, mothers cannot produce “faked” motherese into a microphone (Fernald and Simon, 1984): the full range of prosodic modifications in mother's speech is evoked only in the presence of the infant. On the contrary, baby's reactions improve the amplitude of prosody contours in mother's speech (Burnham et al., 2002). But when a baby seems indifferent, can his/her parents still continue to produce motherese? If motherese can no longer be produced by such parents, the deficit of motherese would appear to be one of the cofactors in the development of an interactive vicious circle. Figure 3 shows how the detector may be useful in performing such a study. In a sequence of duration 12 seconds during which a positive interaction occurred (note the baby's smile), the detector can automatically analyse the sound signal, prosodic characteristics, and main speaker and determine whether motherese is being produced.

Figure 3.

Use of the detector as a tool to study motherese in a home video. In a sequence of duration 12 seconds during which a positive interaction occurred (see baby's smile), the detector automatically analyses the acoustic signal, prosodic characteristics (frequency, in hertz, in blue; intensity, in dB, in red), main speaker, and whether motherese is produced. In parallel, existing positive interactions are computerized with the same temporal scale in the Pisa home video database. The red vertical line indicates the timing of the baby's picture shown (in this case, eight seconds after the sequence's starting point).

Possible applications outside the field of autism

If our previous hypotheses are confirmed (i.e. if a decrease of motherese appears as a cofactor in the pathogenic process), a possible application for infants at risk for autism could be to improve parental motherese by training using a screen warning for motherese production (biofeedback). As motherese has been shown to improve responsiveness and learning in various developmentally disabled children (Santarcangelo and Dyer, 1988), such training could also be useful for nurses, teachers, or other professionals working in the field of infant mental health outside the field of autism (e.g., maternal depression).

As motherese is an important factor and a valuable tool to study both first interactions and language development, we can easily imagine possible applications of its automatic detection in various research projects performed with an experimental or naturalistic design (e.g., Braarud and Stormark, 2008). Similarly, motherese prosody reflects the emotional affect conveyed in the voice (Trainor et al., 2000). Because manual investigation are highly time consuming, automatic motherese detection could also be useful for various types of research on emotional speech (e.g., Beaucousin et al., 2007).

Conclusion

By using different features, segmental and supra‐segmental, we were able to automatically discriminate motherese segments from normal speech segments with low error rates. We conclude that the motherese detector is a powerful tool to study motherese in home videos, allowing in‐depth study of large samples of sequences and exhibiting good performance characteristics.

Declaration of interest statement

The authors have no competing interests.