An Automated Approach to Examining Pausing in the Speech of People With Dementia

Abstract

Dementia is a common neurodegenerative condition involving the deterioration of cognitive and communication skills. Pausing in the speech of people with dementia is a dysfluency that may be used to signal conversational trouble in social interaction. This study aimed to examine the speech-pausing profile within picture description samples from people with dementia and healthy controls (HCs) within the DementiaBank database using the Calpy computational speech processing toolkit. Sixty English-speaking participants between the ages of 53 and 88 years (M_age = 67.43, SD = 8.33; 42 females) were included in the study: 20 participants with mild cognitive impairment, 20 participants with moderate cognitive impairment, and 20 HCs. Quantitative analysis shows a progressive increase in the duration of pausing between HCs, the mild dementia group, and the moderate dementia group, respectively.

Introduction

Dementia is a neurodegenerative condition involving deterioration of cognitive functioning. In conjunction with memory and executive functioning impairments ¹ , the progressive decline in language skills is a hallmark characteristic of dementia which places a considerable burden on both people with dementia and their caregivers. ^2,3 Communication difficulties in dementia may be attributed to the decay of cognitive-linguistic abilities which can manifest as word-finding difficulties, reduced verbal fluency, grammatical and syntactical errors, topic maintenance problems, and reduced communicative coherence and comprehension capacity. ^1,4

Pausing in Dementia Speech

Pausing in the speech of people with dementia is a behavioral hallmark that is typically regarded as a dysfluency that may be used to signify conversational trouble in a social interaction. ⁵ Healthy adults tend to follow predictable patterns of pausing, in order to make conceptual, syntactic, and lexical decisions, in addition to expressing emotional and psychological information. ⁶ Consequently, neurological conditions characterized by cognitive-linguistic deficits, such as dementia, may be expected to result in atypical patterns of pausing in speech. Studies examining discourse in people with dementia using autobiographical narratives have observed increased difficulty in finding words, less efficient speech, and increased duration and number of pauses, compared to healthy adults. ^{7 -12}

Pausing in speech can be classified into 2 types; silent pauses, which represent the silent portion of speech production between utterances, and filled pauses, which indicate hesitation from the speaker (eg, “uh”, “um”, and “well”). Indeed, the occurrence of some pauses is the inevitable consequence of breathing, ¹³ along with grammatical pauses that occur at natural punctuation points. ¹⁴ However, pausing in the speech of people with dementia is often longer and more frequent, which may reflect difficulties with semantic and lexical decision-making, cognitive load, and topic familiarity. ^14,15 Silent pauses are the focus of the current study.

Machine Learning Approaches to Discourse Analysis

Automated approaches for examining conversational dynamics between people with dementia and their caregivers have been previously investigated, ^16,17 and machine learning approaches to assist with communication breakdown in dementia have already been achieved to some extent. ^{16 -19} For example, Discursis is an automated text-analytic tool that provides quantification and visualization of communication behavior between 2 or more speakers. ^20,21 Discursis has previously been applied to conversations of people with dementia to identify topics that facilitate conversational engagement ¹⁷ and also to identify the effectiveness of various communication strategies used by caregivers when conversing with people with dementia. ¹⁶ Although Discursis has successfully demonstrated the capacity to generate measures relating to multiple aspects of conversation behavior, it requires text-based data input and focuses on analyzing the content of conversations as opposed to the properties of speech. Thus, machine learning algorithms that detect conversational trouble using speech-based data are needed.

Automated Software to Detect Pausing in the Speech of People With Dementia

Calpy is an OpenSource Speech Processing Toolkit that was recently developed at the University of Queensland. ²² Calpy is a novel, automated pause identification and coding system which was developed to identify pauses in prerecorded speech data, simplifying the manner with which narrative or conversational data can be collected and subsequently analyzed. Through the input of speech-based data (as opposed to text-based data), Calpy has the ability to quantify communication behavior and examine the turn-taking dynamics during discourse. However, in the current study, monologue data were analyzed exclusively as an initial step. The study of pausing in the narrative speech of people with dementia may contribute to our understanding of conventional expectations of pauses, wait-times, and false starts, which may subsequently assist with better understanding pause dynamics arising within other speech contexts. In addition, it has been suggested that pausing in speech is a possible diagnostic marker of cognitive changes in dementia. ⁶ Thus, an analysis of pausing in the speech of people with dementia is warranted in order to better understand pausing within the context of disease severity and progression.

While previous research has investigated pausing in the speech of people with dementia using machine learning approaches to assist with diagnosing semantic dementia, ²³ in addition to the automatic detection of early-stage dementia through speech analysis, ^17,24 the automated approaches used in previous research are not robust enough for our purposes. Calpy was developed to detect and identify pauses within naturally occurring speech, with the additional advantage of examining the turn-taking dynamics during discourse in terms of speaker uptakes, overlaps, and overtakes. The current study applied Calpy to monologue data using picture description samples, as opposed to conversational data.

Study Aims

This study aimed to examine the number and duration of pausing in speech for people with dementia with mild or moderate cognitive impairment, compared to healthy adult counterparts to ascertain whether pausing during narrative speech is differentially affected by disease severity for people with dementia. On the basis of previous research demonstrating the common nature of word-finding difficulties and pausing in the speech of people with dementia, ^14,15 it was predicted that there would be an upward trajectory in the pausing profile (ie, duration and number of pauses in speech) for that of healthy controls (HCs), people living with dementia with mild cognitive impairment, and people with dementia with moderate cognitive impairment, respectively.

Method

Corpus

The current study used the Pitt Corpus ²⁵ of the DementiaBank database. Contained in this corpus are audio recordings and transcriptions of participants undertaking the Cookie Theft picture description task from the Boston Diagnostic Aphasia Examination. ²⁶ The data set is password protected and only available for research purposes upon request.

Participants

The corpus includes participants with possible/probable Alzheimer disease (AD) and HCs. Participants with other diagnoses (ie, psychological diagnoses such as anxiety or depression) were excluded from the current study due to their limited numbers in the corpus. Inclusion criteria for the current study required participants to have a diagnosis of probable AD with mild or moderate cognitive impairment according to the Mini-Mental State Examination (MMSE) ²⁷ together with the availability of audio files for analysis. An equal number of HC participants were also selected from the corpus for inclusion. Participants were excluded (n = 11) due to their audio files containing excessive background noise, including the audible sound of other participants being tested in the background. On this basis, data from 60 participants were included in the study: 20 with probable AD and mild cognitive impairment (MMSE scores between 19 and 24 points); 20 with probable AD and moderate cognitive impairment (MMSE scores between 10 and 18 points); and 20 HC participants with MMSE scores in the normal range (ie, scores between 25 and 30 points; see Table 1 for demographic characteristics). Two analysis of variances (ANOVAs) revealed no significant between-group differences in entry age [F(2,57) = 0.402, P = .671, n ² _p = 0.014] or onset age [F(1,38) = 0.630, P = .432, n ² _p = 0.016]. Additionally, while the total sample was unbalanced for sex, a χ² test revealed this difference was not significant, χ²(2) = 0.476, P = .788 (see Table 1 for distribution of males and females).

Table 1.

Demographic Characteristics of the Sample.

Demographics	Moderate dementia	Mild dementia	Healthy controls
N	20	20	20
Entry age (M ± SD, range)	72.20 ± 7.62, 57-85	70.05 ± 9.42, 53-88	66.90 ± 8.73, 50-78
Onset age (M ± SD, range)	68.80 ± 7.51, 54-82	66.60 ± 8.93, 51-84	-
Males	6	5	7
Females	14	15	13

Speech Sample

All participants undertook the Cookie Theft picture description task from the Boston Diagnostic Aphasia Examination. ²⁶ Audio recordings ranged in length from 26 seconds to 2 minutes and 35 seconds (mean length 1 minute and 2 seconds, SD = 24.02 seconds). All preprocessing of audio waveforms was performed using Audacity ²⁸ software. Prior to the analysis, each audio recording was reviewed for the presence of speech from the interviewer (ie, at the beginning or end of the recording), which was then removed from the waveforms. Filled pauses were maintained; however, they were not measured as a pause event.

Measures

Calpy

Calpy was developed as a visualization system for audio speech–based data to assist users with investigating complex and unstructured narrative or conversational communication behavior. Calpy is an open-source system, capable of processing speech that is independent of language. Calpy implements a range of existing speech and language processing algorithms and techniques, all of which are documented in the software library. The algorithm for detecting speech activity is based on the work of Sakhnov et al ²⁹ and extracts features from input signals, compares these values with thresholds extracted from sound-only periods, and classifies these moments as speech or silence. Put simply, Calpy determines if speech is present or absent even in background noise. Thus, pauses are defined as the absence of sound beyond an identified threshold. Sakhnov et al ²⁹ reported a misclassification rate between 5% and 7% for phone audio, which share similar noise profiles to the cases studied here. Audio files are input as .wav or .zip files with the relevant analytic command executed to produce output data. While Calpy algorithm demonstrates robustness under noisy conditions, users should be mindful that excessive background noise (>20dB) or unusual audio effects may impair the accuracy of the results. ³⁰ Thus, it is ideal to ensure accuracy in audio recordings (ie, clean speech sounds and no background noise) prior to processing.

Pause duration

Pause duration was detected via periods of silence in the acoustic waveform. Silences were considered a pause if the silences between speech were greater than 150 milliseconds in duration. This duration was selected on the basis of previous research using monologue speech samples. ^31,32 Pause duration was calculated in milliseconds and also as a percentage of the total audio recording duration (speech duration + pause duration = total sample duration). ³³ Mean pause duration was defined as the average duration of all pauses in milliseconds, and the total number of pauses was also calculated.

Prior research on the appropriate length of monologue pauses is limited; however, an appropriate “normal” conversational pause length is thought to be approximately 1 second. ³⁴ It is well established that people with dementia often experience delayed turn-taking and can require additional processing time. ³⁵ Thus, pauses of 2 seconds or longer are generally considered indicative of conversational trouble for people with dementia. ³⁶ While this study focusses on monologue, and not on conversation, the number of pauses that were 2 seconds or longer was also recorded and examined as an additional variable.

Speech duration

Speech was defined as the point in the acoustic waveform where the signal exceeded the background noise level. Total speech duration was calculated in milliseconds and also as a percentage of the total audio recording duration. Phrases are defined as sections of continuous speech between pauses. The mean phrase duration was obtained and is defined as the average duration of all phrases in seconds.

Statistical Analyses

Statistical analyses were performed using IBM SPSS Statistics (v.23). A series of one-way analyses of covariance (ANCOVAs) were conducted to examine the research questions. Given that the audio recordings varied in length across participants, the total recording length was also controlled for as a covariate in each of the statistical analyses. Additionally, all post hoc analyses were conducted using the Bonferroni adjustment to control for family-wise error rate. All pause and speech durations were calculated in milliseconds; however, the total pause and speech duration analyses were converted and reported in seconds to assist with ease of reading.

Results

Mean Pause Duration

Figure 1 shows the mean pause duration in milliseconds for each group. A 1-way ANCOVA revealed a significant difference in the mean pause duration between groups [F(2,56) = 12.83, P < .001, n² _p = 0.31]. Post hoc tests revealed no significant difference in the mean pause duration between the HC group (M = 643.70, SD = 288.29) and the mild dementia group (M = 985.71, SD = 388.38; P = .06); however, the moderate dementia group (M = 1385.03, SD = 601.78) had a significantly longer mean pause duration than both the HC group (P < .001) and the mild dementia group (P = .03).

Figure 1.

Mean pause duration in milliseconds (with standard errors). One-way ANCOVA significance value < .001, and post hoc pairwise significance is indicated as *P < .05, **P < .01. ANCOVA indicates analysis of covariance.

Total Pause Duration

The total pause duration was calculated in milliseconds for each group and converted to seconds. A 1-way ANCOVA revealed a significant difference in the total pause duration between groups [F(2,56) = 19.00, P < .001, n ² _p = 0.40]. Post hoc tests revealed a significantly lower total pause duration for the HC group (M = 21.71, SD = 12.02) relative to the mild dementia group (M = 32.56, SD = 8.50; P = .01) and the moderate dementia group (M = 47.28, SD = 21.08; P < .001). Moreover, the mild dementia group had a significantly lower total pause duration than the moderate dementia group (P = .01).

Additionally, a 1-way ANOVA was conducted to compare the percentage of pause duration (of the total audio recording) between the 3 groups (see Figure 2). Since the homogeneity of variance was violated, Welch F test was used. The ANOVA revealed a significant difference between the groups [F(2,36.56) = 17.22, P < .001, n² _p = 0.39]. Post hoc tests, using the Games-Howell procedure, showed that the HC group (M = 41.38, SD = 19.17) had a significantly lower percentage of pause duration compared to the mild dementia group (M = 56.42, SD = 11.45; P = .01) and the moderate dementia group (M = 68.02, SD = 11.08; P < .001). Moreover, the mild dementia group had a significantly lower percentage of pause duration than the moderate dementia group (P = .01).

Figure 2.

Pause duration as percentage of total recording (with SE). One-way ANCOVA significance value < .001, and post hoc pairwise significance is indicated as *P < .05, **P < .01. ANCOVA indicates analysis of covariance.

Number of All Pauses and Number of Pauses 2 Seconds or Greater

A 1-way ANCOVA on the number of pauses revealed no significant differences between groups [F(2,56) = 0.17, P = .85, n ² _p = 0.01], and post hoc analyses were also all nonsignificant.

Figure 3 shows the number of pauses 2 seconds or greater in duration for each group. A 1-way ANCOVA revealed a significant difference in the number of these pauses between groups [F(2,56) = 20.91, P < .001, n ² _p = 0.43]. Post hoc tests revealed that the HC group (M = 1.55, SD = 1.36) had a significantly lower number of these long pauses compared to the mild dementia group (M = 3.85, SD = 2.23; P = .01) and the moderate dementia group (M = 6.45, SD = 3.17; P < .001). Additionally, the mild dementia group had a significantly lower number of these pauses relative to the moderate dementia group (P = .01).

Figure 3.

Number of pauses 2 seconds or greater (with standard errors). One-way ANCOVA significance value < .001, and post hoc pairwise significance is indicated as *P < .05, **P < .01. ANCOVA indicates analysis of covariance.

Mean Phrase Duration

Figure 4 shows the mean phrase duration in milliseconds for each group. A 1-way ANCOVA revealed a significant difference in the mean phrase duration between groups [F(2,56) = 7.06, P = .002, n ² _p = 0.20]. Post hoc tests showed that the mean phrase duration for the HC group (M = 1142.63, SD = 762.53) was significantly higher than the mild dementia group (M = 747.12, SD = 250.59; P = .03), and the moderate dementia group (M = 642.37, SD = 241.05; P = .002). However, there was no significant difference in mean phrase duration between the 2 dementia groups (P = .98).

Figure 4.

Mean phrase duration in milliseconds (with standard errors). One-way ANCOVA significance value = .002, and post hoc pairwise significance is indicated as *P < .05, **P < .01. ANCOVA indicates analysis of covariance.

Total Speech Duration

Figure 5 shows the total speech duration in seconds for each group (ie, originally calculated in milliseconds and converted to seconds), and a 1-way ANCOVA revealed a significant difference in this measure between groups [F(2,56) = 18.82, P < .001, n ² _p = 0.40]. Post hoc tests showed that the HC group (M = 36.16, SD = 25.33) had a significantly higher total speech duration compared to the mild dementia group (M = 26.57, SD = 10.88; P = .01) and the moderate dementia group (M = 21.72, SD = 9.38; P < .001). Additionally, the mild dementia group’s total speech duration was significantly higher than the moderate dementia group (P = .01).

Figure 5.

Total speech duration in seconds (with standard errors). One-way ANCOVA significance value < .001, and post hoc pairwise significance is indicated as *P < .05, **P < .01. ANCOVA indicates analysis of covariance.

Discussion

The current study aimed to identify the speech pausing profiles in people with dementia compared to HCs using the Calpy automated pause detection software. ³⁰ More specifically, the study aimed to examine the number and duration of pauses in speech for people with dementia compared to HCs and identify how such measures are impacted by disease severity. It was expected that the number and duration of pauses would increase with increasing dementia severity.

To determine pausing profiles, Calpy’s automated pause detection software was used to quantitatively examine the number and duration of both pauses and speech during a monologue task undertaken by all participants. The findings revealed that dementia was associated with a significant increase in mean and total pause duration. Notably, while there was no significant difference in the mean pause duration between HCs and those with mild dementia, the mean pause duration was still numerically higher for the mild dementia group compared to HCs, consistent with the expected pattern of results. These findings are consistent with previous research demonstrating the likelihood of pausing in speech as an indication of language (eg, word-finding difficulties) or cognitive (eg, information processing speeds) abnormalities as a function of disease severity.

Contrary to predictions, the current study found no group differences in the total number of pauses. While this finding contradicts some previous research, ^6,7 it is partially consistent with prior research by Pistono et al ¹⁰ who examined pauses within and between utterances separately. They defined between-utterance pauses as occurring at natural punctuation points in the speech, and within-utterance pauses occurring outside these punctuation junctures. Pistono et al ¹⁰ found that participants with AD did not produce more within-utterance pauses compared to controls; however, AD participants did demonstrate more between-utterance pauses than controls. In the current study, total pause count was not differentiated according to pause location (ie, within or between utterances). Accordingly, group differences in the number of pauses may be more a function of pause location based on utterance type (a distinction that was not made in the current study) rather than raw counts of the total number of pauses. In addition, it has been suggested by Singh et al ¹² that effort and planning are reflected more by long pauses than by frequent ones, an explanation determined by their findings that participant’s lower speech rates were due to longer pause durations not more frequent ones. The explanation by Singh et al ¹² corresponds with the current findings, demonstrating that pause durations increase as a function of increased cognitive impairment (as opposed to the frequency with which pauses occur).

Despite the lack of group differences with the total number of pauses made, there was a significant increase in the number of pauses 2 seconds or longer as a function of both the presence of dementia and dementia severity. While an appropriate “normal” conversational pause is thought to be approximately 1 second during conversations, ³⁴ it is well established that people with dementia often require additional processing time. ³⁵ Thus, an examination of pauses 2 seconds or longer were examined separately in the current study as these are generally considered indicative of conversational trouble for people with dementia. ³⁶ While it is acknowledged that the speech samples utilized in the current study were narrative rather than conversational speech, the findings are consistent with the notion that pauses exceeding 2 seconds increase with dementia severity.

The current study also found that the percentage of pause duration (relative to the total audio recording duration) was increased in dementia. The average time spent pausing during the picture description was 41% for HCs, 56% for people with mild dementia, and 68% for those with moderate dementia. As expected, this finding indicates that time spent in silent pauses is correlated to cognitive impairment severity and is consistent with previous research showing similar trends. ^6,33

The speech profile was also examined comparatively across the 3 groups. In conjunction with an upward trajectory in the pausing profile of participants as a function of disease progression, it was similarly expected that speech durations would decrease in dementia. Findings from the current study supported this conjecture for both the mean and total speech durations, with the exception of no significant difference in the mean speech duration between the 2 dementia groups. Despite this finding, the moderate dementia group did demonstrate a numerically lower mean speech duration than the mild dementia group, consistent with the expected pattern of results. However, it is also important to note that time spent pausing and speech time are inversely related, whereby as pausing time increases, speech time inevitably decreases. Thus, the observed reduction in speaking time as a function of increased cognitive impairment is not independent of time spent pausing.

While the current findings show support for the analysis of pausing as an indication of speech dysfluency using an automated approach, some limitations should be acknowledged. Given the preliminary nature of analyzing pausing in speech using Calpy, the current study utilized monologue data as opposed to conversational data. While monologue data provides a useful way of analyzing certain characteristics of speech, the results are not generalizable to conversations of spontaneous speech involving 2 or more individuals. In this way, the current study was unable to examine pausing in the context of the turn-taking dynamics that occur in dyadic conversations, and how pausing affects conversational flow. In order to accurately ascertain when pausing in speech is a marker of conversational trouble, the analysis of conversational data with Calpy is needed. Additionally, the current study neglected to qualitatively analyze the function of pausing in speech. While pausing may be used to signify conversational trouble during social interaction, not all pauses are reflective of conversational breakdown. Future research in this area would benefit from identifying and coding the pausing functions during a conversation in terms of topic familiarity, word-finding difficulties, sentence formulation difficulties, false-starts, a distraction from the task/theme, semantic or lexical decision-making, or the cognitive load of the conversation. Identification of pause functions would need to be human-coded initially, in order to inform future machine learning algorithms that could be incorporated into Calpy. Finally, it should be noted that the current study examined silent pauses exclusively, as opposed to examining both silent and filled pauses. While the unstated assumption of sounds such as uh and um is generally considered pauses that are filled with sound and not silence (ie, filled pauses), there are several conflicting views that lead to uncertainty around whether these are indeed filled pauses. Specifically, 3 separate views suggest: that these sounds are symptoms of certain problems in speaking; that they are nonlinguistic signals for dealing with certain problems in speaking; or that they are linguistic signals that are considered words of English. ³⁷ Thus, given the ambiguity surrounding the contention that filled pauses may not be considered an authentic pause, in conjunction with the preliminary nature of examining a novel pause detection program, the current study maintained a focus on the unanimous view of pausing in speech (ie, silent pauses). Nevertheless, it would still be beneficial for future research with Calpy to analyze filled pauses as a way of determining the function and utility of filled pauses in association with dementia.

The present findings have several implications for understanding the role of pausing in the speech of people with dementia. Primarily, these findings provide preliminary support for the utility of an automated approach to the detection of pausing in speech using Calpy pause detection software. Findings from the current study demonstrate that pausing in the speech of people with dementia does indeed increase as a result of disease progression and emphasizes the importance of examining pausing for the purposes of identifying conversational breakdown in future research. Furthermore, the methodology and findings from the current study verify the feasibility of the concept of Calpy as a system that performs according to the preestablished coding criteria with which it was designed (ie, detect pausing in speech). Prior research using automated software to detect pausing in speech have predominantly used systems such as Praat ^9,11,38 and Audacity. ^10,28 While these systems are useful support tools for identifying pausing in speech, they are not robust enough to compute all the features of conversational timing analysis to satisfy the objectives of many discourse analysts. As such, Calpy’s ability to process conversational information via the input of audio data is a significant advantage over alternative systems with regard to the systematic examination of pause and turn-timing dynamics in naturally occurring speech.