Part of Speech Production in Patients With Primary Progressive Aphasia: An Analysis Based on Natural Language Processing

Abstract

Background

Primary progressive aphasia (PPA) is a neurodegenerative disorder characterized by a progressive decline of language functions. Its symptoms are grouped into three PPA variants: nonfluent PPA, logopenic PPA, and semantic PPA. Grammatical deficiencies differ depending on the PPA variant.

Aims

This study aims to determine the differences between PPA variants with respect to part of speech (POS) production and to identify morphological markers that classify PPA variants using machine learning. By fulfilling these aims, the overarching goal is to provide objective measures that can facilitate clinical diagnosis, evaluation, and prognosis.

Method and Procedure

Connected speech productions from PPA patients produced in a picture description task were transcribed, and the POS class of each word was estimated using natural language processing, namely, POS tagging. We then implemented a twofold analysis: (a) linear regression to determine how patients with nonfluent PPA, semantic PPA, and logopenic PPA variants differ in their POS productions and (b) a supervised classification analysis based on POS using machine learning models (i.e., random forests, decision trees, and support vector machines) to subtype PPA variants and generate feature importance (FI).

Outcome and Results

Using an automated analysis of a short picture description task, this study showed that content versus function words can distinguish patients with nonfluent PPA, semantic PPA, and logopenic PPA variants. Verbs were less important as distinguishing features of patients with different PPA variants than earlier thought. Finally, the study showed that among the most important distinguishing features of PPA variants were elaborative speech elements, such as adjectives and adverbs.

Morphosyntactic deficits have been identified as one of the key symptoms of primary progressive aphasia (PPA; Mesulam et al., 2014, 2012; Mesulam & Weintraub, 2014), a neurodegenerative condition that results in a considerable deterioration of speech and language skills (Thompson, Lukic, et al., 2012; Thompson & Mack, 2014). PPA is characterized by substantial variability of symptoms as an effect of the degree of neurodegenerative decline, underlying pathology, and areas of brain damage (Mesulam, 2013; Thompson, Lukic, et al., 2012; Thompson & Mack, 2014). To understand the symptoms, recently established consensus criteria classify patients into three main PPA variants: the nonfluent PPA variant (nfvPPA), the logopenic PPA variant (lvPPA), and the semantic PPA variant (svPPA; Gorno-Tempini et al., 2011; Gorno-Tempini & Pressman, 2016).

Morphosyntactic production is key for language communication, and as it becomes impaired in PPA, it can provide objective markers for the classification of patients with PPA into variants, for clinical evaluation, prognosis of the condition, and intervention. Patients with nfvPPA are characterized by sparsity of function words and abnormal syntax due to peak atrophy at the posterior inferior frontal gyrus (Broca's area; Gorno-Tempini et al., 2011; Thompson et al., 1997, 2013; Thompson, Cho, et al., 2012). Patients with lvPPA are characterized by deficits in single-word retrieval and naming. Repetition of long words and phrases is also impaired, and their speech is characterized by phonological errors, such as paraphrases. Atrophy in individuals with lvPPA is related to brain volume loss in the left posterior temporal and inferior parietal regions (Gorno-Tempini et al., 2008, 2011; Gorno-Tempini & Pressman, 2016). Patients with svPPA are characterized by impaired object knowledge, impaired confrontation naming, and limited single-word comprehension. They are also characterized by impaired semantic memory of familiar objects, “empty speech” in verbal production, and surface dyslexia, as an effect of damage at the anterior temporal lobe (Gorno-Tempini et al., 2011; Gorno-Tempini & Pressman, 2016; Wilson et al., 2010).

Morphosyntactic production in PPA has been analyzed qualitatively (Kertesz et al., 1998; Mesulam, 2001) and quantitatively (Thompson, 2012; Thompson et al., 1997, 2014; Wilson, Brandt, et al., 2014; Wilson, DeMarco, et al., 2014; Wilson et al., 2010). Several early studies have shown that patients with nfvPPA produce proportionally fewer function (also known as grammatical) words than content words from patients with lvPPA and svPPA (Thompson, Cho, et al., 2012; Thompson & Mack, 2014; Thompson et al., 2013; Wilson et al., 2010). Function words are words with grammatical meaning and syntactic purpose, such as conjunctions and articles. In contrast, content words have semantic meaning, such as nouns and verbs. However, studies, such as Fraser et al. (2014), did not find a pronounced deficit in function words in individuals with nfvPPA in comparison to controls and patients with svPPA.

In addition to a specific deficit in content versus function words, patients with nfvPPA have been characterized by selective impairment in naming verbs relative to naming nouns (Thompson, Lukic, et al., 2012). The opposite of this pattern was found to exist in patients with svPPA, who are characterized by selective impairment in naming nouns relative to verbs (Cappa et al., 1998; Daniele et al., 1994; Hillis et al., 2004; Kertesz et al., 1998; Thompson, Lukic, et al., 2012). Overall, patients with PPA display substantial heterogeneity with respect to content word production, especially nouns and verbs. For example, Thompson, Cho, et al. (2012) did not find significant differences between patients with nfvPPA, patients with lvPPA, and healthy controls in noun-to-verb ratio. Only patients with svPPA displayed differences in noun-to-verb ratio from both patients with nfvPPA and controls (Thompson, Cho, et al., 2012).

Previous studies examined connected speech productions in PPA using picture description tasks (see Faroqi-Shah et al., 2020; Wilson et al., 2010). Connected speech reveals a wide array of naturally occurring morphosyntactic content and function word productions, such as the production of adjectives, adverbs, pronouns, and conjunctions, and can determine the distribution of content words and function words across PPA variants (Ash et al., 2006; Graham et al., 2004; Kavé et al., 2007; Meteyard & Patterson, 2009). Nevertheless, the manual analysis of connected speech productions is labor intensive, as it requires the identification of the part of speech (POS) for every word that appears in the transcriptions of connected speech. Our study offers a less labor-intensive automatic analysis of POS from connected speech productions and employs the analysis to classify patients into lvPPA, svPPA, and nfvPPA. Automatic methods were employed successfully in earlier research for Alzheimer's disease diagnosis, classification, and so forth (Bucks et al., 2000; Fraser et al., 2018; Guinn & Habash, 2012; Orimaye et al., 2014; Rentoumi et al., 2014; Themistocleous et al., 2018; Thomas et al., 2005) and in the automatic identification of patients with svPPA, nfvPPA, and healthy controls (Fraser et al., 2014).

The aims of this study were (a) to quantify the production of POS in patients with different PPA variants, (b) to determine whether and which POS markers can classify patients into PPA variants automatically, and (c) to provide a methodological approach that can be employed in clinical settings to enable clinicians to quantify morphosyntactic productions in patients with nfvPPA, lvPPA, and svPPA in a timely manner. Our overarching goal is to identify objective morphosyntactic measures that can be employed in clinical diagnosis, evaluation, and prognosis. We employed samples of connected speech from picture description tasks from all three PPA variants and automatically extracted measures of POS production. Subsequently, we employed computational morphosyntactic analysis, namely, morphosyntactic tagging, to identify the morphosyntactic categories automatically.

Method

Participants

Recordings of the Cookie Theft picture description task from 52 patients diagnosed with PPA were analyzed (Goodglass et al., 2001). Diagnosis of PPA was provided by neurologist(s) according to consensus criteria (Gorno-Tempini et al., 2011). Variant subtyping was based on magnetic resonance imaging results, clinical and neuropsychological examination, and speech and language evaluations following consensus criteria. Nineteen participants were subtyped as nfvPPA, 18 were subtyped as lvPPA, and 15 were subtyped as svPPA. Unclassified patients with PPA and apraxia of speech–only cases were excluded (see Table 1).

Table 1.

Demographic and neuropsychological data for each participant group.

Variable	nfvPPA	lvPPA	svPPA
Demographics
No. of participants	19	18	15
Gender	9F 10M	8F 10M	7F 8M
Age	69 (6)	68 (8)	67 (6)
Education	16 (1)	17 (2)	16 (2)
Severity
Language severity	2.77 (0.5)	1.39 (0.8)	2.27 (0.6)
Total severity	10 (6.5)	11 (6.3)	12 (5.5)
Language production
Boston Naming Test (30)	17 (5.6)	14 (6.5)	13 (11.1)
Category fluency (fruits, animals, vegetables)	22.9 (10.1)	20.8 (14.3)	7.8 (6.3)
Learning and memory
Rey Auditory Verbal Learning Test	33 (14)	19 (12)	18 (8)

Note. Recorded materials and speakers' demographic information. Shown are the number of participants, gender, age, education, language severity, and total severity. The numbers in parentheses indicate the standard deviation. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant; F = female; M = male.

Data were collected as part of a clinical trial conducted at Johns Hopkins University (NCT02606422; Tsapkini et al., 2018). Inclusion criteria comprised progressive speech and language deficits, native English speakers, minimum of high school education, and normal or corrected vision. Exclusion criteria comprised absence of any developmental or nondegenerative neurological disorder (e.g., stroke) and absence of primary deficits in other cognitive domains, such as memory. The present data are from baseline neuropsychological and neurolinguistic evaluations.

One-way analysis of variance tests showed that there were no significant differences between patients with nfvPPA, lvPPA, and svPPA for gender (F = 5.82, df = 2, p =.09), age (F = 0.354, df = 2, p = .705), education (F = 0.162, df = 2, p = .853), language severity (F = 0.154, df = 2, p =.86), and total severity (F = 1.162, df = 2, p = .33; see Table 1). Participants' age, education, and language severity were considered in data analysis (Riello et al., 2018). Language severity in Table 1 is a five-level rating subscale in the Clinical Dementia Rating–Modified Frontotemporal Lobar Degeneration Scale (Knopman et al., 2008). This scale was found effective in representing the progression of language deficits due to neurodegeneration, distinguishing patients based on a corresponding language severity outcome measure (Knopman et al., 2008). The Clinical Dementia Rating–Modified Frontotemporal Lobar Degeneration Scale showed that participants in this study were matched for overall language severity.

Materials

The Cookie Theft picture description task was part of the oral evaluation (Goodglass et al., 2001). This wordless picture shows three people performing different activities: two children trying to steal cookies from a cupboard as their mother is washing the dishes. A clinician presented the picture to the participant and prompted the participant following the standard Boston Diagnostic Aphasia Examination–Third Edition (Goodglass et al., 2001) instructions (“Tell me everything you see going on in this picture”). The patient is instructed to speak in complete sentences, making reference to the objects, people, and activities shown. No prompts or cues are provided. Speech productions were audio-recorded with a digital audio recorder. The audio recordings were all converted into 16000 Hz mono format and transcribed using speech-to-text technology implemented in an in-house software platform, Themis (Themistocleous et al., 2018; Themistocleous & Kokkinakis, 2018). Twenty percent of the transcriptions were selected randomly and evaluated by two independent evaluators. The automatic transcriptions were compared word by word with the transcripts of the two independent evaluators and marked as true (“T”) or false (“F”) for each word based on whether the word was correctly transcribed by the software (T) or whether the software provided a different transcription or no transcription (F). Using Cohen's kappa, the agreement of the automatic transcription with the first transcriber was .95 (p < .0001), and the agreement with the second transcriber was .96 (p < .0001); the agreement between the two transcribers was .97 (p < .0001). After establishing that transcriptions were close to human transcriptions, no further transcription modifications were employed as we were interested to see how well the transcriptions perform without manual modifications. Most fillers (e.g., “um” and “uh”) were transcribed (and analyzed) but were not considered word utterances, that is, they were not included in the total word count. Repetitions and false starts were transcribed in Roman alphabet; repetitions of words were included in the total word count. Neologisms were transcribed using standard orthography in Roman alphabet. Table 2 shows the mean number of words per variant. Individuals with nfvPPA, lvPPA, and svPPA produced different numbers of words; individuals with nfvPPA produced fewer words than patients with svPPA and lvPPA.

Table 2.

Number of words and mean number of words per participant in each variant in Cookie Theft picture description task.

Materials produced	nfvPPA	lvPPA	svPPA
Total no. of words	1,816	4,121	3,126
Mean words	121 (85)* , a	217 (165)	165 (97)

Note. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

^a Significantly different from the logopenic primary progressive aphasia variant.

^* p < .05.

Natural Language Processing and Morphosyntactic Measures

The transcripts were analyzed morphosyntactically using the NLTK Python library (Bird et al., 2009). Morphosyntactic analysis involved the following steps: (a) identification of words and (b) sentence structure analysis and POS identification in the text (for an overview of these methods, see Jurafsky & Martin, 2009). From the tagged corpus, both content words and function words were counted, and their corresponding proportions of the main POS categories (POS category/number of word count) were estimated. In addition to proportions, the ratios of all POS classes were calculated. Specifically, we calculated the following POS measures:

1. Content words, namely, the proportion of nouns, verbs, adjectives, and adverbs in the text; content words have a specific lexical meaning as they refer to objects, actions, and so forth.

2. Function words, namely, the proportion of conjunctions (e.g., and, or, but); prepositions (e.g., in, of); determiners and predeterminers (e.g., the, a/an, both); pronouns such as he, she, and it and wh– pronouns such as what, who, and whom; modal verbs (e.g., can, should, will); possessive ending ('s); adverbial particles (e.g., about, off, up); and conjunction (e.g., infinitival to, as in to do).

3. The corresponding ratio of nouns and verbs with other POS: noun/verb ratio, noun/adjective ratio, noun/adverb ratio, noun/pronoun ratio, noun/preposition ratio, noun/conjunction ratio, verb/adjective ratio, verb/adverb ratio, verb/pronoun ratio, verb/preposition ratio, verb/conjunction ratio, adjective/adverb ratio, and content word/function word ratio.

The analysis of ratios identifies the morphosyntactic selectivity in the speech, as ratios can determine specific preferences in the use of one POS category over another POS category. For example, the reduction of attributes assigned to nouns (e.g., “the red car” vs. “the car”) can be expressed by the noun/adjective ratio. The verb/adverb ratio provides information about the use of adverbs with respect to verbs in PPA productions “(she) reads” versus “(she) reads fast,” and the adverb/adjective ratio provides information about the selective use of adverbs with respect to adjectives. As the overall data size was small, POS ratios provide further information that can improve the automatic classification. The automatic POS tagging was compared to the manual tagging provided by three raters with background in linguistics. Cohen's kappa shows that there was substantial agreement between the automatic morphological analysis and the manual one provided from Rater 1 (κ = .983, p < .0001), Rater 2 (κ = .985, p < .0001), and Rater 3 (κ = .983, p < .0001).

Statistical Analysis

To determine whether patients with nfvPPA, svPPA, and lvPPA differ in the production of POS, generalized linear models were conducted with the PPA variants as predictors and the POS measures as response variables (see also Lien et al., 2017). Estimated marginal means (also known as least squares means) provided contrasts of the three PPA variants. The statistical analysis was performed in R (R Core Team, 2016), using lmer4 (Bates et al., 2015), lmerTest (Kuznetsova et al., 2017), and emmeans (Russell, 2018) packages in R.

Machine Learning Classification

To determine whether variants differed with respect to morphosyntactic measures and identify the morphosyntactic measures that enable variant classification, a machine learning classification task was conducted. Specifically, a binary one-against-all classification was employed, where the model distinguishes one variant from the other two variants: nfvPPA versus (lvPPA and svPPA), svPPA versus (nfvPPA and lvPPA), and lvPPA versus (nfvPPA and svPPA). Three machine learning models were employed: decision trees (DTs), random forests (RFs; Breiman, 2001; Breiman et al., 1984), and support vector machines (SVMs; Cortes & Vapnik, 1995). DTs split the data in a binary manner and provide a classification output represented as a dendrogram (tree diagram). The main disadvantage of DTs is that they often tend to create very large and complicated trees that cannot generalize to unknown data. RFs aim to address DTs' disadvantage by producing several tree models that are subsequently averaged. SVMs employ hyperspaces, that is, demarcating planes, to distinguish different groups, using both linear and nonlinear kernels and often perform better than DTs and RFs.

For the classification task, predictors were first individually scaled, such that they ranged between zero and one. To find the best performing models during evaluation, several models were tested using multiple cross-validations from 5-, 10- (…) to 35-folds (Hastie et al., 2009). We optimized the number of trees in RF, the size of the kernels in SVM models, and the initial random state for DTs. The optimization process showed that the output of the 35 cross-validation provided better results overall.

Machine learning models were trained and evaluated 35 times using a different set of 34 folds as a training set and a different single fold as an evaluation set. During the evaluation phase, the machine learning models provided a prediction of the PPA variant of each patient. After a trained model predicts the variant of a new and unknown (to the model) patient, the prediction of the model is compared to the actual PPA variant from the neurological evaluation to estimate whether the model made a correct prediction. From the comparison of the predicted variant and the actual variant, the following cases are estimated: (a) the true positive (TP) cases (i.e., the cases when patients belong to variant X [e.g., nfvPPA] and the model predicts correctly that they belong to that variant X), (b) the false negative (FN) cases (the cases that the model predicts that patients do not belong to variant X, when in actuality their variant is X), (c) the false positive (FP) cases (the cases that the model predicts that patients belong to variant X, when in actuality they do not belong to variant X), and (d) the true negative (TN) cases (i.e., when the model predicts that a patient does not belong to variant X correctly). These cases are employed to calculate the following evaluation metrics:

1. Precision (Precision = TP/[TP + FP]): The precision is a measure of the true positives divided by the sum of true positives and FPs; if there are many FPs, the precision is lower.

2. Recall (Recall = TP/[TP + FN]): The recall is a measure of the true positives divided by the sum of true positives and FNs, that is, a low recall indicates that there are many FNs.

3. Accuracy and balanced accuracy (Accuracy = [TP + TN]/[TP + TN + FP + FN]): The accuracy is a measure of the correct predictions made by the model divided by the total number of all cases. The balanced accuracy is the average of recall obtained on each class. The accuracy is a number from 0 to 100; an accuracy of 50 means that the model is correct half of the times, and a 100% accuracy means that the model is always correct.

4. The F1 score (F1 score = 2 × [(Precision × Recall)/(Precision + Recall)]): The F1 score measures the weighted average of Precision and Recall. It is a score from 0 to 100 that is similar to accuracy, which is adjusted if the design is unbalanced.

5. ROC/AUC curve: The receiver operating characteristic (ROC) and the area under the curve (AUC) are two evaluation measures that display the performance of a model. The ROC is a curve that is created by plotting the TP rate against the FP rate. The ROC/AUC measures the entire two-dimensional area under the entire ROC curve from (0, 0) to (1, 1). An optimal model has an ROC/AUC closer to 1, whereas a bad model whose predictions are 100% wrong has an ROC/AUC of 0 (see also Baldi et al., 2000; Chicco, 2017, for a discussion on the selection of evaluation metrics in machine learning).

Results

First, we present the overall distribution of morphosyntactic words into content words and function words. Second, we present the distribution of POS features in PPA variants and the results of the regression analysis. Third, we present the machine learning models mentioned above (RF, SVM, and DT) that classify patients into PPA variants using these features.

Distribution of Content Words and Function Words

Content Versus Function Word Production

In nfvPPA, the proportion of content words is higher than the proportion of function words (see Figures 1A and 1B and Table 3). Patients with nfvPPA also produce a significantly higher content word/function word ratio compared to patients with svPPA and lvPPA, β = 1.720, t(47) = 13.97, p < .0001 (see Figure 2C and Table 3). Estimated marginal means (also known as least squares means) for pairwise differences for the content word/function word ratio show significant differences for nfvPPA and lvPPA (p < .05), and nfvPPA and svPPA (p < .05), but there is no significant contrast between lvPPA and svPPA. These findings demonstrate that content word/function word ratio distinguishes patients with nfvPPA from svPPA and lvPPA.

Figure 1.

Violin plots with the proportion of content (A), function words (B), and the content/function word ratio (C). The shape of the plots indicates the probability density of the data. Dots indicate the means and error bars indicate 95% confidence intervals. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Table 3.

Regression models showing the statistical effects of primary progressive aphasia (PPA) variants on the distribution of part of speech (POS).

POS	PPA	Estimate	SE	t	Pr(>|t|)	F test
Content word/function word ratio	nfvPPA	1.72	0.123	13.97	< .0001***	F(2, 47) = 4.9, p < .01, R 2 = 17%
	lvPPA	−0.466	0.174	−2.67	.0103*
	svPPA	−0.502	0.186	−2.69	.0098**
Content word	nfvPPA	0.6401	0.0234	27.36	.0001***	F(2, 49) = 1.47, p < .24, R 2 = 2%
	lvPPA	−0.0571	0.0335	−1.7	.095
	svPPA	−0.0342	0.0352	−0.97	.336
Function word	nfvPPA	0.3800	0.0261	14.54	.0001***	F(2, 49) = 3.74, p < .05, R 2 = 13%
	lvPPA	0.0996	0.0375	2.66	.011*
	svPPA	0.0706	0.0393	1.79	.079
Noun	nfvPPA	0.3606	0.0284	12.71	.0001***	F(2, 49) = 11.4, p < .001, R 2 = 32%
	lvPPA	−0.1468	0.0407	−3.61	.0007***
	svPPA	−0.1901	0.0427	−4.45	.0001***
Adverb	nfvPPA	0.0337	0.0113	2.98	.0044**	F(2, 49) = 6.99, p < .01, R 2 = 22%
	lvPPA	0.0315	0.0162	1.94	.0577
	svPPA	0.0636	0.0170	3.73	.0005***
Adjective	nfvPPA	0.0215	0.0142	1.51	.138	F(2, 49) = 3.16, p < .05, R 2 = 11%
	lvPPA	0.0232	0.0204	1.13	.263
	svPPA	0.0539	0.0214	2.51	.015*
Pronoun	nfvPPA	0.2243	0.0172	13.04	.0001***	F(2, 49) = 1.45, p = .25, R 2 = 6%
	lvPPA	0.0351	0.0247	1.42	.16
	svPPA	0.0385	0.0259	1.49	.14
Verb	nfvPPA	0.2243	0.0172	13.04	.0001***	F(2, 49) = 1.45, p = .24, R 2 = 6%
	lvPPA	0.0351	0.0247	1.42	.16
	svPPA	0.0385	0.0259	1.49	.14
Determiner	nfvPPA	0.1172	0.0130	8.99	.0001***	F(2, 49) = 2.66, p < .01, R 2 = 10%
	lvPPA	0.0344	0.0187	1.84	.072
	svPPA	−0.0076	0.0196	−0.38	.702
Modal verb	nfvPPA	0.0017	0.0025	0.68	.501	F(2, 49) = 5.6, p < .001, R 2 = 19%
	lvPPA	0.0106	0.0035	3.02	.004**
	svPPA	0.0098	0.0037	2.67	.01*
Preposition	nfvPPA	0.1049	0.0112	9.41	.0001***	F(2, 49) = 1.53, p = .2, R 2 = 6%
	lvPPA	0.0279	0.0160	1.74	.088
	svPPA	0.0119	0.0168	0.71	.481
Conjunction	nfvPPA	0.0521	0.0078	6.69	.0001***	F(2, 49) = 0.78, p < .46, R 2 = 3%
	lvPPA	0.0063	0.0112	0.56	.58
	svPPA	0.0147	0.0117	1.25	.22
Particle	nfvPPA	0.0144	0.0037	3.88	.0003***	F(2, 49) = 0.54, p = .58, R 2 = 2%
	lvPPA	−0.0039	0.0053	−0.74	.4657
	svPPA	0.0017	0.0056	0.31	.76
Possessives	nfvPPA	0.0040	0.0015	2.67	.01*	F(2, 49) = 0.94, p = .39, R 2 = 3%
	lvPPA	−0.0006	0.0021	−0.27	.79
	svPPA	−0.0029	0.0022	−1.32	.19

Note. The intercept of the model corresponds to the nonfluent PPA variant (nfvPPA). It also provides the standard error (SE), the t value, and the p value, Pr(>|t|). The last column provides the overall analysis of variance score of the model and its corresponding R ². lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

^* p < .05,

^** p < .01,

^*** p < .001.

Figure 2.

Violin plots with the distribution of nouns normalized to the total number of words produced by patients with nfvPPA, lvPPA, and svPPA. The shape of the plots indicates the probability density of the data. Dots indicate the means and error bars indicate 95% confidence intervals. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Noun Production

Patients with nfvPPA produce proportionally more nouns than patients with lvPPA and svPPA (see Figure 2). Consequently, patients with PPA differ significantly in the proportion of nouns, F(2, 47) = 15.7, p < .01. Both lvPPA (t = −4.06, p < .01) and svPPA (t = −5.30, p < .01) are statistically different from nfvPPA. However, pairwise comparisons reveal that patients with lvPPA and svPPA do not differ in the proportion of nouns (nonsignificant).

Verb Production

Patients with PPA do not significantly differ in verb production (nfvPPA, M = 0.24, SD = 0.06; lvPPA, M = 0.26, SD = 0.6; svPPA, M = 0.28, SD = 0.05). The comparison of verb production in patients with nfvPPA and svPPA shows that patients with nfvPPA produce fewer verbs (t = 2.2, p = .03), but the overall model was not significant (see Figure 3).

Figure 3.

Violin plots with the distribution of verbs normalized to the total number of words produced by patients with nfvPPA, lvPPA, and svPPA. The shape of the plots indicates the probability density of the data. Dots indicate the means and error bars indicate 95% confidence intervals. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Adjective and Adverb Production

Patients with nfvPPA produced fewer adjectives and adverbs than patients with svPPA and lvPPA (see Figures 4A and 4B). This resulted in an overall statistically significant effect of the variant on both adjectives and adverbs (see Figures 4A and 4B and Table 3).

Figure 4.

Violin plots with the distribution of adjectives (A) and the distribution of adverbs (B) normalized to the total number of words produced by patients with nfvPPA, lvPPA, and svPPA. The shape of the plots indicates the probability density of the data. Dots indicate the means and error bars indicate 95% confidence intervals. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Modal Verb, Conjunction, Determiner, and Preposition Production

There were fewer productions with modal verbs in patients with nfvPPA (M = 0.002, SD = 0.003); patients with lvPPA (M = 0.012, SD = 0.014) and patients with svPPA (M = 0.012, SD = 0.013) produced approximately the same number of modal verbs. Patients with lvPPA (β = 0.011, t = 2.94, p < .01) and patients with svPPA (β = 0.01056, t = 2.76, p < .01) produced significantly more modal verbs than patients with nfvPPA. Although patients with nfvPPA produce fewer conjunctions, determiners, particles, and prepositions from patients with svPPA and lvPPA, the statistical models for these morphonsyntactic categories were not significant.

The Appendix shows the noun/verb ratio, noun/adjective ratio, noun/adverb ratio, noun/pronoun ratio, noun/preposition ratio, and noun/conjunction ratio. Figure 5 shows the results for the noun/adverb ratio (Figure 5A) and the verb/adverb ratio (Figure 5B). These ratios distinguish patients with nfvPPA, svPPA, and lvPPA significantly manifesting different structural effects in the patterns of POS in PPA individuals.

Figure 5.

Violin plots showing the noun/adverb ratio (A) and the verb/adverb ratio (B). The shape of the plots indicates the probability density of the data. Superimposed are error bars showing 95% confidence intervals from the mean (black circle in the middle). PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Machine Learning Classification

To evaluate the contribution of POS markers for identifying patients with different PPA variants and determine POS features that enable an automatic classification of patients with nfvPPA, lvPPA, and svPPA, we employed three machine learning models, RFs, SVMs, and DTs, in a supervised one-against-all classification task. The accuracy, balanced accuracy, F1 score, precision, recall, and ROC/AUC produced in supervised one-against-all classification from RF, SVM, and DT are shown in Table 4; the ROC/AUC plots are shown in Figure 6. Patients with nfvPPA and svPPA are identified better with 79% and 77% classification accuracy, respectively, than patients with lvPPA who are identified with 64% classification accuracy. Although, SVM achieved 84% classification accuracy, it was accompanied with a lower ROC/AUC. As such, we are presenting the feature hierarchy results from the RF model, which indicated both the highest ROC/AUC and accuracy. It is important to note that feature hierarchies depend on the respective machine learning model. Therefore, different models might result into different featural rankings. Regardless of rankings, they demonstrate the properties that influence the model, which can be indicative of which morphological features influenced the classification outcome of the machine learning model.

Table 4.

Mean and standard deviation (in square brackets) of model accuracy, balanced accuracy, F1 score, precision, recall, and area under the curve (AUC) produced in supervised one-against-all classification from random forests (RF), support vector machines (SVM), and decision trees (DT).

PPA		RF	SVM	DT
nfvPPA	*Accuracy	79% [36%]	84% [31%]	79% [36%]
	Balanced Accuracy	71% [40%]	71% [40%]	68% [41%]
	F1 score	71% [41%]	70% [41%]	68% [41%]
	Precision	71% [40%]	71% [42%]	71% [42%]
	Recall	71% [40%]	70% [40%]	68% [41%]
	AUC	78% [25%]	71% [36%]	78% [25%]
lvPPA	*Accuracy	64% [42%]	57% [43%]	57% [42%]
	Balanced Accuracy	64% [42%]	57% [43%]	57% [42%]
	F1 score	63% [43%]	55% [44%]	58% [42%]
	Precision	64% [45%]	56% [46%]	63% [46%]
	Recall	64% [42%]	57% [43%]	57% [41%]
	AUC	64% [23%]	57% [17%]	43% [32%]
svPPA	*Accuracy	77% [40%]	69% [43%]	73% [38%]
	Balanced Accuracy	77% [40%]	69% [43%]	73% [38%]
	F1 score	76% [41%]	70% [43%]	72% [39%]
	Precision	76% [42%]	72% [44%]	74% [41%]
	Recall	77% [40%]	69% [43%]	73% [38%]
	AUC	87% [22%]	81% [35%]	75% [25%]

Note. Shown with boldface are the models that provided the highest classification accuracy. The star symbol (*) indicates nonweighted measures; weighted measures consider the unbalance in a design. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

Figure 6.

Mean receiver operating characteristic (ROC) curves from the classification models of nvfPPA, lvPPA, and svPPA. We run three separate supervised one-against-all classification models using rain forests (RFs), support vector machines (SVMs), and decision trees (DTs). Shown in the figure is the ROC with the mean area under the curve (AUC) for the DT, SVM, RF models and the corresponding SD. The best classification is the one whose ROC approaches the right upper corner of the plot and that has the highest AUC; here, the best mean AUC resulted from RFs for nfvPPA, RFs for lvPPA, and SVM models for svPPA. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

The POS features with the greatest impact on the machine learning classification model for the identification of each PPA variant are shown in Table 5. The relative importance of POS as a measure of PPA variance is calculated from the relative FI ranking, which is employed as a decision node in trees generated by the RF algorithm. FI indicates that highly ranked features have a greater contribution to the final classification output; this denotes the relative importance of a POS measure. The FIs shown in Table 5 are standardized and add up to one. The highest six rankings for each variant are shown.

Table 5.

Feature importance (FI) of the classification models for patients with nfvPPA, lvPPA, and svPPA from random forests.

Ranking	nfvPPA		lvPPA		svPPA
	Features	FI	Features	FI	Features	FI
1	Noun/adverb ratio	0.11	Determiners	0.10	Nouns	0.12
2	Nouns	0.09	Adverb/preposition ratio	0.08	Noun/adverb ratio	0.09
3	Adjectives	0.07	Verb/pronoun ratio	0.07	Adverbs	0.07
4	Adverbs	0.07	Content/function word ratio	0.07	Adverb/preposition ratio	0.05
5	Noun/adjective ratio	0.06	Adjectives	0.06	Noun/verb ratio	0.05
6	Content/function word ratio	0.04	Noun/verb ratio	0.06	Noun/adjective ratio	0.05

Note. The FI ranking is a standardized measure with positive values that sum to 1; the table shows the first six highest ranked features. PPA = primary progressive aphasia; nfvPPA = nonfluent PPA variant; lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

This output demonstrates the morphosyntactic selectivity in patients with nfvPPA, lvPPA, and svPPA variants. The noun/adverb ratio; the proportion of nouns, adjectives, and adverbs; and the noun/adjective ratio distinguish patients with nfvPPA from patients with the other two PPA variants. The high predominance of content words, especially nouns in the productions of patients with nfvPPA, demonstrates the overall impact of content words for the classification of this variant. The machine learning model employed different POS measures for the identification of lvPPA. Specifically, the proportion of determiners, the adverb/preposition ratio, the verb/pronoun ratio, the content word/function word ratio, and the proportion of adjectives contribute to the classification of patients with lvPPA. Finally, the svPPA classification model employed the proportions of nouns and adverbs, the noun/adverb ratio, the adverb/preposition ratio, and the noun/verb ratio as predictors. This FI suggests that the proportion of nouns, the noun/adverb ratio, and the noun/verb ratio have a high impact for the classification of patients with svPPA.

Discussion

Grammatical production in PPA (Mesulam et al., 2014, 2012; Mesulam & Weintraub, 2014) varies considerably between patients with different variants. Despite the long-standing interest in PPA morphosyntactic production, research has not led to conclusive morphosyntactic patterns for the systematic classification of PPA variants. This article aimed to address the need to determine morphosyntactic production in patients with PPA using POS measures and highlighted their role for subtyping PPA using machine learning. This was accomplished by charting a procedure for the quantitative assessment of connected speech and the comparative evaluation of morphosyntax in all three PPA variants using natural language processing. Our findings show substantial heterogeneity of POS production in patients with PPA. Grammatical selectivity is manifested in the distribution of content words (particularly nouns) and function words in distinguishing patients with different variants. In short, automatic morphosyntactic tagging of connected speech samples from an isolated picture task corroborated earlier findings that were elicited using a time-consuming battery of tasks. It also provided novel results that identify the effects of PPA on the production of adjectives and adverbs and highlights differences in verb production in patients with PPA compared to previous studies.

POS Production in nfvPPA

Patients with nfvPPA produced more content words than function words. Specifically, nouns were more frequent in patients with nfvPPA than in patients with lvPPA and svPPA. This finding corroborates earlier studies that suggest naming is not as impaired in nfvPPA as in svPPA (i.e., making reference to objects, people, etc., in a picture description task; e.g., Bastiaanse & Jonkers, 2012; Gorno-Tempini et al., 2011). By contrast, the production of function words was significantly diminished in patients with nfvPPA compared to patients with lvPPA and svPPA. Our findings showed that patients with nfvPPA produced fewer adjectives and adverbs (recognized as content words) than patients with svPPA and lvPPA. Patients with nfvPPA are known to have impaired production of function words, so this finding suggests that they have impairment with certain content words as well, namely, adverbs and adjectives. However, the reduced use of adverbs by patients with nfvPPA compared to patients with svPPA does not imply that the language of patients with nfvPPA is grammatically less dense or less meaningful than that of patients with svPPA. In fact, speech production in individuals with svPPA is often indefinite and recognized as “empty.” The linear model showed that the ratios of nouns with other POS (i.e., noun/adjective, noun/preposition, noun/adverb, noun/verb) can reliably and significantly distinguish patients with nfvPPA from other variants. The contribution of these ratios to the identification of patients with PPA was further supported by the subsequent classification task. This finding highlights the significant predominance of nouns and the interplay of nouns with adjectives, prepositions, adverbs, and verbs as a characteristic to the speech of patients with nfvPPA.

Given the central role of verbs in sentence structure, it was surprising that patients with nfvPPA did not differ in the production of verbs compared to patients with svPPA and lvPPA. This suggests that verb production might not be a good predictor for PPA variant classification (Fraser et al., 2014; Kim & Thompson, 2000). However, when verbs are taken together with nouns, the noun/verb ratio is higher in patients with nfvPPA than in patients with lvPPA and svPPA. This ratio can distinguish patients with all three PPA variants (for a different finding, see Thompson, Cho, et al., 2012).

Modal verbs (e.g., can, should) were analyzed separately from matrix verbs, as modal verbs are grammatical words that constitute a closed class and are employed in speech to convey deontic modalities (e.g., the expression of wishes, commands; Bybee & Fleischman, 1995; Bybee et al., 1994). We found a different distribution of modal verbs from matrix verbs in PPA. Patients with nfvPPA produced significantly fewer modal verbs than patients with lvPPA and svPPA. This finding on modal verbs can be interpreted as a manifestation of the agrammatism that characterizes patients with nfvPPA (Knibb et al., 2009; Thompson & Bastiaanse, 2012; Thompson, Cho, et al., 2012; Wilson et al., 2010).

POS Production in svPPA

SvPPA is characterized by a combination of word comprehension deficits, fluent aphasia, and a particularly severe anomia, whereas the acoustic or phonological characteristics are viewed as unimpaired (Wilson et al., 2010). We found that patients with svPPA produced fewer nouns than patients with nfvPPA and lvPPA, which aligns with our expectations: Patients with svPPA are characterized by impaired naming abilities (Gorno-Tempini et al., 2011). Also, they frequently substituted nouns with pronouns, which was manifested in the noun/pronoun ratio (e.g., I see him). This can be attributed to accommodation or compensation strategies for noun (e.g., I see a child) production difficulties (Wilson et al., 2010). As such, the noun/pronoun ratio can be employed to distinguish patients with svPPA from patients with nfvPPA.

Verb production did not constitute a marker that could differentiate patients with svPPA from patients with lvPPA and svPPA. This finding replicates earlier studies on verb production in patients with svPPA (Fraser et al., 2014; Thompson, Cho, et al., 2012). Patients with svPPA produced more adverbs than patients with nfvPPA and lvPPA. Adverbs modify the sentence and/or the verb phrase by providing information about manner, place, time, frequency, degree, and so forth. Adverbs were found to be important in distinguishing individuals with svPPA from healthy controls, but not from nfvPPA (Fraser et al., 2014). Unlike patients with nfvPPA, patients with svPPA produced a higher number of function words than patients with other PPA variants (Meteyard & Patterson, 2009).

POS Production in lvPPA

Patients with lvPPA were found to differ significantly from nfvPPA with regard to function words, modal verbs, and nouns. Patients with lvPPA produced more modals and function words compared to patients with nfvPPA. Interestingly, patients with lvPPA produced morphosyntactic constituents that overlapped with both patients with nfvPPA patients and patients with svPPA (i.e., their proportions were usually in-between those with svPPA and nfvPPA). Therefore, it is not surprising that the classification accuracy of patients with lvPPA was lower—just 64% classification accuracy—than in patients with nfvPPA (71% classification accuracy) and svPPA (77% classification accuracy).

Diagnostic Utility of the Automatic Analysis of POS

The automatic analysis of POS employed in this study and in previous studies (e.g., Fraser et al., 2014) highlights the contribution of computational methods for analyzing language production in patients with PPA and bridging the gap between computational linguistic analysis and manual evaluation of language productions in PPA. In clinical settings, the automatic analysis of POS in the form of a computer application enables physicians, neuropsychologists, and speech-language pathologists to elicit POS measures of connected speech, discourse, and so forth. At the same time, it provides objective markers that estimate patients' grammatical competence. Clinicians can employ these objective markers for diagnosisng PPA, subtyping variants, and estimating linguistic changes due to speech-language therapy and/or the natural progression of degenerative condition. It also enables clinicians to measure additional POS categories that can inform individualized therapeutic programs (e.g., targeting increased use of specific POS productions). Finally, the methodological approach of this study as a diagnostic tool can be employed in other conditions affecting language, such as aphasia due to stroke (Fyndanis & Themistocleous, 2019).

Limitations and Future Directions

A major limitation in determining the difference between PPA variants in POS production is the type of task. Grammatical productions, specifically the distribution of POS, verb tense, and so forth, depend on the nature of the task. A common criticism of picture description tasks (e.g., Cookie Theft) that instruct patients to discuss everything they see is that patients are inclined to elicit labeling rather than narration and verbs produced are mostly in the present tense (past tense and future tense are relatively uncommon). Storytelling and discourse settings (e.g., free conversation) can facilitate the production of POS and also differ in their POS distributions as shown by several studies of speech genres (Bhatia, 1993; Swales, 1990). Future research is important to employ computational analysis of morphosyntactic production in different types of discourse settings.

With regard to the aim of classifying PPA variants based on morphological (POS) markers, this study is limited by the small corpus used. Cookie Theft description of 52 participants with PPA may be a respectable sample for PPA given the rarity of the syndrome, but it is a small sample for totally unbiased unsupervised machine learning methods. A larger corpus would enable us to characterize POS productions with greater precision and reduce variability. A related methodological limitation is that the machine learning models do not classify all PPA variants at the same time but classify one variant against the other two. The decision for one against all classification was necessitated by the limited number of data.

Future work should employ unsupervised machine learning models that offer simultaneous classification of all PPA variants against each other. Outputs should be used in different types of discourse settings, and multifactorial designs should be implemented with predictors that include speech acoustics, grammar, and so forth, to capture both speech and overall language deficits.

Appendix

Regression Models Showing the Statistical Effects of Primary Progressive Aphasia (PPA) Variants on the Distribution of Part of Speech

POS ratio	PPA	Estimate	SE	t	Pr(>|t|)	F test
Noun/verb ratio	nfvPPA	1.583	0.172	9.21	< .0001***	F(2, 47) = 7.3, p < .01, R 2 = 24%
	lvPPA	−0.678	0.243	−2.79	.0076**
	svPPA	−0.939	0.26	−3.61	.0007***
Noun/adjective ratio	nfvPPA	11.87	1.36	8.7	< .0001***	F(2, 43) = 7.58, p < .01, R 2 = 26%
	lvPPA	−5.5	1.79	−3.07	.0037**
	svPPA	−6.89	1.86	−3.7	.0006***
Noun/adverb ratio	nfvPPA	12.69	1.9	6.67	< .0001***	F(2, 46) = 8.09, p < .001, R 2 = 26%
	lvPPA	−8.16	2.69	−3.03	.0040**
	svPPA	−10.51	2.78	−3.78	.0005***
Noun/pronoun ratio	nfvPPA	5.414	0.952	5.69	< .0001***	F(2, 44) = 4.37, p < .05, R 2 = 17%
	lvPPA	−2.544	1.327	−1.92	.0617
	svPPA	−4.043	1.394	−2.9	.0058**
Noun/preposition ratio	nfvPPA	3.29	0.384	8.56	< .0001***	F(2, 46) = 5.36, p < .01, R 2 = 19%
	lvPPA	−1.442	0.536	−2.69	.0099**
	svPPA	−1.678	0.572	−2.93	.0052**
Noun/conjunction ratio	nfvPPA	7.302	0.941	7.76	< .0001***	F(2, 45) = 6.01, p < .01, R 2 = 21%
	lvPPA	−2.708	1.293	−2.09	.0419*
	svPPA	−4.735	1.377	−3.44	.0013**
Verb/adjective ratio	nfvPPA	9.223	1.585	5.82	< .0001***	F(2, 43) = 0.142, p = .86, R 2 = 0.6%
	lvPPA	−1.021	2.079	−0.49	.6300
	svPPA	−0.969	2.165	−0.45	.6600
Verb/adverb ratio	nfvPPA	9.25	1.25	7.4	.0000***	F(2, 46) = 5.34, p < .001, R 2 = 19%
	lvPPA	−4.29	1.77	−2.43	.0191*
	svPPA	−5.64	1.83	−3.09	.0034**
Verb/pronoun ratio	nfvPPA	3.779	0.699	5.41	< .0001***	F(2, 44) = 1.19, p = .3, R 2 = 5%
	lvPPA	−0.529	0.973	−0.54	.5900
	svPPA	−1.558	1.023	−1.52	.1300
Verb/preposition ratio	nfvPPA	2.254	0.256	8.8	< .0001***	F(2, 46) = 0.15, p = .86, R 2 = 0.6%
	lvPPA	−0.016	0.357	−0.04	.9600
	svPPA	0.174	0.381	0.46	.6500
Verb/conjunction ratio	nfvPPA	5.071	0.642	7.9	.0000***	F(2, 45) = 0.31, p = = .73, R 2 = 1%
	lvPPA	0.234	0.882	0.27	.7900
	svPPA	−0.487	0.939	−0.52	.6100
Adjective/adverb ratio	nfvPPA	0.7450	0.1506	4.95	0.000011***	F(2, 46) = 0.23, p = .79, R 2 = 1%
	lvPPA	0.0748	0.2130	0.35	0.73
	svPPA	−0.0753	0.2200	−0.34	0.73
Adjective/pronoun ratio	nfvPPA	0.3438	0.1186	2.90	0.0058**	F(2, 47) = 0.9, p = .41, R 2 = 3%
	lvPPA	0.2086	0.1653	1.26	0.2137
	svPPA	0.0365	0.1737	0.21	0.8346
Adjective/preposition ratio	nfvPPA	0.2478	0.0948	2.62	0.012*	F(2, 46) = 1.74, p = .19, R 2 = 7%
	lvPPA	0.1579	0.1321	1.20	0.238
	svPPA	0.2579	0.1410	1.83	0.074
Adjective/conjunction ratio	nfvPPA	0.626	0.266	2.35	0.023*	F(2, 45) = 0.87, p = .4, R 2 = 3%
	lvPPA	0.474	0.366	1.30	0.202
	svPPA	0.165	0.390	0.42	0.673
Adverb/pronoun ratio	nfvPPA	0.430	0.244	1.77	0.084	F(2, 44) = 0.93, p = .4, R 2 = 4%
	lvPPA	0.458	0.339	1.35	0.184
	svPPA	0.185	0.357	0.52	0.607
Adverb/preposition ratio	nfvPPA	0.309	0.131	2.36	0.022*	F(2, 46) = 2.83, p = .07, R 2 = 10%
	lvPPA	0.350	0.182	1.92	0.061
	svPPA	0.419	0.194	2.16	0.036*
Adverb/preposition ratio	nfvPPA	0.736	0.180	4.09	0.00018***	F(2, 45) = 2.88, p = .07, R 2 = 11%
	lvPPA	0.571	0.248	2.31	0.02570
	svPPA	0.452	0.264	1.71	0.09348

Note. The intercept of the model corresponds to the nonfluent PPA variant (nfvPPA). It also provides the standard error (SE), the t value, and the p value, Pr(>|t|). The last column provides the overall analysis of variance score of the model and its corresponding R ². lvPPA = logopenic PPA variant; svPPA = semantic PPA variant.

^* p < .05.

^** p < .01.

^*** p < .001.

Funding Statement

We are grateful to our participants and referring physicians for their interest in our study. We thank Olivia Hermann and Bronte Ficek for their assistance in our study. This work was supported through grants from the National Institutes of Health (Grant NIH/NIDCD R01 DC014475) and the Science of Learning Institute at Johns Hopkins University to K. T.