Neural bases of elements of syntax during speech production in patients with aphasia

Abstract

The ability to string together words into a structured arrangement capable of conveying nuanced information is key to speech production. The assessment of the neural bases for structuring sentences has been challenged by the need of experts to delineate the aberrant morphosyntactic structures in aphasic speech. Most studies have relied on focused tasks with limited ecological validity. We characterized syntactic complexity during connected speech produced by patients with chronic post-stroke aphasia. We automated this process by employing Natural Language Processing (NLP). We conducted voxel-based and connectome-based lesion-symptom mapping to identify brain regions crucially associated with sentence production and syntactic complexity. Posterior-inferior aspects of left frontal and parietal lobes, as well as white matter tracts connecting these areas, were essential for syntactic complexity, particularly the posterior inferior frontal gyrus. These findings suggest that sentence structuring during word production depends on the integrity of Broca’s area and the dorsal stream of language processing.

1. Introduction

While much information can be conveyed through isolated words, more complex communication requires the hierarchical organization of linguistic elements within a structure that is equally understood by the speaker and the listener. This, broadly defined as syntax, is the process by which communicative elements are organized into an arrangement that provides order, context, sequence, and often times, a progression of ideas within communication. Thus, by entangling words with objective meaning (e.g., “professor”) with words with practical meaning (e.g., “in”), the speaker can organize thoughts into a symbolic sequence that conveys complex and nuanced information.

In spite of its tremendous importance in communication, it is relatively difficult to study sentence structuring when speech is severely impaired, for example, after aphasia resulting from a dominant hemisphere stroke. In these cases, the assessment of impairment is performed through tasks such as confrontation naming or sentence completion Even though such tasks may reveal important information on language impairment, they do not fully capture the complexity of day-to-day communication, making these tasks less ecologically meaningful.

Constrained tasks may not always be able to detect subtle or even real-life language performance deficits (Dell, Schwartz, Martin, Saffran, & Gagnon, 1997; Malyutina, Richardson, & den Ouden, 2016), whereas tests capable of eliciting connected speech (as opposed to isolated words), such as picture description tasks have the potential for a more detailed analysis of sentence structuring (Boschi et al., 2017). Commonly used instruments include the Western Aphasia Battery, the Philadelphia Naming Test, and storytelling tasks (A. Kertesz, 2007; Andrew Kertesz, 2020; Roach, Schwartz, Martin, Grewal, & Brecher, 1996; Saffran, Berndt, & Schwartz, 1989). While picture naming tasks are not without limitation, particularly the potential for reliance on the present tense, they tend to elicit complete sentences.

At the moment, delineation of morphosyntactic markers of speech is regularly done by hand by trained raters, which is arduous, time-consuming, and contingent on good reliability across raters. Recently, additional programs have been developed to automate speech analysis and validated against trained raters, such as computerized language analysis (CLAN) and the computerized version of the Northwestern narrative Language Assessment (C-NNLA) (Fromm et al., 2020; Fromm et al., 2021). CLAN (C-QPA) includes measures of frequency of parts of speech, measures of complete sentences, as well as several features of verbs (MacWhinney, 2000). Additionally, C-NNLA can extract many essential aspects of language, such as frequency of comprehensive parts of speech, inflection, syntax, and numerous error types (MacWhinney, 2000). These are both exciting tools for automated language analysis that overlap with many features extracted with NLP. Noteworthy, both previously mentioned studies on CLAN and C-NNLA examined storytelling (specifically the Cinderella story), whereas the current study uses different tasks and includes measures of complexity of speech in addition to frequency.

The development of automated analysis techniques is crucial because it can greatly impact the understanding of how connected discourse can be impaired in aphasia, and more importantly, what neural mechanisms support sentence structure that are necessary for communication and lost in individuals with aphasia. In this study, we applied an alternative promising computational tool entitled Natural Language Processing (NLP) in conjunction with imaging analyses (Hirschberg & Manning, 2015) to quantify syntactic complexity in individuals with aphasia.

NLP automatically parses each sentence, identifying the syntactic role that each word plays in its grammatical context. It also allows for lexical analysis, as it can identify different classes of words and, using complementary databases, compute different features such as familiarity, age of acquisition, imageability, and so forth. NLP is at the intersection of linguistics and computer science. It is a technology capable of applying advanced computational algorithms to human language and has the potential to automate the structural analysis of long samples of human discourse. NLP has been used in research laboratories as a tool to analyze grammatical errors produced by language learners (Leacock, Chodorow, Gamon, & Tetrault, 2010) and to identify individuals with mild cognitive impairment (Roark, Mitchell, Hosom, Hollingshead, & Kaye, 2011). Automated classification has also been applied to those with language impairments, such as primary progressive aphasia. For example, Fraser et al (2014) successfully used automated classification to categorize subtypes of primary progressive aphasia based on lexical and syntactic profiles. Among patients with post-stroke aphasia in particular, NLP has been used to classify different kinds of paraphasic errors with excellent accuracy (Fergadiotis, Gorman, & Bedrick, 2016).

However, currently there are no studies that have applied NLP algorithms in conjunction with neuroimaging to investigate sentence structuring during speech production in patients who develop aphasia following a stroke. This is particularly important, as agrammatism is a hallmark among many aphasic speakers as a result of deficits in morpho-syntactic encoding both during perception and production of language, typically favoring simple utterances that lack subordination while omitting and/or substituting function words (e.g. Bastiaanse, Bouma, & Post, 2009; de Roo, Kolk, & Hofstede, 2003; Lee, Milman, & Thompson, 2008).

Furthermore, the study of syntactic deficits in individuals with brain lesions after a stroke can permit, in the classical neuropsychological discipline of lesion-based studies (Rorden, Karnath, & Bonilha, 2007), the identification of regions that are critical in syntactic processing (Ding et al., 2020; Matchin et al., 2019; Mirman et al., 2019). Current models of sentence production assume three levels of processing, beginning with the creation of a pre-linguistic message, followed by a grammatical encoding phase involving the selection of lexical-semantic representations and subsequent construction of syntactic structure, and ending at the level of phonological encoding (Bock & Levelt, 1994; Dell, 1986). Indeed, the grammatical encoding / syntactic processing phase has been widely studied (Hagoort & Indefrey, 2014; Rogalsky et al., 2018), with the bulk of research centered on the importance of left hemisphere (Brennan et al., 2012; Walenski, Europa, Caplan, & Thompson, 2019), inferior frontal areas (Caramazza & Zurif, 1976; Grodzinsky & Friederici, 2006; Meltzer, McArdle, Schafer, & Braun, 2010; Pattamadilok, Dehaene, & Pallier, 2016; Santi & Grodzinsky, 2007), anterior temporal areas (Brennan et al., 2012; Grodzinsky & Friederici, 2006; Lukic et al., 2021), temporoparietal areas (Ding, Martin, Hamilton, & Schnur, 2020; Pattamadilok et al., 2016), and the structural connections between inferior frontal and posterior perisylvian cortices (Friederici, 2015; Wilson, Galantucci, Tartaglia, & Gorno-Tempini, 2012). However, there are mixed findings regarding the involvement and functions of the inferior frontal areas (Brennan et al., 2012; Grodzinsky & Friederici, 2006; Walenski et al., 2019). Syntax processing also relies on more domain-general components, such as working memory (Caplan & Waters, 1999; Fiebach, Schlesewsky, & Friederici, 2001; Makuuchi, Bahlmann, Anwander, & Friederici, 2009). In fact, Broca’s area (W. G. Matchin, 2017) and the frontotemporal functional network (Wright, Stamatakis, & Tyler, 2012) have both been associated with syntax as well as working memory. However, recent studies have suggested that a dissociation may exist between lesions to temporoparietal areas causing impaired retrieval/production of words in increasingly complex sentence contexts while damage to the inferior frontal regions may lead specifically to impairments in the production of syntactically accurate structures (Ding, Martin, et al., 2020; Pattamadilok et al., 2016).

While the relative roles of distinct brain areas in syntax remain disputed, a meta-analysis of numerous neuroimaging studies by Hagoort and Indefrey (2014) established the mechanism of a dorsal/ventral gradient in both the left inferior frontal cortex and left posterior temporal cortex, such that dorsal foci are recruited more for syntactic processing and ventral foci for semantic processing. In the current study, which focuses on speech production, there is also the issue of phonological selection mechanisms. Selecting a word in connected speech requires a balance of target word activation and inhibition of alternative forms (previous, future, competitors), all of which are coordinated across multiple hierarchical levels (Garrett, 1975; Lashley, 1951). Therefore, selecting a word during discourse likely depends on cognitive operations that extend beyond lexical processing (Fergadiotis, Wright, & West, 2013), and as such, rely on areas beyond inferior frontal gyrus, extending to the middle and superior frontal cortex, middle and superior temporal gyri encompassing even their more posterior aspects, as well as different portions of the parietal lobe (den Ouden, Hoogduin, Stowe, & Bastiaanse, 2008; Haller, Radue, Erb, Grodd, & Kircher, 2005; Kircher, Oh, Brammer, & McGuire, 2005; Menenti, Segaert, & Hagoort, 2012; Mirman, Kraft, Harvey, Brecher, & Schwartz, 2019; Schonberger et al., 2014; Timmers et al., 2015; Vigneau et al., 2006).

While most of our understanding of the neural underpinnings of syntax stems from functional approaches, as detailed above, lesion-symptom mapping of sentence production has been limited, with most studies focusing rather on syntax processing, usually with the use of non-natural tasks. Impaired sentence comprehension has been broadly mapped to lesions of the perisylvian region (Baldo & Dronkers, 2007; Kim, Jeon, & Lee, 2010; Thothathiri, Kimberg, & Schwartz, 2012) with left-hemisphere anterior superior and middle temporal gyri in particular appearing to be more specifically associated with impaired processing of non-canonical than canonical sentences (Magnusdottir et al., 2013). A recent study by Thye and Mirman found a critical role of dorsal stream lesions in speech production, but this domain included tests of repetition, serial recall span, and phonological errors during naming, rather than syntax (Thye & Mirman, 2018).

Given the complex neurobiology underlying syntax production (for review, see Petersson & Hagoort, 2012; Sprouse & Lau, 2013), there is a need to complement findings from functional activation studies with a lesion-symptom mapping approach (both voxel and connectome based) to better characterize which brain areas and white matter tracts support this language function. This is because functional studies reveal both indispensable and non-indispensable brain regions during a given task, but only lesion-symptom mapping techniques can provide objective evidence about which regions are crucial for a given behavior (Gleichgerrcht, Fridriksson, Rorden, & Bonilha, 2017). While several studies have examined neurobiological correlates of syntactic features (e.g., Matchin et al., 2020) and others have examined automated methods for analyzing connected speech (e.g., Fromm et al., 2021), the current study seeks to bring these two approaches together by analyzing connected speech with NLP in conjunction with lesion-symptom neuroimaging analysis. The ultimate goal is to better understand the underlying neurobiology of connected speech by providing a proof-of-concept approach to the application of NLP in patients with aphasia. Specifically, we applied voxel-based and connectome-based lesion-symptom mapping techniques to one aspect of syntax (parts of speech) derived from automated NLP processing of connected speech elicited by a large cohort of patients with varying types and severities of post-stroke aphasia, as well as several controls.

2. Methods

The study was approved by the Institutional Review Board at the University of South Carolina. All experiments were performed in accordance with relevant guidelines and regulations. We obtained informed consent based on the Institutional Review Board at the University of South Carolina, where all participants were tested.

2.1. Participants

We included 65 participants (63% male, 37% female) who had a single left-hemisphere ischemic stroke at least six months prior. Participants were recruited from a larger cohort of patients at the University of South Carolina and the Medical University of South Carolina. This cohort is part of a large repository of patients recruited either through self-referral, as part of trials for the treatment of aphasia, or identified for inclusion on outpatient follow-up based on a previous diagnosis of aphasia in the context of ischemic stroke. The sample in this study was selected based on the availability of baseline (i.e., pre-treatment of any kind) imaging data as well as behavioral data for the specific tasks analyzed here (see below). At time of assessment, mean age was 58.5 (SD = 10.3) and mean months post-stroke was 37.2 (SD = 40.2) months. The majority of participants identified as White (78.5%), followed by African American (20%), and Hispanic/Latino (1.5%). Most participants (87.5%) were right-hand dominant prior to stroke (12.5% left-hand dominant prior to stroke). Based on standardized language assessment (the Western Aphasia Battery, described below), 19 patients did not have aphasia, 18 had Broca’s aphasia, 14 had anomic aphasia, 9 had conduction aphasia, 3 had Wernicke’s aphasia, 1 had global aphasia, and 1 had transcortical aphasia. Assessment of aphasia diagnosis and subtype are discussed below. Figure 1 shows the extent of lesion overlap across patients. Patients had no history of other neurological or psychiatric diseases and no history or imaging evidence of other prior strokes.

Figure 1. Lesion overlay.

Lesion overlay across patients in standard neuroanatomical space.

We also recruited ten comparable healthy controls with no history of neurological or psychiatric disease, age- (66 ± 9.3, t₆₈ = 2.12, p = .06), gender (X² = 2.53, p = .11) (see Supplementary Information). We chose to include healthy controls in order to validate the parser produced by non-aphasic patients who had not sustained brain injury in addition to non-aphasic patients who had obtained brain injury. This provides more comprehensive results as to the type of language that NLP can analyze. All participants were native English speakers with adequate hearing and vision for the experimental tasks.

2.2. Language tasks

We administered the revised version of the Western Aphasia Battery (WAB; A. Kertesz, 2007) to diagnose aphasia (Aphasia Quotient, WAB-AQ), where a score below 93.7 indicates the presence of aphasia and a score below indicates the lack of aphasia. Additionally, we derived aphasia subtypes (i.e., Broca’s, Wernicke’s, etc.) based on WAB subscores (A. Kertesz, 2007). The WAB includes four subscales: speech fluency, auditory comprehension, speech repetition, and naming (See Table 1). For the purposes of this study, we focused on naming performance as a potential confounder of syntax during connected speech (see below). The naming score of the WAB is composed of a confrontation object naming task (twenty items, maximum score of 60), a semantic fluency task (patient is asked to elicit all possible animal names during one minute, maximum score of 20), a responsive speech task (patient is asked to elicit correct object name in response to specific questions, maximum score of 10), and a sentence completion task (patient is asked to elicit correct object name that completes a sentence with semantic-lexical accuracy, maximum 10 points). The total of 100 points obtained from these tasks is divided by ten to obtain the WAB’s total naming subscore (up to 10 points). Note that each one of these tasks is one component of the WAB naming subscore. For the purposes of this manuscript, when we denote “WAB naming,” we will refer to the entire subscore and specify when we refer to only one task within this subscore (e.g., confrontation naming). As discussed in the limitation section, we recognize a bias towards object naming. Additionally, in order to quantify presence and severity of apraxia of speech, as this could confound overall speech production, we also administered the Apraxia of Speech Rating Scale (ASRS), a sixteen item, 5-point scale scored based on patients responses during conversational speech, picture description, as well as word and sentence repetition tasks (Strand, Duffy, Clark, & Josephs, 2014).

Table 1:

WAB Scores by Aphasia Type

Aphasia Type Mean (SD)	*Overall Score Mean (SD)	Speech Fluency Mean (SD)	Naming Mean (SD)	Speech Repetition Mean (SD)	Auditory Comprehension Mean (SD)
Anomic	90.09 (9.85)	17.64 (1.60)	8.94 (0.61)	9.03 (0.57)	9.44 (0.59)
Broca	52.69 (16.67)	8.67 (3.01)	5.33 (2.63)	4.90 (2.45)	7.61 (1.51)
Conduction	65.06 (16.75)	13.33 (2.78)	6.10 (2.89)	5.28 (2.78)	8.38 (1.29)
Wernicke	41.77 (9.18)	11.00 (1.00)	2.23 (1.46)	1.77 (1.50)	5.88 (2.27)
**Global	23.60	3.00	1.60	3.00	4.20
**Transcortical	57.80	13.00	3.20	8.20	4.50
All Aphasia Types	65.26 (22.21)	12.43 (4.69)	6.25 (2.94)	6.06 (2.96)	8.06 (1.77)
Stroke, No Aphasia	98.44	19.95 (0.23)	9.69 (0.22)	9.73 (0.20)	9.90 (0.14)

^*Note, overall score is out of 100, speech fluency is out of 20, naming, speech repetition, and auditory comprehension are out of 10. Recall that lower scores reflect poorer performance.

^**Note these are not summary statistics since there is only one person in each of these categories

We then presented participants with three pictographic scenes placed sequentially in front of them in random order: (1) Cookie Theft scene from the Boston Diagnostic Aphasia Examination (BDAE; Goodglass & Kaplan, 1983), (2) Picnic scene from the WAB (A. Kertesz, 2007), and (3) Circus scene from the Apraxia Battery for Adults (ABA-2; Dabul, 2000). For each picture, we asked participants to describe the contents of the picture in as much detail as possible during a two-minute period (approximately six minutes of discourse). Participants produced on average 380 words (SD = 233.54) and on average 152 (SD = 68.35) unique words. We chose these instruments due to their common use in clinical practice and established validity in the literature (Dabul, 2020; Goodglass & Kaplan, 1983; Kertesz et al., 2020). Additionally, these tasks have not been used as frequently as other tasks, such as storytelling, in previous studies attempting to automate language tasks (Fromm et al., 2020; Fromm et al., 2021).

2.3. Speech transcriptions

We recorded the patients’ picture descriptions and subsequently transcribed their speech samples verbatim. The initial transcription included all fillers (e.g., uh, er, um, etc.), as well as phonetic transcriptions of neologisms and phonemic errors. Short pauses (≤ 2 sec) were indicated with a comma, and longer pauses (> 2 sec) were indicated with a full stop. In the latter case, the contiguous utterance consequently started with a capital letter. The few instances of incomprehensible words were indicated with “???” marks. Besides long pauses, sentence boundaries also reflected completed ideas or clauses, based on prosody and semantic features, following prior studies analyzing transcriptions from narrative speech (e.g. Fraser et al., 2014; Thompson et al., 2012).

2.4. Inter-rater reliability analysis of speech samples

For fifteen patients selected randomly, two independent judges received an identical, verbatim, unprocessed transcription of the speech samples from the picture description tasks free of commas or periods. They were instructed to listen to the original audio recordings of the corresponding speech sample and to edit the sample as needed to reflect short pauses (≤ 2 sec) with a comma and longer pauses (> 2 sec) with a period. They were also instructed to place periods to reflect sentence boundaries as complete ideas or clauses, based on prosody and semantic features, following prior studies analyzing transcriptions from narrative speech (e.g. Fraser et al., 2014; Thompson et al., 2012). Intra-class correlation coefficients with a two-way random model revealed excellent inter-rater reliability for the total number of commas (intraclass correlation = .98, CI = .95 to .99, df = 14, p < .001) and the total number of periods (intraclass correlation = .93, CI = .82 to .98, df = 14, p < .001).

2.5. Analysis of connected speech using Natural Language Processing

For the analysis, we removed fillers from the transcriptions. The final samples were processed using the Stanford parser (Wang, Marchina, Norton, Wan, & Schlaug, 2013) to identify parts of speech and construct parse trees. Using the latter as well as Lu’s (2012) Syntactic Complexity Analyzer, we derived twenty syntactic features, representing different aspects of the parsed tree, including sentences, clauses (structures containing at least one noun phrase and one verb phrase), noun phrases, and verb phrases (Table 2). For each of these measures, which could be interpreted as reflective of speech quantity, we also obtained measures of syntactic complexity, including width (which computes the number of leaves, i.e., a terminal tag that branches no further, in each sub-tree), height (number of levels within the sub-tree), and Yngve depth. The latter is based on Yngve’s model (1960) of language complexity, for which left-branched phrases (e.g. The paper that the researcher wrote was interesting) are more cognitively demanding than right-branched phrases (e.g. The researcher wrote a paper that was interesting). Based on the parts of speech tagging, we also counted the total number of words, the total number of nouns, and the total number of verbs (excluding modal verbs and auxiliary verbs, such as ‘be’, and ‘have’ when preceding another verb). Figure 2 shows sample sentences and the way they were parsed by the NLP algorithm. Visualization of the parsed trees was achieved by means of the Natural Language Toolkit (Bird, Loper, & Klein, 2009).

Table 2.

Definition of syntax features derived from Natural Language Processing of speech samples elicited by 65 patients with varying degrees of chronic post-stroke aphasia. Column PCA1 shows the factor loadings for each variable in regards to the first component extracted from principal component analysis including all twenty variables. Column PCA2 shows the factor loadings for each variable in regards to first component extracted from principal component analysis using only those features more reflective of sentence structure complexity than quantity, which have been identified in the table with a star.

Feature	Definition	PCA1	PCA2
Noun phrases (NP)	Number of noun phrases	0.801	-
Height*	Mean number of levels of all noun phrases	0.011	0.191
Yngve depth*	Mean Yngve depth of all noun phrases	0.003	0.063
Width*	Mean width of all noun phrases	0.007	0.110
Verb phrases (VP)	Number of verb phrases	0.471	-
Height*	Mean number of levels of all verb phrases	0.018	0.325
Yngve depth*	Mean Yngve depth of all verb phrases	0.003	0.041
Width*	Mean width of all verb phrases	0.009	0.155
Clauses	Mean number of clauses per picture description transcript [clauses have at least a noun phrase and a verb phrase]	0.354	-
Length of sentence*	Mean number of words per sentence	0.029	0.705
Clause height*	Mean number of levels per clause tree	0.022	0.419
Sentences	Mean number of sentences per picture description in transcript	0.097	-
Clause width*	Mean width per tree clause	0.009	0.163
NP to NP distance*	Mean number of elements between two contiguous noun phrases	0.009	0.195
NP to VP distance*	Mean number of nodes between two NP and VP within the same clause	0.008	0.174
VP to VP distance*	Mean number of elements between two contiguous verb phrases	0.008	0.170
Length of clause*	Mean number of words per clause	0.003	0.101
Clauses per sentence*	Mean number of clauses per sentence	0.003	0.077
Clause Yngve depth*	Mean depth per tree clause based on Yngve’s definition (see main text)	0.003	0.058
Dependent clause per clause*	Mean number of clauses that could not form a sentence on their own per total number of clauses	0.000	0.015

Figure 2. Sample parsing.

Natural Language Processing (NLP) parsing output of two sentences with one clause each but with different structural complexity. For the purposes of these examples, we show how width and height can be computed from sample noun phrases. For example, "The young boy" in Sentence A has a height = 1 and a width = 3. The noun phrase "the jar for his sister" in Sentence B has a height = 3 and a width = 5. Parts of speech tagged in these sentences include: S = clause, NP = noun phrase, DT = determiner, JJ = adjective, NN = noun, NNS = noun plural, IN = preposition, VP = verb phrase, VBZ = verb in the 3rd person, VBG = verb, gerund/present participle, PRP$ = possessive pronoun.

2.6. Performance of the parser

In order to better understand the way in which the automated parser dealt with connected speech generated by a sample of patients with varying degrees of aphasia, we performed a qualitative analysis of the parsed trees, available as supplementary material. To achieve this, two independent raters were asked to evaluate the tags assigned by the NLP algorithm to each of the three transcriptions per patient. When there was discordance between raters, the final determination of error and error type was achieved by consensus between the two raters and the PI. This occurred for 116 of 1277 total elements (53 by rater A; 63 by rater B), yielding an inter-rater kappa coefficient of 0.82 (SE = 0.02, 95% CI = 0.79 to 0.85). It is important to note that the algorithm will parse the input independently of its accuracy. This means that it will try to apply its known set of rules to structures that pose challenges intrinsic to oral (as opposed to written) discourse as well as to aphasia. For example, if a participant said, “The coo, coo, cookie jar”, the parser would then classify the repetition of the fragment ‘coo’ without the ability to make sense of what the participant is saying (in this case, the fact that ‘coo’ is an attempt to elicit the word ‘cookie’). However, we noticed that many parser errors tended to underestimate rather than inflate syntactic structure. For example, a very common and recurring error was the misidentification of the “ ‘s” verb contraction (e.g., “The boy’s stealing cookies”), which was interpreted as possessive. The parser, however, properly identified the present particle (i.e., “stealing”) as the verb, thus leading to one fewer unit of verb phrase width.

In surveying other types of errors, we noticed that some did not frankly affect output measures for the purposes of this study. For instance, “cirtrus” as a phonemic paraphasia of the word “circus” was classified as a plural noun (NNP) when it should have been tagged as a singular noun (NP). When we counted to the total number of nouns, for example, we pooled together singular and plural forms, so this mistagging was irrelevant for the outcome measures. Other errors related to paraphasias resulted again from the parser not being able to guess or interpret the overall context. For example, in the verb phrase “wash the disses,” the word “disses” is a phonemic paraphasia of “dishes” which was tagged as a verb. A human parser would have likely interpreted the paraphasia and classified it as a noun.

Importantly, we noticed that errors as determined above were consistent across all participants. That is, the parser incurred in the same errors in participants with less severe aphasia who still had stigmata of erratic speech or even non-aphasic participants who elicited speech distortions intrinsic to oral speech production (e.g., saying a fragment of a word before the target word). Excluding the common error of “’s” contraction misinterpreted as a possessive, overall, we determined that 1250 of 38,414 (3.25%) errors had some degree of misclassification, although some of these were contentious, as they implied understanding what the participant meant. When focusing solely on non-contentious errors, the error rate was 2.12% (813 words). Here, we clarify that the accuracy of the parser was interpreted solely in terms of parts-of-speech tagging and not along alternative aspects of syntax such as relational structures, accuracy of functional word classification, etc. The supplementary material includes the parsed trees for all patients as well as a detailed list of errors detected.

2.7. Statistical analyses

2.7.1. Dimension reduction of sentence production variables

The use of principal component analysis to understand and group various dimensions of neurobiological data is a widely employed approach to simplify multivariable models into clusters of variables that measure similar underlying constructs. PCA is not without limitation as a method for dimension reduction, such as the potential for grouping somewhat dissimilar variables; however, its application is often used in conjunction with lesion-symptom mapping methods in order to decrease the number of contrasts that could potentially lead to spurious significant findings (Alyahya, Halai, Conroy, & Lambon Ralph, 2018; Ding, Chen, et al., 2020; Halai, Woollams, & Lambon Ralph, 2017, 2018; Ingram et al., 2019; Schumacher, Halai, & Lambon Ralph, 2019; Zhao, Halai, & Lambon Ralph, 2020). Essentially, PCA reduces overlapping comparisons and provides latent and underlying factors. In order to reduce the number of dimensions and present a more parsimonious model for this study and future research, we initially entered all twenty syntactic features into a principal components analysis (PCA1) employing the alternating least square (ALS) algorithm in Matlab 2017b. Starting with a matrix of size 75 (patients and controls) x 20 (NLP-derived syntax variables), the ALS algorithm found the best rank-k approximation by factoring the matrix into a 65-by-k left factor matrix and a 20-by-k right factor matrix. Here, k is the number of principal components. The reduction of the dimensions was conducted by means of an iterative method that started with random initial values. The result was k principal components, each explaining a certain percentage of the variance in the raw data. Each of the twenty variables was associated with a loading coefficient for each principal component k. Each participant also had a score for each component based on regression. It is important to highlight that, as one would expect in a sample of patients with aphasia, the first component that explained a very large portion of the variance in connected speech performance was reflecting the quantity of speech produced. As detailed in section 3.1 below, this could be interpreted as speech production, which does not necessarily reflect syntax complexity beyond the volume of language output. Hence, we felt it was important to explore the set of variables derived from NLP that yielded features reflecting syntax complexity in the production of language. In a second step, we conducted principal component analysis as detailed previously but including only the 16 variables that we identified as being most reflective of syntactic complexity (PCA2). By doing so, we sought to exclude variables that solely inform absolute numbers (e.g., number of sentences) as these could bias our interpretation towards quantity of sentences and its elements rather than their complexity. A priori and per standard practice in PCA, we decided to focus on the minimum number of components that together explained at least 80% of the variance.

2.8. Neuroimaging

2.8.1. MRI scanning

We scanned patients using a 3T Siemens Trio equipped with a 12-channel head coil set up with the following specifications: (a) T1-weighted imaging sequence using an MR-RAGE (TFE) sequence with a voxel size = 1 mm3 , FOV = 256 × 256 mm, 192 sagittal slices, 9-degree flip angle, TR = 2250 msec, TI = 925 msec, and TE = 4.15 msec, GRAPPA = 2, 80 reference lines; (b) T2-MRI with a 3D SPACE (Sampling Perfection with Application optimized Contrasts by using different flip angle Evolutions) protocol with the following parameters: voxel size = 1 mm3 , FOV = 256 × 256 mm, 160 sagittal slices, variable flip angle, TR = 3200 msec, TE = 352 msec, no slice acceleration, and the same slice center and angulation was used as with the T1 sequence; (c) Diffusion EPI scan (30-directions with b=1000 s/mm2 and b=2000 s/mm2, TR = 6100ms, TE = 101ms, 82×82 matrix, 222×222mm FOV, parallel imaging GRAPPA=2, 80 45 contiguous 2.7mm axial slices, TA=390s).

2.8.2. Neuroimaging Rationale

For each main variable, we sought to use two complimentary forms of image analysis to achieve our goal of understanding neurobiological features of connected speech. First, we conducted voxel-based lesion-symptom mapping (VLSM) to identify lesions to the structural grey matter critically associated with the behavioral performance of interest. Next, we conducted a connectome-based approach to examine the critical association between loss of specific white matter tracts with the behavioral measures. This is an important perspective in understanding the neurobiology of language, since loss of function can often result not from lesion to the grey matter but from its disconnection due to loss of white matter tracts (Bonilha & Fridriksson, 2009; Catani & Mesulam, 2008). Together, these approaches provide a comprehensive and complementary view of both damaged and non-damaged tissue that contribute to connected speech. We provide details on each approach in the sections below.

2.8.3. Voxel-based lesion-symptom mapping (VLSM)

We conducted VLSM analyses to identify specific areas of brain injury associated with the syntactic features identified in our NLP analysis. In VLSM, for each voxel, a group comparison of the behavioral score is performed using pooled-variance t-tests between those patients with and without a lesion in that specific voxel (Bates et al., 2003; Gleichgerrcht et al., 2017). We only studied voxels that were damaged in at least 10 of the 65 patients. Importantly, we elected not to control for lesion volume to minimize type II errors given the statistical co-occurrence between large middle cerebral artery stroke with both inferior frontal gyrus and superior temporal gyrus lesions due to the perfusion pattern of ramifications of intracranial vessels. While isolated ischemic insults to either of these brain regions alone is possible, this scenario is far less likely. Attention to avoid type I statistical errors involved correction to each brain-behavioral analysis for multiple voxel-wise or connection-wise comparisons. We also attempted to reduce the number of multiple analyses by grouping behavioral measures into components, as described above.

VLSM was initially performed on each patient’s score for each principal component analyzed (see PCA1 and PCA2 above). We then conducted the analyses controlling for different potential confounders. These included a) the naming subscore of the WAB in order to identify brain areas associated with syntax beyond a patient’s ability to produce object names, b) the sentence completion task within the subscore of the WAB, as this task required the ability to elicit pertinent lexical items in proper sentence contexts, and c) the total score of the ASRS in order to control for the potential effect of apraxia of speech. In order to control for the effect of confounding variables on VLSM, we applied Freedman-Lane permutations (Winkler, Ridgway, Webster, Smith, & Nichols, 2014), which computes a permutation threshold for each main variable (e.g., PCA1 or PCA2) using the other variables (e.g., naming score or ASRS score) as a nuisance regressor.

We also note that controlling for naming would be incompatible with psycholinguistic models such as the cognitive grammar model (e.g. Givón, 1995; Lakoff, 1987; Langacker, 1987), which argues that sentence structure is achieved through lexical access: for these models, controlling for naming would potentially remove the step necessary to build the sentence structure. Contemporary evidence suggests a far more complex picture of independent and dissociable processes (Petersson & Hagoort, 2012; Sprouse & Lau, 2013), although some psycholinguistic models argue in favor of lexically driven syntactic priming (e.g. Bock & Ferreira, 2014). For this reason, we also presented VLSM findings of sentence structure complexity when controlling for apraxia of speech only.

Following this method, we also performed VLSM of noun phrases and verb phrases separately, as well as controlling for each other, and for the total number of nouns and verbs. The alpha value for these analyses was set at 0.05 and one-tailed, as we predicted injured tissue to cause poorer, not better, performance. We controlled for familywise error rates by means of permutation thresholding. With this approach, 3000 permutations were performed with the data to obtain a null distribution. Each statistical result (for example, a correlation coefficient) was then compared with the distribution of that same test with null data; results were considered significant if the real data were more strongly associated than 95% of the null distribution, correcting for multiple comparisons (Winkler et al., 2014).

2.8.4. Building the connectome

Two experts initially traced each stroke lesion on individual T2-weighted sequences in native space until consensus was achieved about final lesion demarcation. The sequential steps to build each patient’s individual connectome included: 1) segmenting the probabilistic gray matter map from T1-weighted images; 2) dividing the probabilistic gray matter map into 189 regions of interest (ROIs) based on the Johns Hopkins University (JHU) atlas (Faria et al., 2010); 3) segmenting the probabilistic white matter map from T1-weighted images; 4) registering the individual white matter map and cortical ROIs into the individual diffusor tensor imaging (DTI) space; 5) computing probabilistic DTI fiber tracking; 6) iterative evaluation of the number of tractography streamlines connecting each possible pair of grey matter ROIs generated in step 2 above. We employed methods designed and previously used by our group (Bonilha, Rorden, & Fridriksson, 2014) to attenuate the distorting effect of stroke-related necrotic changes on the brain, thus preserving the anatomical authenticity of grey and white matter without computing fibers embedded in these necrotic areas. Of note, the preprocessing steps excluded the lesion site from all tractography tracings (i.e., from cortical seeding, from cortical waypoints, or from white matter tracking regions). The end result of this process is a 189 x 189 matrix M for each patient, in which M(i,j) contains a measure of probabilistic connectivity strength between ROI i and ROI j.

2.8.5. Connectome-based lesion-symptom mapping (CLSM)

CLSM focuses on white matter tracts, the structural framework of the brain. A whole-brain connectome approach establishes a statistical relation between the behavioral variables and the strength of the connection between all possible pairs of ROIs (Gleichgerrcht et al., 2017). Again, we initially conducted CLSM on the principal components derived from syntax variables and then repeated the analyses with the naming and sentence completion subscores of the WAB. We then analyzed noun phrases and verb phrases, and as above, controlled for each other as well as for the number of nouns and verbs, respectively. Because of the continuous nature of both behavioral (the variables derived from speech NLP analysis) and neuroimaging (strength of all possible connections between all ROIs) variables in CLSM, we chose the general linear model. Again, the alpha value for these analyses was set at 0.05 and one-tailed, as we predicted injured white matter tracts (i.e., weaker links) would be associated with worse, not better, performance. Confounders were regressed using the Freedman-Lane extension (Winkler et al., 2014), and familywise error rates were controlled with permutation thresholding (3000 permutations), as detailed above. The methods used in this study are freely available under our software implementation niistat (https://www.nitrc.org/projects/niistat/).

3. Results

3.1. Dimension reduction of sentence structure variables

PCA of all twenty features (PCA1) extracted a first component that explained 95.1% of the variance. We will refer to this component as “sentence production.” The remaining components explained < 5% of the variance altogether and were thus excluded from further analyses (see Supplementary Table 2). Table 2 shows factor loadings for each feature in regard to the main sentence structure component. Noun phrases and verb phrases had, distinctly, the highest loading factors for this main component: 0.801 and 0.471, respectively, and were thus analyzed separately in subsequent analyses as described above.

PCA of the sixteen features that were more reflective of syntactic complexity per se (PCA2) extracted a first component that explained 81% of the variance. We hence conducted lesion mapping analyses on this component alone, controlling for the WAB naming domain subscore, apraxia of speech (ASRS total score), as well as for the total number of nouns produced. Note that the total number of nouns and the total number of verbs produced in the narrative sample were highly correlated (r = 0.81, p < .01), so we only entered the former as a nuisance regressor. We will refer to this second main component as “syntactic complexity.” Table 2 shows the factor loadings for each feature of PCA2. For a more comprehensive view of specific parts of speech, please refer to table 3 that shows the number of noun phrases and verb phrases by aphasia type, and table 4 that includes clause production by aphasia type.

Table 3:

Noun Phrase and Verb Phrase Production by Aphasia Type

Noun Phrases
Aphasia Type	Number of Phrases Mean (SD)	Height Mean (SD)	Width Mean (SD)	Yngve Mean (SD)
Anomic	158 (91)	7.65 (2.20)	5.67 (0.0.95)	2.83 (0.77)
Broca	163 (84)	7.27 (1.65)	5.54 (0.85)	2.44 (0.52)
Conduction	106 (65)	5.84 (1.44)	4.99 (1.02)	2.95 (0.95)
Wernicke	127 (47)	7.20 (1.97)	5.21 (0.46)	2.93 (0.25)
**Global	182	6.78	5.70	2.75
**Transcortical	266	9.75	7.65	3.11
All Aphasia Types	151 (82.04)	7.15 (1.88)	5.50 (0.95)	2.70 (0.72)
Stroke, No Aphasia	187 (106)	7.96 (2.24)	6.05 (1.17)	3.09 (0.58)
Controls	179.6 (76.73)	8.55 (2.27)	6.23 (0.82)	2.83 (0.51)
Verb Phrases
Aphasia Type	Number of Phrases Mean (SD)	Height Mean (SD)	Width Mean (SD)	Yngve Mean (SD)
Anomic	75 (61)	8.05 (2.72)	6.61 (1.27)	1.88 (0.56)
Broca	79 (55)	7.35 (2.73)	6.64 (0.36)	1.70 (0.36)
Conduction	58 (38)	6.75 (2.70)	6.38 (1.24)	1.87 (0.97)
Wernicke	57 (12)	7.64 (1.20)	6.88 (1.27)	1.69 (0.42)
**Global	82	7.74	6.77	1.83
**Transcortical	163	11.38	9.28	1.94
All Aphasia Types	74 (52.43)	7.56 (2.62)	6.66 (1.35)	1.79 (0.57)
Stroke, No Aphasia	101 (64)	9.37 (2.82)	7.50 (1.31)	2.18 (0.53)
Controls	183 (102)	7.37 (2.40)	5.79 (1.15)	2.86 (0.50)

^**Note these are not summary statistics since there is only one person in each of these categories

Table 4:

Clause Production by Aphasia Type

Aphasia Type	Number Mean (SD)	Length Mean (SD)	Height Mean (SD)	Width Mean (SD)	Yngev Mean (SD)
Anomic	57 (44)	7.83 (1.50)	11.55 (2.84)	7.73 (0.64)	1.28 (0.64)
Broca	58 (40)	8.11 (3.27)	10.26 (3.19)	7.43 (1.47)	1.04 (0.52)
Conduction	51 (34)	7.05 (1.61)	11.37 (3.82)	7.40 (1.50)	1.35 (0.94)
Wernicke	46 (10)	7.42 (2.12)	11.41 (2.68)	7.86 (0.80)	1.22 (0.53)
**Global	62	6.58	11.33	7.44	1.27
**Transcortical	119	8.38	14.42	9.64	1.50
All Aphasia Types	57 (38.44)	7.74 (2.36)	11.06 (3.12)	7.59 (0.64)	1.20 (0.64)
Stroke, No Aphasia	79 (49)	7.96 (1.10)	13.49 (3.57)	8.53 (1.40)	1.63 (0.49)
Controls	67 (39)	8.30 (2.27)	11.87 (3.71)	8.06 (1.28)	1.41 (0.59)

^**Note these are not summary statistics since there is only one person in each of these categories

3.2. Lesion-symptom mapping of sentence structure and complexity

VLSM revealed that sentence production (PCA1) when controlling for naming was significantly associated with voxels in the left pars opercularis, the precentral gyrus, the postcentral gyrus, and the supramarginal gyrus.

Syntactic complexity (PCA2, i.e., excluding variables that reflect speech/sentence quantity) while controlling for naming (Figure 4A and 4B) or for number of nouns (Figure 4C and 4D) revealed a similar underlying neural substrate, except that it extended more anteriorly into the frontal lobes, spanning into supplementary motor areas. Controlling for the effect of apraxia of speech, syntactic complexity was associated with integrity of the left inferior frontal gyrus, supramarginal gyrus, the more dorsal aspect of the superior temporal gyrus, and the posterior aspects of the insula (Figure 4E and 4F). Furthermore, controlling simultaneously for both naming and apraxia of speech revealed that the pars opercularis and pars triangularis of the left inferior frontal gyrus as well as the left insula were all critical regions for syntactic complexity (Figure 4G and Figure 4H).

Figure 4. Grey matter lesion mapping of sentence structure complexity (PCA2).

VLSM mapping for the first component extracted from principal component analysis of syntactic features reflecting exclusively on complexity (PCA2 or “sentence structure complexity”) controlling for (A) WAB naming, (B) total number of nouns, (C) apraxia of speech; (D) WAB naming and apraxia of speech simultaneously. Each result is presented a of three-dimensional representations (first line of each panel) and sequential axial slices (middle and bottom line of each panel).

CLSM showed that syntactic complexity was associated with damage to links connecting the superior frontal gyrus with the middle temporal gyrus, the supramarginal gyrus, and the posterior temporal gyrus, as well as the connection between the supramarginal gyrus with the superior parietal gyrus and the insula. There were no significant associations with specific links when controlling for naming on the WAB, but when controlling for the sentence completion task, syntactic complexity during connected speech was associated with the connection between the superior frontal gyrus and the supramarginal gyrus (Figure 5A). In contrast, performance on the sentence completion task when controlling for sentence structure mapped links between the supramarginal gyrus and both the superior temporal pole and the angular gyrus, as well as between the latter and the posterior superior temporal lobe (Figure 5B).

Figure 5. White matter lesion mapping.

Connectome-based lesion-symptom (CLSM) mapping for (A) the first component extracted from principal component analysis of syntactic features (“sentence structure”) controlling for the subscore on the sentence completion task of the WAB naming section and (B) vice versa. The connection between the superior frontal gyrus and the supramarginal gyrus was significantly associated with syntax beyond the effect of naming.

3.3. Lesion-symptom mapping of noun and verb phrases

Given the strong loadings of noun phrases (NP) and verb phrases (VP) onto the first component extracted from syntactic features (PCA1), further analyses focused specifically on these two variables. VLSM on VP controlling for the total number of verbs (Figure 6A and 6B), showed a similar pattern as when analyzing NP controlling for the total number of nouns elicited by the patient (Figures 6E and 6F). Relevant brain areas spanned from the posterior aspect of the frontal lobes to the post-central and supramarginal gyri posteriorly, to the posterior aspect of the superior temporal gyrus ventrally. Importantly, when the reverse analysis was conducted, that is, VLSM for number of nouns controlling for NP and for number of verbs controlling for VP, no significant brain areas were identified.

Figure 6. Lesion mapping of nouns and verbs.

The left column shows VLSM results (A) and CLSM results (B as centroids and C as stick-and-figure model) for verb phrases when controlling for the total number of verbs. The right column shows VLSM (D) and CLSM (E and F) results for noun phrases (NP) when controlling for the total number of nouns (right column, i.e., E-H). Very similar brain fronto-parietal regions and fronto-temporal, fronto-parietal, and parieto-temporal connections were associated with both classes of syntax features.

CLSM analyses followed a similar pattern: almost identical connections were seen in association with VP controlling for number of verbs (Figures 6C and 6D) and NP controlling for the number of nouns (Figures 6F and 6G); namely, the supramarginal gyrus connecting with the superior frontal gyrus, the superior parietal gyrus, and the insula. VP was also associated with the link between the posterior aspect of the superior temporal gyrus and the supramarginal gyrus, whereas NP was associated with a link between the superior frontal gyrus and the middle temporal lobe. As was the case for the comparable VLSM analyses, no significant CLSM results were revealed for the number of nouns controlling for NP and the number of verbs controlling for VP.

4. Discussion

In this study, we combined novel behavioral and neuroimaging methods to evaluate the neural bases of one aspect of syntactic complexity and sentence production during connected speech. Using natural language processing applied to normal and aphasic speech, we quantified syntactic features in terms of parts-of-speech tagging, which were mapped to brain structure via lesion-based and connectome-based analyses. Overall, our results indicate that the integrity of Broca’s area (grey matter) and the dorsal stream (white matter) are important for sentence structuring and production. More specifically, our results demonstrated that production of syntactically complex structures, as defined by increased number of parts-of-speech, controlling for the ability to produce names and for apraxia of speech, are related to the integrity of the left inferior frontal gyrus and anterior portions of the frontal lobes. Sentence production while controlling for the ability to elicit nouns or verbs in isolation (not connected speech), appears to depend primarily on fronto-parietal cortical regions and their underlying white matter connections. The production of core syntactic structures, namely noun phrases and verb phrases, was associated with fronto-temporo-parietal regions. Below, we discuss these findings in detail, examine their implications, and explore strengths and weaknesses of our methods.

4.1. The use of NLP to characterize sentence structure from connected speech

To the best of our knowledge, this is the first study to implement NLP to analyze transcripts of connected speech elicited by patients with post-stroke aphasia in conjunction with neuroimaging. It is important to highlight that the raw discourse data subjected to NLP were derived from picture description tasks. These yield outputs that mimic real-life demands more closely than constrained language, tasks such as naming or speech repetition. One potential limitation of picture description tasks, however, is that they tend to elicit a limited array of syntactic structures, primarily statements employing declarative present tense (Boschi et al., 2017). This limitation is in contrast with the output of other discourse tasks demanding temporal-sequential timelines (de Lira, Ortiz, Campanha, Bertolucci, & Minett, 2011), such as story narration (e.g. Cinderella story), or tasks that may prompt for specific types of answers, such as semi-structured interviews (Boschi et al., 2017; Ding, Martin, et al., 2020; Knibb, Woollams, Hodges, & Patterson, 2009). Manual analyses of this quantity of verbal output would be demanding both from a skills and time perspective. NLP, instead, automatically parses each sentence, identifying the syntactic role that each word plays in its grammatical context. Nonetheless, NLP required the manual transcription of the audio recordings for each participant and determination of sentence boundaries prior to NLP analyses. The former is time consuming but requires less training than syntax parsing training. The latter required more specific rules due to the potential for “contamination” of the speech samples, i.e., it required examiners to identify short vs. long pauses (commas vs. periods, respectively), which could later influence the NLP algorithm during the computation of clauses versus sentences. This was also the approach of prior studies analyzing transcriptions from narrative speech in primary progressive aphasia (e.g. Fraser et al., 2014; Thompson et al., 2012) in an attempt to capture the manner in which real-life written text reflects oral discourse. Our inter-rater analysis based on a sub-set of speech samples (described in the Supplementary Materials) demonstrated that the pre-processing of speech samples was reliable across raters, although we recognize that there are aspects of speech transcription other than sentence boundary to take into consideration when analyzing inter-rater reliability.

Without doubt, NLP parsing in aphasic speech is challenging. NLP makes mistakes in non-aphasic speech, so it is expected that the automated parsing algorithm will lead to erroneously tagged parts of speech in agrammatical language. We probed this by detecting errors in parts-of-speech tagging by manually analyzing the parsed trees. The error rate was relatively low as it relates to this aspect of syntax (available as supplementary information). Yet, there are other possible sources of error that may need to be considered, such as what the parser considers a constituent and how it deals with ambiguous words, among others. Our study does not propose that NLP is the ultimate error-free solution for analysis of connected aphasic speech, nor do we propose that this approach is able to readily and reliably capture all aspects of syntax. Instead, we present a computational technique that yields a level of noise/error in one aspect of speech production sufficiently low to warrant a lesion-symptom mapping investigation. The need for further validation of the NLP output against human parsing will be pivotal in furthering the application of automated parsing for both research and clinical uses.

PCA extracted a single component for sentence production that explained more than 95% of the variance, and this narrowed analyses to one major behavioral variable that reliably captured the array of syntactic features displayed by our heterogeneous group of post-stroke aphasic speakers. We recognized that many of the features were probably capturing speech and sentence quantity rather than complexity per se, namely number of sentences, number of clauses, number of noun phrases, and number of verb phrases. We thus repeated the PCA on a sub-set of features that captured sentence structure complexity, focusing on the main component, as it explained more than 80% of the variance.

Any lesion-based analysis employing behavioral variables from sentence structure is expected to be strongly influenced by speech production. That is, if a participant is not able to produce speech, he/she is also unlikely to properly structure sentences. For this reason, it was paramount to control for the ability to produce speech and the ability to properly string words together, as opposed to the ability to merely say words (i.e., sentence structure vs. naming). For example, an individual with aphasia may be able to say “boy, cookies, stealing” but unable to form the phrase “The boy is stealing cookies.” Besides presenting the results of the second PCA focusing on complexity of sentence structure, we used several other different levels of control measures to fully dissociate sentence structure from other features of speech production. Specifically, the analyses controlled for naming, sentence completion, the ability to produce nouns and verbs, as well as for apraxia of speech. Because the first two nuisance regressors were derived from the WAB, we recognize a bias towards object naming, as these tasks do not test action naming.

In this study, we used apraxia of speech instead of “speech fluency,” because the latter is highly intertwined with grammatical production. Specifically, the WAB speech fluency subscore is computed from the examiner’s overall evaluation of a semi-structured interview and from the picnic scene picture description task that we used as a stimulus in our study (A. Kertesz, 2007). The WAB’s speech fluency subscore is subjectively rated by the examiner taking into account grammatical competence and paraphasias. Thus, controlling for fluency would necessarily subtract the main effect that was investigated in this study, i.e., grammatical complexity during sentence production.

In addition, prior characterization of agrammatism in stroke survivors with aphasia identified several salient features. Our approach to behavioral data relied most strongly on paucity of main nouns and verbs and syntactic simplification (classically, reduced range of syntactic structures with frequent halting or inappropriate pauses), but NLP has the potential to derive information to quantify other aspects of agrammatism, such as omission of functional words, and reliance on canonical word order (Menn & Obler, 1990).

4.2. Gray matter structures

Controlling for the effect of object naming, both during confrontation naming and in the context of sentence completion, sentence production during connected speech was associated with the integrity of fronto-parietal regions. Remarkably, when focusing specifically on features that reflect syntactic complexity, the frontal regions were critical for sentence structuring and included more anterior portions of the frontal lobes, extending into areas with motor function. When controlling for the effect of naming and apraxia of speech, both of which can limit speech production, the posterior inferior frontal gyrus and posterior insula were critically associated with syntactic complexity. These results are in line with Ding et al (2020), which also found evidence for temporal-parietal connections leading to poorer syntax and Matchin et al (2020), who found damage to the middle and posterior portions of the temporal gyrus to be associated with syntactic and comprehension deficits, while damage to the posterior inferior frontal gyrus was associated with agrammatism.

A number of prior functional imaging studies employed an analogous approach to ours in that they subtracted activation during word-list encoding from activation during sentence encoding using the words elicited in the non-syntactic condition (e.g., activation when saying “boy – cookie” subtracted from activation when saying “The boy has a cookie”). Results from PET showed a clear activation of Broca’s area (Indefrey et al., 2001; Indefrey, Hellwig, Herzog, Seitz, & Hagoort, 2004), and fMRI studies further supported activation of this and additional regions, including the left supplementary motor area, central sulcus / postcentral gyrus, parietal lobe regions including superior lobule, angular gyrus, and supramarginal gyrus, superior and middle temporal lobe regions, as well as the insula (Haller et al., 2005; Mirman et al., 2019; Polczynska et al., 2017; Silbert, Honey, Simony, Poeppel, & Hasson, 2014; Tremblay & Small, 2011). In stark contrast, object naming when controlling for the effect of syntax was associated with parieto-temporo-occipital regions. This is not surprising, as prior studies have linked deficits in the production of nouns to lesions in the left, more ventral aspects of the middle and posterior temporal lobe (Kemmerer, 2014; W. Matchin et al., 2020; Matzig, Druks, Masterson, & Vigliocco, 2009; Mirman et al., 2019; Vigliocco, Vinson, Druks, Barber, & Cappa, 2011).

There is also evidence from functional imaging studies that the brain regions mapped as crucial for object naming in our study, at least as captured by the WAB, have roles in visuospatial processing and visual and spatial imagery (Buchsbaum et al., 2006; Knauff, Mulack, Kassubek, Salih, & Greenlee, 2002; Lloyd, Morrison, & Roberts, 2006; Platel et al., 1997), word retrieval (Abrahams et al., 2003), word generation (Friedman et al., 1998), and eliciting visual and semantic knowledge about objects (Kellenbach, Hovius, & Patterson, 2005), all of which are important steps in object naming (Gleichgerrcht, Fridriksson, & Bonilha, 2015).

Given the above results and the heavy load of noun and verb phrases in the first component of our principal component analysis, we also specifically mapped the brain regions that support noun phrases and verb phrases, controlling, respectively, for number of nouns and number of verbs. By doing so, we confirmed which brain regions (and later in CLSM, with white matter connections) were associated with the production of key syntax structures beyond the lexico-semantic core elements inside each component (i.e., common hierarchical features of noun and verb phrases). For instance, the ability to form the noun phrase “The boy on the stool” extends beyond the ability to name the objects “boy” and “stool.” As expected, we found that both NP and VP were associated with the integrity of similar regions as those for sentence structure. The key finding here, however, is the almost equal distribution of brain regions for these two distinct syntactic constructs, suggesting that the fronto-temporo-parietal regions identified are essential for the generation of the syntactic constructions per se rather than the generation of nouns and verbs. In fact, VLSM for the latter when controlling for NP and VP did not yield any significant results, possibly reflecting the distributed nature of lexical representations.

An important point to consider is that the application of VLSM constricts analyses only to the extent of cortical damage elicited by the vascular event. As such, we can only make neuroanatomical conclusions confined within the limits of lesioned tissue. It is also important to note that modern approaches to VLSM have suggested that the univariate approach fails to capture the nature of cortical organization; adjacent neurons are likely to share similar functions, and thus, a multivariate approach to lesion mapping may help reduce neuroanatomical bias (DeMarco & Turkeltaub, 2018). This should certainly be explored in future studies.

4.3. White matter networks

Production of syntactic features of connected speech was associated with links between the frontal and temporal areas, frontal and parietal areas, and parietal-insular connections. Of these, the links between superior frontal gyrus and the supramarginal gyrus were significantly associated with syntax after controlling for naming. This is in line with proposals that the dorsal pathway between the frontal and temporal cortices is the primary substrate for hierarchical syntactic processing (Bornkessel-Schlesewsky & Schlesewsky, 2013; Ding, Martin, et al., 2020; Friederici, 2009; Wilson et al., 2011). In contrast, a more ventral parieto-temporal network supported the generation of appropriate words in the context of sentences beyond syntactic abilities, in line with prior studies suggesting a similar network in object naming (Gleichgerrcht et al., 2016).

Importantly, and analogous to the VLSM observations described above, NP after controlling for nouns and VP after controlling for verbs were associated with a similar network of parieto-fronto-insular connections, as well as some to superior (VP) and middle (NP) temporal regions. The similarity in lesion-symptom mapping of these two distinct syntactic classes is again suggestive of a shared network that supports sentence structure in and of itself, and beyond the generation of words to refer to objects (nouns) or actions (verbs). This proposal is further supported by prior evidence showing a distinct dorsal network connecting frontal regions to support action naming and a separate ventral temporo-parieto-occipital network supporting the production of nouns (Gleichgerrcht et al., 2016).

4.4. Limitations and Future research

This study is only an initial step towards understanding connected speech and underlying neurobiology through the use of advanced computational methods. The challenges and caveats identified above can guide future research in the field. From a behavioral perspective, it would be beneficial to analyze discourse elicited in response to other types of tasks, such as narrative story tasks (similar to (Ding, Martin, et al., 2020)) and semi-structured interviews, as these may be able to elicit a different variety of syntactic structures than picture description tasks. In order to decrease human burden in the process, as speech recognition systems continue to become more sophisticated and reliable, transcriptions may become automatically derived from audio recordings during these tasks. One major limitation, naturally, will be the challenge associated with the speech of individuals with aphasia, with various degrees of phonemic errors, effortful and fragmented speech, and paraphasias. As computational tools continue to develop, we certainly recognize the importance of comparing the NLP parsing output with the parsing performed by both other parsers (e.g., CLAN, C-NNLA) and human linguists (e.g. Fergadiotis et al., 2016). For example, there is a need for studies to determine whether the way automated algorithms approach pervasive ambiguities intrinsic to human languages matches the processing and interpretation of trained linguists. Ideally, and especially given the challenges that come with aphasic speech, NLP parsing outputs should also be compared against human expert parsing for validation of the automated method. While the latter is outside the scope of this study, we have performed a qualitative analysis of NLP outputs (see supplementary information) and demonstrated a relatively low error rate, in relation to parts-of-speech tagging. We nonetheless stress that this is only one of multiple aspects of sentence complexity and encourage future studies to explore alternative domains of syntax. Indeed, what constitutes syntax and how to measure it will also remain a key aspect of the design of studies using NLP. Thus, future work should explore other syntax features, as well as consider confounders other than object naming, including action naming, speech fluency, and working memory, among others.

It will also be interesting to directly compare NLP outputs from other initiatives aimed at analyzing discourse. For example, CLAN (Computerized Language Analysis) and C-NNLA (Computerized version of the Northwestern Narrative Language Assessment) have been used successfully to automize the analysis of syntactic features of speech (Fromm, MacWhinney, & Thompson, 2020; Macwhinney, Fromm, Forbes, & Holland, 2011). The recently developed C-NNLA is an exciting tool that provides a comprehensive output to describe speech production. As CLAN and C-NNLA differ from the automated process used in this study, and it would be worthwhile to compare results across these tools. Additionally, if NLP and CLAN or C-NNLA show similar results, the former could be employed as a less burdensome and more cost-effective tool for analysis of syntactic aspects of connected speech. On the other hand, if CLAN or C-NNLA show better or more comprehensive results than NLP, these tools could be adopted in future automated speech studies. This is a crucial step for future research, as it has the capacity to streamline automated speech analysis and increase parsimony in methods used. Long-term, having a gold standard for automated speech analysis would allow easier and improved comparison across studies, potentially contributing to the understanding of language.

Importantly, our lesion-symptom mapping was conducted for behavioral measures derived from PCA. The first model (PCA1) explained a large proportion of the variance, so the results of PCA2 must be taken into consideration with this limitation in mind. However, the first component extracted from PCA2 still explained a large proportion of the variance at 81%. The mere fact that there was a significant association between specific structures and PCA2 is further evidence that this component may indeed have biological validity. By approaching the behavioral measures through PCA, we sought to avoid the collinearity of certain measures. For example, more production of verbs will inevitably be associated with more production of verb phrases (as verb phrases contain verbs), although the redundancy of certain measures in this first exploratory approach is evident.

NLP may also help identify speech biomarkers that allow for the classification of aphasias in sub-types beyond the classic categorization (i.e., expressive vs. receptive vs. conduction and so forth). For example, it may be useful in the future to identify subtypes of patients with Broca’s aphasia who exhibit different patterns of agrammatic speech, as this may lead to personalized and tailored rehabilitation goals and strategies.

Finally, and more importantly, the combination of NLP with neuroimaging can provide important information about the elements of sentence structure that are preserved after a stroke and may enable recovery or benefit from targeted therapies (Thompson et al., 1997; Thompson, Shapiro, Kiran, & Sobecks, 2003). Likewise, the absence of function or brain structure may also guide compensatory treatment strategies and enable the development of assistive devices.

5. Conclusion

In sum, our results show the importance of the dorsal stream for sentence structuring and production. By conducting automated analyses of connected speech elicited by individuals with post-stroke aphasia, we have demonstrated that syntactic production relies on brain regions primarily in the more posterior and inferior aspects of the left frontal and the parietal lobes, as well as white matter tracts connecting these areas. These regions and connections appear to be crucial for syntax beyond the ability to name objects, which our results showed relied on a more ventral and posterior set of brain regions. We also demonstrated that sentence structure complexity beyond naming ability and apraxia of speech is critically dependent on the integrity of the posterior inferior frontal gyrus and posterior insula. The use of computational methods for speech analysis in the context of lesion-symptom mapping has the potential to further our understanding of the neurobiology of human language, contribute to the identification of potential subtypes of aphasia, and thus potentially improve tailored rehabilitation strategies.

Figure 3. Grey matter lesion mapping of sentence production (PCA1).

Voxel-based lesion-symptom (VLSM) mapping for the first component extracted from principal component analysis of syntactic features (“sentence production”) controlling for the effect of naming. VLSM results are shown at the voxel-level both in 3D representation (first row) as well as in continuous axial slices (middle and bottom row). Sentence production was associated with the integrity of fronto-parietal regions.

Highlights.

• Natural Language Processing (NLP) quantifies syntactic parts-of-speech

• NLP can be applied to connected speech generated by patients with aphasia

• Syntactic complexity is associated with posterior/inferior left frontal and parietal areas

• Some aspects of sentence structuring depend on integrity of the dorsal stream of language