Abstract
Background
Low‐grade glioma (LGG) is a type of brain tumour often situated in or near areas involved in language, sensory or motor functions. Depending on localization and tumour characteristics, language or cognitive impairments due to tumour growth and/or surgical resection are obvious risks. One task that may be at risk is writing, both because it requires intact language and memory function and because it is a very complex and cognitively demanding task. The most commonly reported language deficit in LGG patients is oral lexical‐retrieval difficulties, and poor lexical retrieval would be expected to affect writing fluency.
Aims
To explore whether writing fluency is affected in LGG patients before and after surgery and whether it is related to performance on tasks of oral lexical retrieval.
Methods & Procedures
Twenty consecutive patients with presumed LGG wrote a narrative and performed a copy task before undergoing surgery and at 3‐month follow‐up using keystroke‐logging software. The same tasks were performed by a reference group (N = 31). The patients were also tested using the Boston Naming Test (BNT) and word‐fluency tests before and after surgery. Writing fluency was compared between the patients and the reference group, and between the patients before and after surgery. Relationships between performance on tests of oral lexical retrieval and writing fluency were investigated both before and after surgery.
Outcome & Results
Different aspects of writing fluency were affected in the LGG patients both before and after surgery. However, when controlling for the effect of typing speed, the LGG group differed significantly from the reference group only in the proportion of pauses within words. After surgery, a significant decline was seen in production rate and typing speed in the narrative task, and a significant increase was seen in pauses before words. Strong positive relationships were found between oral lexical retrieval and writing fluency both before and after surgery.
Conclusions & Implications
Although aspects of writing fluency were affected both before and after surgery, the results indicate that typing speed is an important factor behind the pre‐surgery differences. However, the decline in overall productivity and the increase in pauses before words after surgery could be related to a lexical deficit. This is supported by the finding that oral lexical‐retrieval scores were strongly correlated with writing fluency. However, further exploration is needed to identify the language and cognitive abilities affecting writing processes in LGG patients.
Keywords: low‐grade glioma, writing, writing fluency, keystroke logging, lexical retrieval, language
What this paper adds
What is already known on the subject
Several studies have shown that a language impairment commonly encountered in the presence of an LGG is oral lexical‐retrieval deficits. It is not known whether these problems are also present in functional writing. Writing requires an intact language and memory function, and it is a very complex and cognitively demanding task. The language and cognitive processes underlying writing are not observable in the final text but can be investigated if the writing process is recorded using keystroke‐logging software.
What this paper adds to existing knowledge
Patients with LGG were assessed before and after surgery for writing fluency in narrative writing, based on keystroke‐logging data, and for oral lexical retrieval, using tests of naming and word fluency. Compared with a reference group, these patients differed on various aspects of writing fluency, but after controlling for typing speed, the only remaining difference related to the occurrence of one type of word‐level pause. Further, the LGG patients produced fewer words per minute and made more pauses before words post‐ than pre‐surgery.
What are the potential or actual clinical implications of this work?
Combined with the finding of a strong correlation between oral lexical retrieval and writing fluency in the LGG patients, this might indicate an increase in the cognitive effort required for narrative writing and a decline in written lexical‐retrieval ability after surgery. From a clinical perspective, these results show that in order to obtain a full picture of a patient's language ability, both the oral and the written modalities need to be investigated.
Introduction
Low‐grade glioma (LGG) is a type of slow‐growing brain tumour often located in eloquent areas, i.e., those involved in language, motor or sensory functions (Duffau and Capelle 2004). Owing to its infiltrative nature, its surgical removal may cause language or other cognitive impairments. The language deficit most commonly observed in patients with LGG is lexical‐retrieval deficits in spoken production (e.g., Papagno et al. 2012). These patients also often report such difficulties themselves: in a study by Satoer et al. (2012), 56.5% of LGG patients with a tumour in the left hemisphere did so pre‐surgery and 61.9% post‐surgery. Lexical‐retrieval deficits in patients with a right‐hemisphere tumour are less studied but have also been reported; patients with such tumours were found to have lexical‐retrieval deficits in word fluency but not in naming tasks (Papagno et al. 2012).
In spoken production, lexical‐retrieval difficulties can lead to disfluent speech in that the effort to retrieve the desired word can cause pauses in the spontaneous stream of speech. Satoer et al. (2013) performed a linguistic analysis of spontaneous speech and found that patients with gliomas near language‐eloquent brain areas produced more incomplete sentences both before and after surgery, which they speculated could be due to a lexical problem. Despite the fact that lexical problems could be expected to affect written production similarly, to our knowledge no studies have investigated if disfluencies also occur in written production in this patient group.
Lexical retrieval involves an extensive neural network, predominantly in left‐hemisphere regions (Indefrey 2011), and is an important part of language, both oral and written. Several models of the components and steps are involved in lexical retrieval. Levelt et al. (1999) claimed that the process of lexical retrieval in oral language starts with conceptualization and lexical selection, involves encoding at morphological and phonological level, and ends with the articulation of a word. To this should be added that written lexical retrieval is dependent on orthographic representation and graph motor skills, whereas oral lexical retrieval builds on articulatory motor skills. Historically, written production has been assumed to mirror oral production, but Kay et al. (1992) proposed in their model that while written and spoken modalities share a common semantic system, the output lexicons are modality specific.
Analysis of a written text can provide information about lexical and syntactical features, similar to the analysis of spontaneous speech. Although text analysis can be used to investigate text quality or spelling, it cannot reveal the work and effort required by the writer, e.g., the revisions made in the text or how fluent the writer is. However, by studying the writing process by means of keystroke logging, i.e., the gradual emergence of a text while it is being produced by the writer, one can obtain very precise knowledge about the temporal patterns of language production. Keystroke logging has previously been used to investigate the writing process for persons with aphasia (Behrns et al. 2008, Leijten et al. 2015) and for persons with mild cognitive impairment due to Alzheimer's disease (Van Waes et al. 2017). Keystroke‐logging software records all keystrokes and mouse activities and their temporal distribution (Leijten and Van Waes 2013). This enables the analysis of non‐fluent and disfluent parts of the writing process, which presumably reflect the aspects of this process that are the most challenging to writers and the strategies they use to handle those challenges. Fluency analysis often involves studying where the writer pauses. Analysis of where and how often pauses occur can reveal the cognitive and linguistic processes behind text composition and language production (Leijten and Van Waes 2013). In fluent writers, low‐level processes such as spelling, lexical retrieval and transcribing (typing) tend to be automatized, meaning that such writers can devote most of their cognitive effort to high‐level processes pertaining to aspects such as text structure and evaluation. De‐automatization of low‐level writing processes causes disfluencies in the writing process and can have a negative impact on the final text (Chenoweth and Hayes 2001, Wengelin 2007). Such disfluencies occur in all text production, but they have been found to be more frequent in persons writing in a second language (Lindgren et al. 2008), persons with reading and writing disabilities (Wengelin 2007), and persons with post‐stroke aphasia (Behrns et al. 2008). In persons with reading and writing disabilities, disfluencies often occur at word level, specifically within words (Wengelin 2007).
Writing fluency has previously been measured using two types of measurements: product based and process based. A product‐based measure is obtained by dividing the size of the final text with the total time spent on the writing task. Such a measure does not take into account the fact that texts may have been edited to a varying extent. In the present study, writing fluency is instead measured using a process‐based measure. As suggested by Abdel Latif (2012), it is calculated as the mean length of ‘bursts’ or ‘chunks’ of text production between pauses. This focuses on what happens in the real‐time writing process and is therefore regarded as a valid measure of writing fluency (Abdel Latif 2012).
The language deficits found in LGG patients are often subtle, and several studies stress the importance of using sensitive tasks to evaluate language and cognition in such patients (Bello et al. 2007, Papagno et al. 2012). Even though it has been suggested that writing impairment is a sensitive indicator of mild language impairment (Keenan 1971), writing has often been overlooked. Instead, the focus has traditionally been on oral language, in both research and therapy. Despite the fact that in oral language the most commonly reported language deficit in LGG patients is lexical‐retrieval difficulties, this has not yet been investigated in writing. In written production, lexical retrieval is an important part of what Flower and Hayes (1981, later revised in Hayes 2012) term the ‘translating part’ of the writing process (i.e., translating ideas into texts). Their early model of the writing process includes three sub‐processes: planning, translating and reviewing. It describes the writing process as dynamic, stressing that all its parts interact with, and are dependent on, cognitive functions and motivation. Poor lexical retrieval would be expected to affect writing fluency, since lexical retrieval is a vital part of the translation process.
Hence, the present study aimed to explore whether writing fluency is affected in LGG patients before and after surgery and whether writing fluency is related to performance on naming or word‐fluency tests (tests of oral lexical retrieval). This was done by investigating: (1) whether there were any differences in writing fluency and word‐level pauses between LGG patients before surgery and a reference group; (2) whether LGG patients’ writing fluency and word‐level pauses at follow‐up 3 months after surgery differed from their pre‐operative performance; and (3) whether patients’ writing fluency was related to their performance on tests of oral lexical retrieval before and after surgery.
Materials and methods
Ethical considerations
The present study is covered by two ethical‐approval decisions (reference numbers 625‐14 and 525‐14) adopted by the Regional Ethical Review Board of Gothenburg, Sweden. The participants received oral and written information about the study and gave written consent. All participants were informed that they could withdraw their participation at any time.
Participants
Twenty‐five consecutive patients with presumed LGG, scheduled for surgery or biopsy at the Sahlgrenska University Hospital in Gothenburg, were recruited to the study between November 2014 and June 2016. The inclusion criteria were: over 18 years old, Swedish as a native language and no developmental reading or writing difficulties.
For five of these 25 patients, data were missing from the 3‐month follow‐up. In four patients, the reasons for missing data were the side‐effects of post‐surgery tumour treatment (such as fatigue and dizziness), while the reason was technical errors in one patient. Hence, 20 patients were included in the study. Table 1 shows their clinical characteristics.
Table 1.
Clinical characteristics of the low‐grade glioma (LGG) patients studied
| ID | Sex/age/education (years) | Handedness | Tumour location | Location in the eloquent area | Tumour type and grade | Previous tumour treatment | Seizures | A‐ning pre‐surgery | A‐ning follow‐up after 3 months | |
|---|---|---|---|---|---|---|---|---|---|---|
| 5 | F/57/16 | R | LH | Frontal | Motor | OA II | No | Yes | 5.0 | 4.2 |
| 6 | F/46/16 | R | LH | Frontal | Language | OA III | Yes | Yes | 5.0 | 4.8 |
| 8 | F/55/9 | n/a | LH | Frontal, temporal, insula | Language, motor | A IV | Yes | Yes | 4.8 | 4.4 |
| 12 | M/31/22 | R | LH | Frontal | Language, motor | A IV | No | Yes | 5.0 | 5.0 |
| 14 | M/52/16 | R | LH | Frontal, insula | Language | A IV | No | Yes | 5.0 | 4.9 |
| 15 | M/46/15 | R | LH | Multi‐focal | Several | A II | No | Yes | 5.0 | 5.0 |
| 16 | M/62/14 | R | LH | Frontal | Language | OA III | Yes | Yes | 5.0 | 4.8 |
| 23 | M/55/16 | R | LH | Temporal, insula | Language, motor | A IV | Yes | Yes | 4.9 | 5.0 |
| 24 | M/31/16 | L | LH | Insula | Language, motor | A II | Yes | Yes | 4.9 | 4.8 |
| 25 | M/25/12 | R | LH | Fronto‐temporal | Language | O III | No | Yes | 4.8 | 4.6 |
| 27 | M/56/15 | R | LH | Temporal | No | A II | No | Yes | 4.9 | 4.8 |
| 29 | M/26/16 | R | LH | Temporal | No | Ganglio‐glioma | No | Yes | 4.9 | 5.0 |
| 32 | M/53/13 | R | LH | Frontal | Motor | O II | No | Yes | 4.9 | 5.0 |
| 7 | M/43/20 | R | RH | Gyrus cinguli | No | OA III | No | Yes | 5.0 | 5.0 |
| 11 | F/56/12 | R | RH | Insula, frontal | Motor | O II | No | Yes | 5.0 | 5.0 |
| 13 | F/39/16 | R | RH | Frontal | No | O II | No | Yes | 5.0 | 5.0 |
| 17 | F/34/15 | R | RH | Parietal | No | O II | Yes | Yes | 5.0 | 5.0 |
| 20 | F/42/12 | R | RH | Frontal | Motor | A II | No | Yes | 5.0 | 5.0 |
| 26 | M/44/17 | R | RH | Frontal, temporal, insula, thalamus | Motor | O II | No | Yes | 4.9 | 5.0 |
| 28 | F/62/11 | R | RH | Frontal | Motor | A II | Yes | Yes | – | – |
Notes: F, female; M, male; R, right handed; L, left handed; LH, left hemisphere; RH, right hemisphere; OA, oligoastrocytoma; A, astrocytoma; O, oligodendroglioma.
A‐ning is a Swedish aphasia test providing both a profile of the aphasia symptoms and an index indicating the degree of severity. The index should be interpreted as follows: mild ≈ 4.5, moderate ≈ 3.2, moderately severe ≈ 3.4, severe ≈ 1.8 and very severe ≈ 0.5 (Werner and Lindström 1995). Presented are aphasia indexes from pre‐operative tests and tests performed at follow‐up 3 months after surgery.
The study also included a matched reference group of healthy adults (N = 31, 45.2% males) with no (self‐reported) neurological disease. Except for the absence of neurological disease, the inclusion criteria for the reference group were the same as for the patient group. Its members were recruited from various workplaces and organizations. No statistically significant differences were found between the two groups in age (LGG group: mean = 47.75, reference group: mean = 45.55; t(49) = 0.064, p = .949) or in duration of formal education (LGG group: mean = 14.95, reference group: mean = 16.00; t(49) = −1.336, p = .188).
Procedures
All patients in the LGG group performed writing tasks before and after surgery, and they were all assessed before and after surgery using tests measuring oral lexical retrieval and with the ‘A‐ning’ aphasia test (Werner and Lindström 1995) (results are presented in table 1). An overview of all tasks and measures used is presented in table 2. The participants in the reference group performed the same writing task as the LGG group and were also asked to perform some extended testing, the latter not included in the present study.
Table 2.
Overview of the tasks and measures used in the study
| RG | LG | Tasks | Variables | Measures of: | Defined as: |
|---|---|---|---|---|---|
| Tests of oral lexical retrieval: | |||||
| – | × | Naming test: BNT | Lexical retrieval, naming ability | Number of correct items | |
| Letter fluency: FAS | Lexical retrieval, executive function | Number of words starting with ‘F’, ‘A’ and ‘S’ respectively produced in 1 min | |||
| Semantic fluency: Animals | Lexical retrieval, executive function | Number of words belonging to the category produced in 1 min | |||
| Semantic fluency: Verbs | Lexical retrieval, executive function | Number of words belonging to the category produced in 1 min | |||
| Writing tasks in ScriptLog: | Fluency measures on text level: | ||||
| × | × | Picture‐elicited narrative task | Burst length | Writing fluency | Mean number of characters produced in linear texta between pauses above 2 s |
| Production rate | Overall productivity | Words in final text divided by total time on the task | |||
| Median inter‐key interval | Typing speed | Median time between two characters within a word | |||
| Fluency measures on word level: | |||||
| Word‐level pauses (before/within/after words) | Word‐level pauses during tasks imposing demands on lexical retrieval | Inter‐key intervals (pauses) before/within/after words above 2 s divided by all inter‐key intervals before/within/after words | |||
| Error analysis: | |||||
| × | × | Word‐level errors | Spelling | Number of misspelled wordsb divided by number of words in final text | |
| × | × | Copy task | Median inter‐key interval | Typing speed | Median time between two characters within a word |
Notes: RG, reference group; LG, low‐grade glioma (LGG) group.
‘Linear text’ is the text as it emerges on the screen, i.e., all text produced—not the final text after editing.
No distinction was made between different types of spelling errors.
Measures of oral lexical retrieval
To investigate the patients’ oral lexical retrieval, a naming test and tests of verbal word fluency were used. In the first category, the Boston Naming Test (BNT), a confrontational naming test, was used. Subjects are asked to name pictures representing nouns. In the present study, the pictures were presented digitally on a laptop (digitalization of picture material was with the permission of the copyright owner; picture material was from Kaplan et al. 2001). The test was administered and scored in accordance with Tallberg (2005).
Tests of verbal word fluency measure a person's ability to generate words within a certain category during a limited time. The tasks used here were a letter‐fluency task using the categories ‘F’, ‘A’ and ‘S’ (meaning that participants are asked to produce words beginning with those letters) (Spreen and Benton 1969) and a semantic‐fluency task using the categories of animals and verbs. Administration and scoring were performed in accordance with Tallberg et al. (2008).
Writing tasks in ScriptLog
To investigate the writing process, the ScriptLog keystroke‐logging software (Frid et al. 2014) was used. ScriptLog records information about all keystrokes pressed and all actions performed using the mouse, provided with a time stamp. All keystrokes and actions are logged on a timeline; see figure 1 for an example. The resulting pattern of keystrokes, pauses, revisions and mouse clicks enables an analysis of language behaviour to be made that sheds a light upon the cognitive processes behind the writing process (Leijten and Van Waes 2013). In addition, each recording also generates files with statistics such as the number of keystrokes, characters and pauses as well as the frequency and length of different types of inter‐key intervals.
Figure 1.
The first part of one participant's final text from the picture‐elicited task in both Swedish and English, along with the corresponding linear (‘LIN’) file with information about all keyboard and mouse actions. Hence any editing operations made by the participant can be seen. The LIN file also indicates the total recording time and the duration of any pauses longer than 2 seconds.
Before the task began, all participants were told to look through a paper version of a picture‐based story to acquaint themselves with the basic plot. The first writing task performed was then a copy task.
The copy task required participants to write a well‐known Swedish proverb, Bättre sent än aldrig (‘Better late than never’), 12 times. The rationale behind the copy task was to obtain records of each participant's text‐production behaviour in a task involving a minimum of planning or formulation so as to identify a person's basic typing characteristics, including average typing speed (measured as median inter‐key interval). Lexical‐retrieval processes in the copy task are assumed to be fast and undemanding, as the proverb is over‐learned and, hence, automatized (Grabowski 2008).
The copy task was followed by a picture‐elicited narrative task. The pictures were presented one by one on the computer screen, next to a frame in which the text was to be written. To elicit a ‘story‐telling structure’, the participants were instructed to imagine that they were writing for a child. There was no time limit on the writing task. To preserve the writers’ natural writing style, no instructions were given on how to type (one or two handed). The examiner was present during the tasks but did not elicit or participate in the writing process.
Two different sets of elicitation pictures were used to avoid a learning effect in the patients, who performed the writing tasks both before and after surgery. The pictures were obtained from two picture books for children: Frog, Where Are You? (Mayer 1969) and One Frog Too Many (Mayer and Mayer 1975). These books have previously been used in various linguistic research into written‐narrative production (for an overview, see Strömqvist et al. 2004). A short version, consisting of six pictures, was constructed for each story. These short versions of the books have been tested in Swedish and found to elicit comparable narratives with regard to text length, overall text quality and typing speed in children (Lindström 2009, Egevad 2009). Frog Where Are You? was used before surgery and One Frog Too Many was used at follow‐up 3 months after surgery.
Writing‐process measures analysed
Table 2 shows an overview of the measures analysed in the copy task and the picture‐elicited narrative task. The primary measure used to investigate writing fluency was mean burst length, but other measures, i.e., median inter‐key interval, production rate and word‐level pauses, were also compared.
Writing fluency was measured as mean burst length, i.e., the mean number of characters produced in linear text between pauses above 2 s in length. This 2‐s pause criterion is the one most commonly applied in writing research focusing on language behaviour and cognitive effort (Kaufer et al. 1986).
The production rate, i.e., the total number of words in the final text divided by the total time spent on the task, was used as a measure of the writers’ overall productivity. This measure relates the final text to the total time consumed by the writing task.
The median inter‐key interval was measured in both the copy task and the picture‐elicited narrative task. This measure reflects average typing speed. To reduce the impact of outliers, the median rather than the mean is reported. Note that in the copy task, the median inter‐key interval is measured in a task with minimal lexical‐retrieval demands.
Pause analysis was performed with regard to all word‐level pauses, i.e., pauses before, within and after words. Inter‐key intervals between words are reported as two types: after and before the pressing of the space bar, referred to as ‘before‐word’ and ‘after‐word’ inter‐key intervals respectively. However, it is unclear whether this distinction is relevant, because it might be a matter of personal writing style whether the inter‐key interval devoted to word retrieval tends to occur before or after the pressing of the space bar.
Data analysis
All writing‐process measures were analysed as described above. The tests of oral lexical retrieval were scored by one of the first authors and the total score on each test was used in the analyses. These scores were also compared with Swedish norms (Tallberg 2005, Tallberg et al. 2008) to identify any patients performing at a level indicating impaired lexical retrieval. The cut‐off for impairment chosen was 2 SDs (standard deviations) below the normed mean.
Statistical analysis
All data were visually inspected and analysed using the Shapiro–Wilk test of normality to determine whether parametric or non‐parametric tests should be used for the statistical comparisons. Since all data had a skewed distribution, non‐parametric tests were chosen in general. Where parametric tests were required, the dependent variables were log‐transformed in order to meet the assumption of normality.
The patients’ pre‐operative performance compared with their post‐operative performance on the naming and word‐fluency tests was analysed with a paired t‐test or Wilcoxon signed‐rank test. To investigate differences in the writing process between the pre‐surgery LGG group and the reference group the Mann–Whitney U‐test for independent samples was used. To explore further whether there was an effect of typing speed (median inter‐key interval in the copy task) underlying the significant differences between the LGG group and the reference group, an analysis of covariance (ANCOVA) test was performed. Because of the skewed nature of the data, all dependent variables were log transformed. The median inter‐key interval in the copy task was also log transformed and added as a covariate.
For the comparisons between the patients’ pre‐operative performance and their performance at follow‐up, the Wilcoxon signed‐rank test was used.
Logistic‐regression models were used to analyse the difference between the patient group and the reference group in terms of the proportion of word‐level pauses, controlled for typing speed. The comparisons between the groups were corrected for typing speed, age and education by adding those variables as independent variables in the regression model. Since over‐dispersion relative to the binomial distribution was observed, William's correction was used. The ‘Logistic’ procedure in SAS was used for the analyses.
The correlations between the LGG patients’ scores on tests of oral lexical retrieval and their burst length were calculated using Spearman's rank‐correlation coefficient. The following guidelines were used to interpret correlation strength: 0.20–0.39 = weak correlation, 0.40–0.59 = moderate correlation, 0.60–0.79 = strong correlation and 0.80–1 = very strong correlation (Martella et al. 2013).
The level of statistical significance was set to p < .05. IBM SPSS Statistics version 23 was used for the comparisons using the Mann–Whitney U‐test, the ANCOVA analyses and the correlation analyses. For the logistic‐regression model, SAS version 9.3 (SAS Institute Inc., Cary, NC, USA) was used.
Results
LGG patients’ performance on tests of oral lexical retrieval (naming and word fluency) before and after surgery
Table 3 presents the LGG group's scores on the language tests before and after surgery. Before surgery, one patient had an impaired performance (i.e., below 2 SD) on the BNT, and one to three patients had an impaired performance on the word fluency scores. After surgery, there was a notable decline on two of the word‐fluency tests. Six patients had an impaired performance on the letter fluency test FAS and nine patients had an impaired performance on Animals. A comparison of the patients’ performance on these tests before and after surgery revealed a significant decline on the FAS and Animals, whereas no change was seen on the BNT and Verbs.
Table 3.
Low‐grade glioma (LGG) patients’ scores on tests of oral lexical retrieval before and (3 months) after surgery
| LGG group pre‐surgery | LGG group post‐surgery | Comparisons between pre‐ and post‐surgery scores | |||||
|---|---|---|---|---|---|---|---|
| Test | Mean (SD) | Median | Number of patients identified as impaired | Mean (SD) | Median | Number of patients identified as impaired | Significance |
| BNT | 51.1 (5.10) | 52.5 | 1 | 49.3 (7.72) | 50 | 2 | Z = −1.280 |
| p = .201 | |||||||
| FAS | 38.7 (13.80) | 42 | 2 | 34.4 (18.04) | 30.5 | 6 | t = 2.214 |
| p = .039* | |||||||
| Animals | 21 (5.97) | 21 | 3 | 15.7 (7.97) | 16 | 9 | t = 3.054 |
| p = .007*, * | |||||||
| Verbs | 18.1 (6.01) | 19.5 | 1 | 19.2 (7.02) | 17.5 | 1 | t = −.861 |
| p = .425 | |||||||
Note: Analyses were performed using a paired t‐test or a Wilcoxon signed‐rank test for paired samples. * p < .05, ** p < .01, *** p < .001.
Comparison of writing fluency and word‐level pauses between pre‐surgery LGG patients and the reference group
Table 4 shows information about the writing fluency and proportions of word‐level pauses of the LGG group and the reference group. The LGG group had shorter burst lengths, meaning that they typed fewer characters between pauses. Further, the LGG group had longer median inter‐key intervals, meaning that they spent more time between keystrokes, in both the copy task and the picture‐elicited task. The LGG group also had a lower production rate (words per min), meaning that there were fewer words in their final texts in relation to the time spent on the task. No significant difference was seen for word‐level errors. Finally, the LGG group had a larger proportion of pauses longer than 2 seconds in all three types of contexts (before, within and after words).
Table 4.
Comparison of writing‐fluency measures between the pre‐surgery low‐grade glioma (LGG) group and the reference group
| LGG group (pre‐surgery), N = 20 | Reference group, N = 31 | ||||
|---|---|---|---|---|---|
| Mean (SD) | Median, minimum–maximum | Mean (SD) | Median, minimum–maximum | Significance | |
| Picture‐elicited task | |||||
| Burst length | 29.9 (22.6) | 26.3 | 43.11 (21.76) | 38.43 | U = 191 |
| 3.93–109 | 11.0–89.3 | p = .022 * | |||
| Production rate (words/min) | 16.8 (9.52) | 15.7 | 22.11 (6.16) | 21.46 | U = 162 |
| 4.48–45.6 | 8.54–33.3 | p = .004 * | |||
| Median inter‐key interval (s) | 0.27 (0.17) | 0.21 | 0.20 (0.08) | 0.17 | U = 202.5 |
| 0.09–0.85 | 0.13–0.52 | p = .038 * | |||
| Word‐level pauses | |||||
| Pauses before words (%) | 9.9 (13.5) | 5.78 | 3.77 (2.94) | 2.75 | U = 206 |
| 0–13.5 | 0–10.5 | p = .045 * | |||
| Pauses within words (%) | 1.01 (2.04) | 0.45 | 0.13 (0.21) | 0.0 | U = 187.5 |
| 0–9.18 | 0–1.03 | p = .013 * | |||
| Pauses after words (%) | 2.60 (3.60) | 1.7 | 1.05 (1.48) | 0.57 | U = 198 |
| 0–16.3 | 0–6.77 | p = .029 * | |||
| Error analysis | |||||
| Word‐level errors (%) | 1.04 (2.33) | 0.19 | 1.10 (2.14) | 0.000 | U = 306.5 |
| 0–10.4 | 0–8.72 | p = .942 | |||
| Copy task | |||||
| Median inter‐key interval (s) | 0.29a (0.21) | 0.24 | 0.21 (0.08) | 0.19 | U = 187 |
| 0.10–1.02 | 0.13–0.53 | p = .032 * | |||
Notes: aMissing data from patient 28 in median inter‐key interval in the copy task (hence, N = 19).
Statistically significant differences are shown in bold. * p < .05.
A one‐way ANCOVA test was conducted to examine whether typing speed (i.e., median inter‐key interval in the copy task) accounted for the differences between the LGG group and the reference group in terms of burst length, word‐level pauses and production rate. There was no significant effect of group (LGG group, reference group) on burst length (F(1, 47) = 2.387, p = .129) or on production rate (F(1, 47) = 3.196, p = .080) after controlling for the median inter‐key interval in the copy task.
Logistic regression was used to examine whether typing speed accounted for the differences between the LGG group and the reference group in terms of word‐level pauses. After adjusting for typing speed, age and duration of education, the odds ratio (OR) between the LGG group and the reference group for pauses within words was 2.526 (95% CI = 1.336–4.775, p = .004), meaning that pauses within words were significantly more likely in the LGG group. Age and education had no significant influence on the model (age: OR = 1.008, 95% CI = 0.975–1.042, p = .634; education: OR = 1.022, 95% CI = 0.905–1.154, p = .724). However, the LGG group was not significantly more likely to have pauses before words (OR = 1.480, 95% CI = 0.913–2.399, p = .112) or after words (OR = 1.433, 95% CI = 0.832–2.470, p = .195). Neither age nor education had any influence on the model for pauses before words (age: OR = 0.953, 95% CI = 0.953–1.002, p = .071; education: OR = 1.015, 95% CI = 0.925–1.113, p = .757) or pauses after words (age: OR = 0.976, 95% CI = 0.949–1.005, p = .101; education: OR = 1.005, 95% CI = 0.905–1.115, p = .930).
Comparison of writing fluency and word‐level pauses in the LGG group pre‐ and post‐surgery
Table 5 shows information about the LGG patients’ writing fluency before and (3 months) after surgery. There was an increase in the median inter‐key interval in the picture‐elicitation task and a decrease in the production rate. There was no significant difference in mean burst length, in the median inter‐key interval in the copy task or in word‐level errors. Finally, an increase in pauses before words was observed after surgery.
Table 5.
Performance pre‐surgery and at follow‐up 3 months post‐surgery (N = 20)
| Low‐grade glioma (LGG) group pre‐surgery | LGG group post‐surgery | ||||
|---|---|---|---|---|---|
| Mean (SD) | Median, Minimum–maximum | Mean (SD) | Median, Minimum–maximum | Significance | |
| Picture‐elicited task | |||||
| Burst length | 29.9 (22.6) | 26.3 | 25.20 (21.93) | 19.5 | Z = −1.643 |
| 3.93–109 | 1.26–88.3 | p = .100 | |||
| Production rate (words/min) | 16.8 (9.52) | 15.7 | 13.65 (8.84) | 11.0 | Z = −3.360 |
| 4.48–45.6 | 0.74–88.3 | p = .001 * | |||
| Median inter‐key interval (s) | 0.27 (0.17) | 0.21 | 0.40 (0.55) | 0.22 | Z = −2.053 |
| 0.09–0.85 | 0.10–2.58 | p = .040 * | |||
| Word‐level pauses | |||||
| Pauses before words (%) | 9.9 (13.5) | 5.78 | 17.5 (18.7) | 13.2 | Z = −2.417 |
| 0–13.5 | 0–78.3 | p = .016 * | |||
| Pauses within words (%) | 1.01 (2.04) | 0.45 | 4.45 (14.5) | 0.62 | Z = −1.008 |
| 0–9.18 | 0–65.4 | p = .313 | |||
| Pauses after words (%) | 2.60 (3.60) | 1.7 | 4.33 (4.49) | 1.7 | Z = −1.923 |
| 0–16.3 | 0.45–17.4 | p = .055 | |||
| Error analysis | |||||
| Word‐level errors (%) | 1.04 (2.33) | 0.19 | 1.15 (1.84) | 0.85 | Z = −1.720 |
| 0–10.4 | 0–4.17 | p = .085 | |||
| Copy task | |||||
| Median inter‐key interval (s) | 0.29a (0.21) | 0.24 | 0.39 (0.45) | 0.25 | Z = −1.228 |
| 0.10–1.02 | 0.10–2.16 | p = .220 | |||
Notes: aMissing data from patient 28 in median inter‐key interval in the copy task (hence, N = 19).
Statistically significant differences are shown in bold. * p < .05.
Relationship between oral lexical retrieval, writing fluency and typing speed
Before surgery, the only significant correlation found between burst length and measures of lexical retrieval was a strong positive correlation with the semantic word‐fluency measure of Verbs (table 6). This relationship implicates that patients who wrote in long bursts also produced many verbs and vice versa.
Table 6.
Pre‐surgery relationship between oral lexical retrieval, writing fluency and typing speed (N = 20)
| Tests of lexical retrieval | Writing fluency | Typing speed | ||||
|---|---|---|---|---|---|---|
| BNT | FAS | Animals | Verbs | Burst length | Median inter‐key interval in the copy task | |
| BNT | 1 | .17 | .42 | .11 | .17 | −.21 |
| FAS | 1 | .48* | .60*, * | .41 | −.42 | |
| Animals | 1 | .60*, * | .35 | −.16 | ||
| Verbs | 1 | .60*, * | −.43 | |||
| Burst length | 1 | −.83*, * | ||||
| Median inter‐key interval in the copy task | 1 | |||||
Note: Correlation analyses were performed using Spearman's rho. * p < .05, ** p < .01, *** p < .001.
By contrast, after surgery (at the 3‐month follow‐up), all measures of oral lexical retrieval showed strong positive correlations with burst length. In addition, the measures of oral lexical retrieval also all had a moderate to strong correlation with the median inter‐key interval in the copy task (table 7).
Table 7.
Post‐surgery relationship between oral lexical retrieval, writing fluency and typing speed (N = 20)
| Tests of oral lexical retrieval | Writing fluency | Typing speed | ||||
|---|---|---|---|---|---|---|
| BNT | FAS | Animals | Verbs | Burst length | Median inter‐key interval in the copy task | |
| BNT | 1 | .47* | .30 | .44 | .60*, * | −.54* |
| FAS | 1 | .83*, * | .74*, * | .79*, *, * | −.58*, * | |
| Animals | 1 | .72*, * | .67*, * | −.59*, * | ||
| Verbs | 1 | .72*, *, * | −.60*, * | |||
| Burst length | 1 | −.85*, *, * | ||||
| Median inter‐key interval in the copy task | 1 | |||||
Note: Correlation analyses were performed using Spearman's rho. * p < .05, ** p < .01, *** p < .001.
Discussion
The present study aimed to explore whether writing fluency was affected in patients with LGG before and after surgery and whether writing fluency was related to performance on tests of oral lexical retrieval. The rationale behind the aim was that patients with LGG often present with impaired oral lexical retrieval, and poor lexical retrieval could be expected also to affect the written modality and writing fluency, since lexical retrieval is an important part of the translation process during writing. The results of the present study show that different aspects of writing fluency were affected both before and after surgery in the LGG patients, and that their performance on tasks of oral lexical retrieval was related to their writing fluency.
Before surgery, only a few patients were identified as having an impaired oral lexical retrieval, as indicated by the naming and word fluency tests. However, after surgery a significant decline was seen on two of the word fluency tasks (FAS and Animals), and 45% of the patients had an impaired performance on the semantic word fluency task Animals. The patients’ writing fluency and the proportion of word‐level pauses were investigated to examine if the problems in lexical retrieval seen in oral modality affected written modality. Before surgery, the LGG patients were less fluent than the reference group in the sense that they typed significantly fewer characters between pauses, produced fewer words per min, typed more slowly and had a higher prevalence of all types of word‐level pauses. However, after controlling for the effect of typing speed (median inter‐key interval in the copy task), only the proportion of pauses within words differed significantly between the LGG group (before surgery) and the reference group. A significant post‐surgery decline was seen in the LGG patients with regard to both production rate and typing speed in the narrative task, and a significant increase was seen in pauses before words.
The patients’ results on tests of naming and word fluency tests were correlated with mean burst length in writing to investigate whether the patients’ writing fluency was related to lexical‐retrieval ability before and after surgery. Positive relationships were found between measures of oral lexical retrieval and writing fluency in the LGG patients both before and after surgery, but these relationships were all stronger after surgery.
Typing speed affects writing fluency
Since there were no differences in writing fluency or production rate between the LGG group and the reference group when controlling for typing speed, factors related to typing speed may account for the differences in writing fluency. Typing speed was measured as the median inter‐key interval in the copy task. While generally regarded as a valid measure of typing speed in persons without language or cognitive deficits (Grabowski 2008), it should be pointed out that a copy task cannot be deemed to yield a pure measure of the motor aspects of typing speed. A copy task at word or sentence level actually does require some cognitive effort, and this may be particularly relevant in persons with impaired cognitive functions. In his revised model, Hayes (2012) makes a distinction between translating (transforming ideas into text) and transcribing (typing or transcribing by hand). For adult skilled writers, transcribing is generally considered to be a thoroughly automatized process. However, several studies have shown that the transcription rate (i.e., typing speed) is negatively affected when cognitive effort is increased. Hayes and Chenoweth (2006) found that their participants’ transcription rate was negatively affected when an articulatory‐suppression task was added (such as having to repeat irrelevant speech while writing), presumably because it increased the cognitive load and reduced the availability of verbal working memory. Similar results have also been reported from investigations of how transcribing speed is affected when the cognitive effort is increased by a requirement to write in upper‐case letters only (Bourdin and Fayol 1994). These results show that, even though transcribing is an automatized process, it does compete with other writing processes for cognitive resources, meaning that it is vulnerable to cognitive impairment. Indeed, a slower typing speed has been reported in persons with mild cognitive impairment or mild dementia compared with age‐matched controls (Van Waes et al. 2017) and in adults with reading and writing difficulties (Wengelin 2002).
One can only speculate what factors explain the differences in typing speed found between the LGG patients and the reference group. It is known that LGG patients may present with deficits in cognitive functions such as executive functions, working memory and processing speed (e.g., Talacchi et al. 2011, Santini et al. 2012, Satoer et al. 2012)—all of which are abilities that could affect typing speed and, hence, writing fluency. In the present study (which is part of a project focusing on effects on language ability following LGG surgery), only tests of lexical retrieval (naming and verbal word fluency) were used. There is some controversy about whether the critical underlying component in oral word‐fluency tests is language processing or executive function (Whiteside et al. 2016), but such tests are generally deemed to measure both language and executive functioning (Moscovitch 1994). Nonetheless, since the tests of oral lexical retrieval were not administered to the reference group, scores on them cannot be used as control variables. It would be desirable for future studies investigating writing fluency in this patient population to include tests of other important cognitive abilities such as verbal working memory, executive functions and processing speed. Further, there are other factors that have not been controlled for in the present study but which may have affected the patients’ typing speed, including motor skills, the effects of antiepileptic treatment, or fatigue due to testing or side‐effects from having a brain tumour.
Higher prevalence of word‐level pauses both before and after surgery
The frequency and distribution of word‐level pauses were investigated since a high prevalence of such pauses could indicate a lexical or orthographic problem. As previously mentioned, word‐level pauses are common in the writing process of persons with reading and writing disabilities (Wengelin 2007, Leijten et al. 2015).
Before surgery, the LGG group had a significantly larger proportion of pauses in all three contexts (before, within and after words) than the reference group. However, after controlling for typing speed, only the frequency of within‐word pauses differed significantly between the groups. Pauses within words tend to be infrequent in adult writers. They are generally considered to be a result of the writer's uncertainty about spelling (Wengelin 2007). Whether this is the case here is unclear. The LGG patients made few spelling errors; they did not have a larger proportion of word‐level errors than the reference group. It was not explored whether they frequently felt uncertain about spelling yet did not end up making a spelling mistake. In the present context, however, it is relevant to consider the possibility that pauses within words could also be interpreted as reflecting difficulties in lexical retrieval, as written lexical retrieval could differ from oral lexical retrieval. Nottbusch et al. (2007) showed that inter‐key intervals within words tend to be longer at syllable and morpheme boundaries; this may indicate that, in written production, a word is retrieved not as one unit but as several parts.
After surgery, there was a significant increase in the proportion of pauses before words in the patients’ written narratives. This may indicate a decline in lexical retrieval, making the patients pause more before words. The mean scores on all tests of lexical retrieval were lower after surgery, and those scores all had a strong relationship with writing fluency.
Decline in production rate after surgery
After surgery, a reduction was seen in the production rate, reflecting a decline in overall writing productivity. Although research on writing by people with acquired language disturbances is scarce, Behrns et al. (2008) showed that the mean production rate of persons with aphasia was three times lower than that of a reference group. This difference was thought to be related to lexical‐retrieval difficulties and time‐consuming editing.
There are probably several reasons for the decline in production rate seen in this study. A slower typing speed would affect the production rate, as would a higher frequency of word‐level pauses (which was observed here after surgery with regard to pauses before words). The LGG group had slower typing speed in both the narrative task and the copy task after surgery, but the decrease was significant only for the narrative task, probably because of the difference in nature between these tasks: the copy task requires mostly automatized processing, whereas the narrative one involves both high‐ and low‐level processing. What is more, the demands placed on orthographic skill were minimal in the copy task, since the participants could see the target sentence written on the computer screen.
Are writing fluency and oral lexical retrieval related?
The relationship between oral lexical retrieval and writing fluency was investigated because lexical‐retrieval difficulties in spoken production are a commonly reported problem in LGG patients (e.g., Papagno et al. 2012, Satoer et al. 2012). A strong positive relationship between scores on the naming test and writing fluency was seen before surgery. After surgery, all tests of oral lexical retrieval had strong positive relationships with writing fluency. The reason why correlations were stronger after surgery is probably that more patients had an impaired lexical‐retrieval ability, which influenced both their performance on the oral lexical‐retrieval tasks and their writing fluency.
The tests of oral lexical retrieval were chosen since they measure different aspects of lexical retrieval. A naming test such as the BNT aims to measure all aspects of retrieving a word from conceptualization, lexical selection, morphological and phonological encoding, and articulation. Word‐fluency tests are also used as measures of lexical retrieval but predominantly measures the executive parts of the word retrieval such as attentional, planning and inhibition aspects of lexical retrieval. Since performance on both types of language tasks (writing tasks and lexical‐retrieval tasks) is dependent on other cognitive abilities such as working memory and executive functions, it cannot be ruled out that such other abilities represent underlying factors influencing the correlation analyses. To explore further the relationship between lexical retrieval and writing fluency, additional measures pertaining to other cognitive functions should be controlled for, or if possible included in a model to identify the factors that predict writing fluency.
Limitations and future directions
One possible limitation of the present study is that two different sets of elicitation pictures were used before and after surgery respectively to avoid a learning effect. Differences in the data might be artefacts of the use of two different stories, as these picture sets were not counterbalanced. However, all analyses were based mainly on relative measures and the types of inter‐key intervals analysed were ones that occurred frequently both pre‐ and post‐surgery.
The choice of a picture‐elicited narrative task was intended to prompt narratives with a comparable basic story‐telling structure. However, the narrative task also had to be open enough to yield a personal narrative that reflected the participants’ functional writing and make demands on their lexical‐retrieval function. A more structured task such as a written naming task using keystroke logging could elicit the comparisons between lexical‐retrieval ability in written and spoken modality. On the other hand, such a task would not have reflected the functional writing of narratives. A less structured task without picture elicitation would have enabled analysis of certain lexical features, such as lexical diversity, as well as holistic assessment of text quality, which in turn would have enabled analysis of how high‐level processes are affected by LGG surgery.
Another limitation was the limited sample size. A larger sample size would have allowed for an exploration of the effect of tumour location, including hemispheric differences and tumour location in different eloquent areas. Other potential sources of variation that would be interesting to explore in future studies including a larger sample size are the effect of previous tumour treatment, seizure load, motor ability and fatigue.
The present study analysed only word‐level pauses, given that its focus was on lexical retrieval. However, further analyses of pauses at boundaries between sentences or clauses could have provided information about whether the patients’ general pause pattern changed or whether only the proportion of word‐level pauses was affected. Such exploration would require a different kind of task to elicit longer texts than the ones used in the present study. Since the patients tended to write only a single sentence per picture, there were very few inter‐key intervals at sentence or clause boundaries in the material studied.
Conclusions
The present study shows that certain aspects of the writing process were affected both before and after tumour surgery. The differences in writing fluency and production rate between LGG patients before surgery and the reference group might be explained by the patients’ slower typing speed. While no conclusions can be drawn about the reasons for that difference in typing speed based on the results of the present study, it is possible that a decline in underlying cognitive abilities, such as processing speed, executive functions and/or working memory, may be at least part of the explanation.
The decline in the production rate and the increase in before‐word pauses seen after surgery could be due to a lexical deficit. This is supported by the finding of strong correlations between scores on the tests of oral lexical retrieval and writing fluency. However, since the oral lexical‐retrieval scores correlated with typing speed as well, it cannot be ruled out that the explanation here, too, is a decline in underlying cognitive abilities. Even so, regardless of what the underlying deficiency may be, investigating both oral and written language is important to understand fully the functional impact of having a LGG. Further exploration is needed to identify the language and cognitive abilities that affect the writing process in patients with LGG.
Acknowledgements
Declaration of interest: The authors report no conflicts of interest. The authors alone are responsible for the content and writing of the paper.
Contributor Information
Malin Antonsson, Email: malin.antonsson@neuro.gu.se.
Charlotte Johansson, Email: charlotte.johansson@neuro.gu.se.
References
- Abdel Latif, M. M. M. , 2012, What do we mean by writing fluency and how can it be validly measured? Applied Linguistics, 34(1), 99–105. [Google Scholar]
- Behrns, I. , Ahlsen, E. and Wengelin, Å. , 2008, Aphasia and the process of revision in writing a text. Clinical Linguistics and Phonetics, 22, 95–110. [DOI] [PubMed] [Google Scholar]
- Bello, L. , Gallucci, M. , Fava, M. , Carrabba, G. , Giussani, C. , Acerbi, F. , Baratta, P. , Songa, V. , Conte, V. , Branca, V. , Stocchetti, N. , Papagno, C. and Gaini, S. M. , 2007, Intraoperative subcortical language tract mapping guides surgical removal of gliomas involving speech areas. Neurosurgery, 60, 67–80; discussion, 80–82. [DOI] [PubMed] [Google Scholar]
- Bourdin, B. and Fayol, M. , 1994, Is written language production more difficult than oral language production? A working memory approach. International Journal of Psychology, 29, 591–620. [Google Scholar]
- Chenoweth, N. A. and Hayes, J. R. , 2001, Fluency in writing: generating text in L1 and L2. Written Communication, 18, 80–98. [Google Scholar]
- Duffau, H. and Capelle, L. , 2004, Preferential brain locations of low‐grade gliomas. Cancer, 100, 2622–2626. [DOI] [PubMed] [Google Scholar]
- Egevad, C. , 2009, Motorisk skrivförmåga och textkvalitet: En jämförande studie av hand‐ och datorskrivna berättelser i skolår 7. Unpublished master's thesis, Department of Logopedics, Phoniatrics and Audiology, Lund University.
- Flower, L. and Hayes, J. R. , 1981, A cognitive process theory of writing. College Composition and Communication, 32, 365–387. [Google Scholar]
- Frid, J. , Johansson, V. , Johansson, R. and Wengelin, Å. , 2014, Developing a keystroke logging program into a writing experiment environment. Paper presented at Writing Across Borders, Paris, France, 19–22 February.
- Grabowski, J. , 2008, The internal structure of university students’ keyboard skills. Journal of Writing Research, 1, 27–52. [Google Scholar]
- Hayes, J. R. , 2012, Modeling and remodeling writing, Written Communication, 29(3), 369–388. [Google Scholar]
- Hayes, J. R. and Chenoweth, A. N. , 2006, Is working memory involved in the transcribing and editing of texts? Written Communication, 23(2), 135–149. [Google Scholar]
- Indefrey, P. , 2011, The spatial and temporal signatures of word production components: a critical update. Front Psychology, 2, 1–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kaplan, E. , Goodglass, H. , Weintraub, S. , Segal, O. and Van Loon‐Vervoorn, A. , 2001, Boston Naming Test (Austin, TX: PRO‐ED; ). [Google Scholar]
- Kaufer, D. S. , Hayes, J. R. and Flower, L. , 1986, Composing written sentences. Research in the Teaching of English, 20(2), 121–140. [Google Scholar]
- Kay, J. , Lesser, R. and Coltheart, M. , 1992, Psycholinguistic Assessments of Language Processing in Aphasia (PALPA) (Hove: Erlbaum: /Psychology Press; ). [Google Scholar]
- Keenan, J. , 1971, The detection of minimal dysphasia. Archives of Physical Medicine and Rehabilitation, 52, 227–332. [PubMed] [Google Scholar]
- Leijten, M. , Drijbooms, E. and Behrns, I. , 2015, Cognitive writing process characteristics in aphasia patients. Paper presented at Collaboration of Aphasia Trialists, CATs, Future Directions for Aphasia Research, City University, London, UK, 6 March.
- Leijten, M. and Van Waes, L. , 2013, Keystroke logging in writing research: using Inputlog to analyze and visualize writing processes. Written Communication, 30(3), 358–392. [Google Scholar]
- Levelt, W. J. , Roelofs, A. and Meyer, A. S. , 1999, A theory of lexical access in speech production. Behavioral and Brain Sciences, 22, 1–38. [DOI] [PubMed] [Google Scholar]
- Lindgren, E. , Spelman Miller, K. and Sullivan, K. P. , 2008, Development of fluency and revision in L1 and L2 writing in Swedish high school years eight and nine. ITL: International Journal of Applied Linguistics, 156, 133–151. [Google Scholar]
- Lindström, C. , 2009, Kognitiva processer i barns skrivande—En jämförande studie av hand‐ och datorskrivna berättelser. Unpublished master's thesis, Department of Logopedics, Phoniatrics and Audiology, Lund University.
- Martella, R. C. , Nelson, J. R. , Morgan, R. L. and Marchand‐Martella, N. E. , 2013, Understanding and Interpreting Educational Research (New York, NY: Guilford; ). [Google Scholar]
- Mayer, M. , 1969, Frog, Where Are You? (New York, NY: Dial; ). [Google Scholar]
- Mayer, M. and Mayer, M. , 1975, One Frog Too Many (New York, NY: Penguin; ). [Google Scholar]
- Moscovitch, M. , 1994, Cognitive resources and dual‐task interference effects at retrieval in normal people: the role of the frontal lobes and medial temporal cortex. Neuropsychology, 8, 524–534. [Google Scholar]
- Nottbusch, G. , Weingarten, R. and Sahel, S. , 2007, From written word to written sentence production In Torrance M., van Waes L. and Galbraith D. W. (eds), Writing and Cognition. Research and Applications (Studies in Writing, vol. 20, pp. 31–53). Amsterdam: Elsevier. [Google Scholar]
- Papagno, C. , Casarotti, A. , Comi, A. , Gallucci, M. , Riva, M. and Bello, L. , 2012, Measuring clinical outcomes in neuro‐oncology. A battery to evaluate low‐grade gliomas (LGG). Journal of Neurooncology, 108, 269–275. [DOI] [PubMed] [Google Scholar]
- Santini, B. , Talacchi, A. , Squintani, G. , Casagrande, F. , Capasso, R. and Miceli, G. , 2012, Cognitive outcome after awake surgery for tumors in language areas. Journal of Neurooncology, 108, 319–326. [DOI] [PubMed] [Google Scholar]
- Satoer, D. , Vincent, A. , Smits, M. , Dirven, C. and Visch‐Brink, E. , 2013, Spontaneous speech of patients with gliomas in eloquent areas before and early after surgery. Acta Neurochirurgica (Vienna), 155, 685–692. [DOI] [PubMed] [Google Scholar]
- Satoer, D. , Vork, J. , Visch‐Brink, E. , Smits, M. , Dirven, C. and Vincent, A. , 2012, Cognitive functioning early after surgery of gliomas in eloquent areas. Journal of Neurosurgery, 117, 831–838. [DOI] [PubMed] [Google Scholar]
- Spreen, O. and Benton, A. L. , 1969, Neurosensory Center Comprehensive Examination for Aphasia (Victoria, BC: University of Victoria; ). [Google Scholar]
- Strömqvist, S. , Nordqvist, Å. and Wengelin, Å. , 2004, Writing the Frog Story—developmental and crossmodal perspectives In Strömqvist S. and Verhoeven L. (eds), Relating Events in Narrative—Typological and Contextual Perspectives (pp. 359–94) (Mahwah, NJ: Lawrence Erlbaum Associates; ). [Google Scholar]
- Talacchi, A. , Santini, B. , Savazzi, S. and Gerosa, M. , 2011, Cognitive effects of tumour and surgical treatment in glioma patients. Journal of Neurooncology, 103, 541–549. [DOI] [PubMed] [Google Scholar]
- Tallberg, I. M. , 2005, The Boston Naming Test in Swedish: normative data. Brain and Language, 94, 19–31. [DOI] [PubMed] [Google Scholar]
- Tallberg, I. M. , Ivachova, E. , Jones Tinghag, K. and Östberg, P. , 2008, Swedish norms for word fluency tests: FAS, animals and verbs. Scandinavian Journal of Psychology, 49, 479–485. [DOI] [PubMed] [Google Scholar]
- Van Waes, L. , Leijten, M. , Mariën, P. and Engelborghs, S. , 2017, Typing competencies in Alzheimer's disease: an exploration of copy tasks. Computers in Human Behavior, 73, 311–319. [Google Scholar]
- Wengelin, Å. , 2002, Text Production in Adults with Reading and Writing Difficulties (Gothenburg: Department of linguistics, Gothenburg University; ). [Google Scholar]
- Wengelin, Å. , 2007, The word‐level focus in text production by adults with reading and writing difficulties In Torrance M., van Waes L. and Galbraith D. W. (eds), Writing and Cognition. Research and Applications (Studies in Writing, vol. 20, pp. 67–82). Amsterdam: Elsevier. [Google Scholar]
- Werner, C. and Lindström, E. , 1995, A‐ning. Neurolingvistisk afasiundersökning (Stockholm: Ersta diakonisällskap; ). [Google Scholar]
- Whiteside, D. M. , Kealey, T. , Semla, M. , Luu, H. , Rice, L. , Basso, M. R. and Roper, B. , 2016, Verbal fluency: language or executive function measure? Applied Neuropsychology: Adult, 23, 29–34. [DOI] [PubMed] [Google Scholar]