The ontogeny of visual–motor memory and its importance in handwriting and reading: a developing construct

Abstract

Humans have evolved a remarkable ability to remember visual shapes and use these representations to generate motor activity (from Palaeolithic cave drawings through Jiahu symbols to cursive handwriting). The term visual–motor memory (VMM) describes this psychological ability, which must have conveyed an evolutionary advantage and remains critically important to humans (e.g. when learning to write). Surprisingly, little empirical investigation of this unique human ability exists—almost certainly because of the technological difficulties involved in measuring VMM. We deployed a novel technique for measuring this construct in 87 children (6–11 years old, 44 females). Children drew novel shapes presented briefly on a tablet laptop screen, drawing their responses from memory on the screen using a digitizer stylus. Sophisticated algorithms (using point-registration techniques) objectively quantified the accuracy of the children's reproductions. VMM improved with age and performance decreased with shape complexity, indicating that the measure captured meaningful developmental changes. The relationship between VMM and scores on nationally standardized writing assessments were explored with the results showing a clear relationship between these measures, even after controlling for age. Moreover, a relationship between VMM and the nationally standardized reading test was mediated via writing ability, suggesting VMM's wider importance within language development.

1. Introduction

An important evolutionary advantage is conferred to humans via their unique ability to communicate across time and space through the transmission of manually produced symbols (e.g. writing) [1]. Thus, gaining insight into the development of the underpinning cognitive processes that have evolved to enable humans to use writing systems for communication is of great interest in understanding the ontogeny of this unique human ability. Moreover, the ability to produce and interpret written symbols remains an essential part of every child's development.

The component skills that enable the motor activity of picking up a pen or pencil and drawing an alphanumerical symbol are both complex and diverse [2]. Nonetheless, the fundamental challenge is one of learning how to generate motor commands that result in an effector (the hand) producing a graphical representation of a memorized shape (alphanumerical symbol). Thus, learning to write is contingent on a cognitive ability to remember visual patterns and recruit the appropriate neural circuit to translate these patterns from memory to page. We define this psychological process as ‘visual–motor memory’ (VMM) for symbolic representations (i.e. memory of a visual pattern and how to reproduce an approximation of the shape via the motor system).

We predict that VMM must underpin the procedural aspects of learning to write and hypothesize that this cognitive skill is the pathway through which increased automaticity in handwriting emerges with practice. Namely, as individuals practise they become quicker to recall and execute the commands necessary to produce legible letter/word forms [3]. It also follows that this ‘routinizing’ should free up cognitive resources for more abstract higher-order language processes (e.g. composition, syntax, spelling), which develop concurrently with learning to write [3,4]. Thus, it is plausible that VMM ability may indirectly influence the rate of development of these non-motoric language processes.

Furthermore, based on an embodied theory of cognition [5], we hypothesize that VMM ability should affect written language recognition, as well as influencing written language production. In other words, VMM should also support reading abilities. Indeed, it is probable that more practised and procedural recall of letter/word forms while writing could aid pattern recognition when reading. This proposal is supported by evidence showing that the motor processes associated with writing reinforce a child's ability to recognize alphanumerical symbols [6]. Longcamp et al. [7] have demonstrated the importance of learning the motor representations of symbols for later visual recognition in adults. They taught participants new characters taken from the Gujarati or Bengali alphabets: half were trained using a typewriter and half by copying the characters by hand. Participants in the handwriting group were better able to recognize the new characters and retained this improved memory over time. Longcamp et al. [8] found improvement for character recognition in 5-year-olds when they learnt the letters through copying compared with typing, whereas Naka [9] showed that repeated writing of Chinese or Arabic characters by Japanese primary school children led to increased recall compared with just looking at the characters. Most recently, brain-imaging research has suggested that in pre-literate children the neural pathways associated with reading only activate in response to viewing letters if a child has previously been trained to print these letters free-form, as opposed to tracing their outline or typing them on a keyboard [10]. This implies that the activity of handwriting (and VMM) is advantageous for reading because it facilitates deeper knowledge of the component features that constitute a letter's form, aiding children's ability to distinguish and categorize letters.

There are a number of different models that attempt to capture the neural processes involved in writing and reading [11]. Nonetheless, most theorists agree that the complex skill of writing relies on a distributed set of cognitive processes that support the creation of orthographic representations. These representations are in turn used to activate the motor cortex and thereby generate hand movements [12,13]. Orthographic representations are presumed to form an autonomous lexicon, which is linked with other language representations in long-term memory. This lexicon is pivotally involved in both encoding (spelling) and decoding (reading) processes [14]. The long-term creation of a durable orthographic lexicon must begin with the production of temporary representations formed when unfamiliar symbols or words are encountered [13]. Repeated exposure would then cause these incomplete representations to become more permanent and autonomous (thereby negating the need for effortful decoding or encoding processes). The production of temporary orthographic representations seems likely to involve the working memory network, particularly visuo-spatial short-term memory. Thus, VMM can be conceptualized within the wider theoretical context of the working memory processes that support the production of temporary orthographic representations (with VMM being a critical component for the long-term creation of an autonomous orthographic lexicon). It is probable that these processes involve a widely distributed neural network including bilateral occipital cortex, the right temporal lobe, both parietal and frontal lobes, and subcortical regions [14].

To date, it has not been possible to test the hypothesized importance of VMM to handwriting or explore the possibility that VMM may play a role in wider aspects of language development. This is because technological limitations have meant that it has not been possible to measure an individual's ability to graphically reproduce a shape in sufficient detail to justify a rigorous scientific investigation of this ability. For example, the Alphabet Writing component of the Written Expression subscale of the Wechsler Individual Achievement Test second edition (WIAT-II) [15] requires children to write the letters of the alphabet on lined paper for 30 s. These letters are then assessed visually by researchers and scored on the basis of factors such as alignment and proportionality [16]. Swanson & Berninger [17] assessed handwriting by asking children to copy a portion of text and then visually examining it to award points for whether or not individual words were legible. These techniques are inappropriately subjective for a scientific investigation of VMM. Widely used standardized assessments of general fine-motor control skills also assess children's manipulation of a stylus in coarse ways. The Beery–Buktenica test of visuo-motor integration [18], widely used to assess handwriting difficulties in children [19], only judges an individual's ability to copy a set of abstract shapes on a set of pass/fail criteria. These types of subjective and categorical measurements are not able to account for subtle differences in the ability to reproduce a pattern from memory using a stylus (i.e. the core functional challenge in handwriting) and will thus inevitably produce unsophisticated estimations of handwriting ability.

Fortunately, recent innovations have allowed researchers to use digital tablets to record with precision children's handwriting and more general stylus manipulation skills [20–23]. One such system, developed by Culmer et al. [24], uses specialist software to capture kinematic data via a tablet laptop, with the screen acting as the writing surface and a digitizer stylus as the pen (the digital equivalent of using a pen with paper). This technology has been used to present two-dimensional line-drawing stimuli on the screen, which participants either need to trace over or simultaneously copy on another area of the screen. Robust point-set registration methods can then be used to post-process the participants' drawings to generate error scores that provide objective measurements of the participants' ability to accurately trace/copy the stimuli presented [25,26]. This is exactly the type of technique required to capture meaningful measures of VMM ability. Using such a testing system, participants can be asked to reproduce (from memory) shapes previously presented on the tablet's screen, providing a direct and objective assessment of VMM.

We therefore set up the following cross-sectional study to measure this ability in a sample of school-aged children (6–11 years old). We examined whether this skill related to and underpinned children's writing ability, and whether it contributed to their reading skill. We addressed these issues by relating VMM scores to UK standardized scores of the children's writing and reading ability supplied by the school. We predicted a relationship between VMM and children's writing ability that would mediate a further relationship between children's writing and reading abilities.

2. Material and methods

(a). Participants

An opportunity sample of 87 children (44 females) was recruited from a primary school in West Yorkshire: 33 from year 2 (age range 6.7–7.7 years, mean = 7.1 years), 27 from year 4 (8.5–9.4 years, mean = 8.9 years) and 27 from year 6 (10.6–11.5 years, mean = 11.0 years). Gender was approximately equally split across each age group. Eight per cent of participants were left-handed, also evenly distributed across all groups. All participants had English as their first language, had normal or corrected-to-normal vision, and no history of neurological disorder.

(b). Apparatus

A specialized software program presented visual stimuli while simultaneously recording participants' kinematic responses via a hand-held stylus [24]. The software platform was used on a tablet computer (Toshiba Portege M700–13P tablet, screen area 260 × 163 mm, 1200 × 800 pixels, 60 Hz refresh rate), with the screen digitizer measuring planar position of the stylus at a rate of 120 Hz, allowing precise measurements of complex movement to be reliably captured.

(c). Procedure

To measure VMM, participants were seated comfortably at a table, and the tablet screen was rotated 180° and folded down to create a horizontal ‘writing surface’ in front of them. Participants used the pen-shaped digitizing stylus as an input device to interact with the screen.

The VMM task required participants to place the stylus on a circle at the bottom of the screen. This subsequently caused a shape to appear on screen for 3 s, then disappear. Upon the shape disappearing, participants then had to reproduce the observed shape as accurately as possible. They drew their reproduction between two dots presented on the screen, and were instructed to starting drawing from the left and finish at the right (figure 1). The shapes were square waves of varying complexity: complexity was varied by altering height and/or width of the wave. For low complexity, the waves had the same height and width; for medium complexity, the waves differed in one dimension (either height or width); and for high complexity, the waves differed in both dimensions. Thus, as a shape's complexity increased, so the number of unique parameters (i.e. lengths of horizontal and vertical straight lines) needing to be stored in memory increased. We deliberately avoided the use of standard letter formations to ensure our results were not confounded by knowledge of letter shapes per se.

Figure 1.

Depiction of the VMM task. Sequence A depicts an example of a trial presenting a low-complexity shape (i.e. the two square-waves’ heights and widths are both equal). Sequence B is an example of a trial presenting a high-complexity shape (i.e. the two square-waves’ heights and widths both differ). Moving top to bottom within each sequence (i.e. following arrows) the time course of a trial was as follows: participants placed their stylus within a circle on an otherwise blank screen to commence trial; target shape was presented on-screen for 3 s; 1 s blank-screen interval followed; parallel ‘start’ and ‘end’ point dots appear on screen; participants completed trial by drawing their reproduction of the target shape from left to right between the dots.

There were 20 trials in total: the first two were practice trials and therefore not included in the final analyses. Children's baseline motor skills were measured via a copying task using an additional set of square waves where the shape remained in the top half of the screen while participants copied it in the bottom (i.e. no memory component). The copying task was always administered before the main task, with a short break between the two. Writing and reading scores (on a numerical scale) standardized against national norms were provided by the school. These scores were obtained from the national curriculum tests that every school in England is required to conduct on the children within their care (https://www.gov.uk/national-curriculum/overview). The national curriculum tests can be criticized on various grounds [27,28], including their inability to provide pure measures of reading or writing (e.g. the reading tests can also involve comprehension). But at a pragmatic level, these are the official government measures that indicate a child's general literacy level in school—measures that have been taken completely independently of our experimental study.

(d). Analysis

For each VMM trial, the accuracy with which participant's drawing (their input path) depicted the target shape (the reference path) was evaluated using the following procedure: point-sets were generated for the input and reference paths by discarding temporal information and resampling the x and y coordinates at a spatial resolution of 1 mm using linear interpolation. A robust point-registration method [29] was then used to determine the rigid transformation (consisting of translation, rotation and isotropic scaling components) that best transformed the input path to match the reference path. A metric, optimized error (OE), was then calculated to represent the ability to accurately reproduce the target shape by quantifying the congruence between input and target shapes. This was determined by evaluating the mean distance between corresponding points in the transformed input and reference path [25], and was thus independent of the scaling and rotation artefacts involved in the shape reproduction.

For statistical analysis, OE was taken as a measure of VMM and specified as the dependent variable in a mixed-measures ANOVA with age categorized by school year (year 2, year 4 and year 6) and shape complexity (low, medium and high) specified as independent variables. Correlational analyses were then conducted on the VMM task and writing and reading performance measurements, followed by a linear regression analysis.

3. Results

VMM (OE) was the dependent variable in a 3 (age) × 3 (shape complexity) mixed-measures ANOVA (α = 5%). There was a main effect of age (F_2,85 = 27.1, ; error decreasing with increasing age), and a main effect of shape complexity (F_2,170 = 166.6, ; error increasing with increasing shape complexity). Post-hoc analyses showed all age groups and all three levels of shape complexity differed significantly from each other (figure 2). The interaction between these main effects was not statistically significant.

Figure 2.

Bar-chart of OE by shape complexity and age. OE is a quantitative measure of the accuracy with which participants' drawings replicated the target shape. Larger OE values indicate lower accuracy of replication and are treated as an index of VMM. Age range, within year groups, is as follows: year 2, 6–7 years old; year 4, 8–9; year 6, 10–11. Statistically significant main effects for shape complexity and age and no statistically significant interaction between these two factors are observed. Error bars represent 95% confidence intervals.

In order to obtain an overall measure of each participant's VMM ability, a composite measure was obtained by calculating each participant's mean average OE score across the three levels of shape complexity. A partial correlation was run between VMM, writing and reading, controlling for age and baseline motor ability (i.e. copying: OE). VMM was correlated with writing (r = −0.42, p < 0.001) and reading (r = −0.32, p < 0.01), and writing and reading were correlated (r = 0.53, p < 0.001). A regression analysis was run with writing as the dependent variable. Age (in months) and copying were entered in step 1, and VMM entered in step 2. The model at step 1 was significant (F_1,87 = 157.7, p < 0.001). The model at step 2 made an additional significant contribution (ΔR² = 0.06, p < 0.001), with VMM a unique predictor of writing (β = −0.31, t₈₇ = −4.22, p < 0.001). The same hierarchical regression was run with reading as the dependent variable. The model at step 2 made an additional significant contribution (ΔR² = 0.04, p < 0.01), and VMM was again a unique and significant predictor (β = −0.21, t₈₇ = −3.10, p < 0.01). To test whether writing mediated this relationship, a second regression analysis was conducted where writing was also entered in step 1, and VMM in step 2. The model at step 2 no longer made an additional contribution (ΔR² = 0.003, p > 0.05). The Sobel test [30] confirmed that the indirect effect of VMM on reading via writing was significant (z = −3.39, p < 0.001).

4. Discussion

We have successfully developed an objective technique for studying VMM, and have found evidence in support of the notion that this cognitive process is an important construct underpinning both handwriting and reading ability in children. VMM provides a plausible cognitive pathway through which the motor aspects of handwriting can become more automated, reducing the cognitive load of the procedural aspects of this activity and freeing resources for the development of higher-order language skills [4]. This proposal is supported by the indirect effect of VMM on academic reading scores (through its relationship with academic writing scores) and is consistent with previous evidence of motor representations of letters reinforcing visual letter recognition in children [8–10].

The validity of the VMM measures is corroborated by the fact that the results show principled alterations in response to age and shape complexity, whereby increased age positively affects VMM but increased memory demands (linked to increasing shape complexity) have a negative effect (with no interaction between age and complexity). Further support for VMM as a meaningful construct can be found in the wider literature. For example, it is known that visual memory skills predict the abilities of individuals who use drawing in a professional capacity to communicate ideas—such as college students of art [31] and technical drawing [32]. It is logical to suggest that the role for visual memory in drawing is analogous to the one we have identified for VMM within handwriting. Artists and draughtsmen rely on memory to represent often encountered patterns/angles (e.g. when constructing compositions or laying out schematics), and thus the automaticity with which they can access such representations will doubtless have a bearing on their drawing's quality. On this basis, we propose that VMM is a core cognitive ability that influences the ability to use any form of communication via visual symbols.

The concept of VMM is also consistent with current theories on the embodied nature of cognition [5] in that basic perceptual and motor control processes must inform the development of higher-order cognitive abilities, such as communication skills. Nonetheless, further empirical investigation is required: longitudinal research looking at whether rate of language acquisition (writing and reading) is mediated by VMM ability would help increase our understanding of the degree to which VMM contributes to writing and reading development. In addition, research across a wider age range might be expected to find the strength of the relationship between VMM and language abilities varying with time. Specifically, once automaticity of handwriting rises above a certain threshold it might be expected that the relative contribution of VMM to wider language ability will diminish [3].

This research opens up the exciting possibility of identifying children at risk of problems in the domains of reading and writing, given that children who performed less well on the VMM task were more likely to have lower scores on national school-based writing and reading tests. Empirical evidence already suggests that in pre-school children nascent handwriting ability is associated with concurrent levels of emergent literacy skill [33,34], indexed by letter identification and word decoding abilities. Meanwhile, within schools there is evidence of a link between the automaticity of children's handwriting (between 7 and 11 years old) and the quality of composition within written work [35,36]. If VMM is the cognitive process within which the critical shift from effortful to proceduralized/automatic production of letters occurs [3] then we have identified a key cognitive component that underpins the early stages of written language acquisition. It should be noted that our data are correlational and therefore we cannot address the issue of causality. It is reasonable to suggest that reduced VMM capabilities might have a negative impact on reading and writing skill, but it is also equally plausible that poor reading and writing skills (produced by genetic and/or environmental factors) cause reduced VMM capabilities. In reality, it is probable that these factors form a complex dynamical system where their development is mutually dependent. This viewpoint is consistent with recent theories which suggest that multiple deficits contribute to complex disorders such as dyslexia [37].

In summary, we have presented a new method for exploring the factors that contribute to the successful formation and use of a visual–motor code in memory. We have used this method to investigate a hypothesized cognitive construct (VMM) that we believe is a central component facilitating handwriting and wider literacy development. This sheds light on an important cognitive process that underpins one of the unique evolutionary advantages possessed by humans—the ability to learn and use complex writing systems in order to store and disseminate information [1].

Ethics statement

The school obtained informed consent from the individual parents/guardians of the children, giving them advanced notice of the study and their right to opt out should they wish to do so. The experimenters then obtained informed written consent from the head teacher of the participating schools (acting in loco parentis for their students) for the participating children. We obtained verbal consent from the children after explaining the study to them. The children gave their verbal consent immediately prior to participating, as written consent was impractical for a number of the children within this age range. This consent procedure and all other aspects of the study received ethical approval from the University of Leeds Research Ethics Committee, and this research was carried out in accordance with the provisions of the World Medical Association Declaration of Helsinki.

Data accessibility

An anonymized version of the dataset reported in this article is available from the Dryad digital data repository (http://datadryad.org), doi:10.5061/dryad.t3g5j.

Funding statement

This work was supported by funding from the National Institute for Health Research Collaboration for Leadership in Applied Health Research and Care (NIHR CLAHRC) for LYB (Leeds, York and Bradford).