Teachers’ Perceptions of Handwriting Legibility Versus the German Systematic Screening for Motoric-Handwriting Difficulties (SEMS)

Abstract

Handwriting problems impact school achievement. The purpose of this pilot study was to evaluate the agreement between teachers’ opinions and the German Systematische Erfassung motorischer Schreibstörungen’s (SEMS) handwriting test and to estimate the prevalence of handwriting problems. This was a prospective, correlational study. Teachers’ ratings of handwriting from German children (Grades 2 and 4) were compared to SEMS scores. The agreement was calculated with Gwet’s AC2; preliminary cutoffs and prevalence were assessed with receiver–operating characteristic curves. SEMS accurately identified Grade 2 children with handwriting problems (specificity = 98.21%, sensitivity = 100%) but fewer Grade 4 children with handwriting difficulties than did teachers (specificity 97.67%, sensitivity 28.57%). Grade 4 item agreement supports the SEMS’ ability to assess underlying handwriting dimensions as perceived by teachers. Teachers identified 12.70% of Grade 2 children versus SEMS 11.11% and 14% of Grade 4 children versus SEMS 6%. The SEMS supports clinical observations and provides information about underlying handwriting dimensions. Because handwriting is used primarily at school, the opinions of teachers are also critical.

Introduction

Despite increasing computer use, handwriting remains a critical life skill supporting literacy and cognitive development in children (Mangen & Balsvik, 2016; Vinci-Booher et al., 2016) and learning in adult educational settings (Mueller & Oppenheimer, 2014). Dysfunctional handwriting prevalence ranges from 27% to 35% in Grades 1 to 2 and decreases to 6 to 13% in higher grades (Karlsdottir & Stefansson, 2002; Overvelde & Hulstijn, 2011). Given the influence of handwriting on personal well-being, learning, and academic success (Feder & Majnemer, 2007; Graham et al., 1997; Vinci-Booher et al., 2016), it is not surprising that children with difficulties are commonly referred for pediatric occupational or physical therapy (Feder et al., 2000; Bosga-Stork et al., 2011, as cited in van Waelvelde et al., 2012).

Efficient handwriting is characterized by two components: legibility and speed (Feder & Majnemer, 2007). Legibility depends upon accepted standards of letter formation (e.g., no added strokes), size, placement, and spacing (Graham & Weintraub, 1996; Stefansson & Karlsdottir, 2003). However, varying demands of different types of writing tasks themselves influence legibility. For instance, copying a text versus writing an essay have inherently different impacts on handwriting due to the varying demands of spelling, language skills, composition, and visual memory, all of which influence letter production (formation) and placement (Graham et al., 2006; Kandel et al., 2017). Speed can be defined as letters per minute or a student’s ability to keep up with written tasks in class. Handwriting is primarily an in-school activity. Hence, it is important to know whether evaluation via handwriting screening tests differs from teacher opinions and, if so, how.

The Systematic Screening for Motoric-Handwriting Difficulties (Systematische Erfassung motorischer Schreib-störungen, SEMS; Vinçon et al., 2015) is the culturally translated version of the revised Dutch Systematic Screening for Handwriting Difficulties-2 (SOS-2). The SEMS has adapted the text to be copied to a German-language text including all letters of the alphabet. As a screening test, the SEMS helps to confirm suspected handwriting problems before referring students for therapy (van Waelvelde et al., 2012). The SEMS showed strong validity in discriminating between the handwriting of students with and without motor difficulties (Vinçon et al., 2015). Using a near-copying task, the SEMS evaluates handwriting legibility and speed. Legibility is assessed by evaluation of letter formation, letter fluency, letter size, regularity of letter size, spacing between words, alignment, and joins (for cursive writing). Speed is evaluated as letters written within 5 min.

The SEMS was designed to help school therapists and teachers identify students in need of handwriting remediation. In this study, ecological validity examines how well a handwriting-screening test administered by a school occupational therapist compares to handwriting performance assessed in everyday school activities by the student’s classroom teachers, that is, the degree to which the SEMS test scores correspond to real-world performance (Vinçon et al., 2017). Because teachers expect therapists to help remediate handwriting problems in their students (Hammerschmidt & Sudsawad, 2004), it is critical to understand how well screening tests typically administered by therapists concur with teachers’ informal assessments of students’ handwriting used in school.

The primary purpose of this pilot study was to evaluate the SEMS’ ecological validity for students in Bavaria, Germany, Grades 2 and 4, with three objectives: (a) To determine the agreement between SEMS’ item scores and teachers’ ratings of underlying components of legible handwriting; (b) to examine whether the SEMS and teachers identify the same students as having handwriting problems; and (3) to estimate the prevalence of handwriting difficulties for students in Grades 2 and 4.

Method

Design

This prospective pilot study was conducted in the School District Lindau in Bavaria, Germany.

Ethical Approval

The University of British Columbia Behavioural Ethics Committee and the Governmental Educational Authority in the Lindau school district approved the study.

Participants

Grade 2 and Grade 4 students were chosen for several reasons. Adequate proficiency can only be expected with enough practice (Duiser et al., 2020), with handwriting stability expected within the second half of Grade 3 (Overvelde & Hulstijn, 2011). Furthermore, in Bavaria, teachers accompany their classes for 2 years: Grades 1 to 2 and then Grades 3 to 4. Hence, teachers in Grades 2 and 4 have extensive knowledge about the handwriting skills of students in their classes. Within-district schools were chosen by convenience, aiming to recruit at least 50 students per grade level. To enhance heterogeneity, all students within the invited classes were eligible to participate. Written consent was obtained from teachers and parents of potential participants. Students assented verbally on the testing day.

All Grade 2 teachers (n = 4) consented to participate. These teachers taught a total of 80 students, all who were invited; 13 parents declined, three forms were not returned, and one student was absent for testing, resulting in 63 Grade 2 students. All Grade 4 teachers (n = 4) consented to participate. These teachers taught a total of 72 students, all of whom were invited; 10 parents declined, seven forms were not returned, and one student was absent for testing. Results from four Grade 4 students were excluded due to writing a different script style than used in school and as evaluated by their teachers, resulting in 50 valid Grade 4 handwriting samples. Demographics are shown in Table 1.

Table 1.

Demographics of Participants.

Grade	Total	Girls	Boys	Left-handed	Right-handed	Mean age	Age SD
2	N = 63	35	28	5	58	7.6	0.55
4	N = 50	22	28	6	44	9.6	0.45

Note. SD = standard deviation.

Measures

SEMS

The SEMS assesses handwriting legibility and speed. Students have 5 min to copy a standardized German printed text, placed directly next to them, including all letters of the alphabet. Students copy the text onto unlined paper, using their usual pen/pencil, and are told to write as they usually do. If a student is unable to completely copy the first five lines of the 21-line text within 5 min, a mark is made to indicate how much text was completed. After that, the student receives more time to finish the first five lines, which are the minimum required to score legibility. As writing illegible letters also takes time, all letters produced within the 5 minare counted.

Legibility is assessed with seven criteria as described in Table 2. Each item is scored as 0, 1, or 2 points with scoring criteria provided. Scores are summed for an overall legibility score; higher scores represent poorer legibility. Calculated separately, handwriting speed equals the number of letters, regardless of accuracy, written per 5 min. The SEMS’ publication is pending. It is designed for either individual or group administration.

Table 2.

SEMS Legibility Criteria.

Letter formation	Differences in formation when compared to the learned script, for example, additional strokes, missing strokes, breaks, or spaces within the letter that cannot be attributed to individual style.
Fluency of writing	Undesirable directional changes, corners, jags, or breaks within the letters.
Letter size	Average letter size, rated per line.
Regularity of letter size	Height ratio between the smallest and largest letters on a line as calculated with a template using the students’ average letter size.
Spacing between words	Between-word spacing of at least an average letter “o” from that student.
Sentence alignment	Maximal allowed vertical variance of the letters on an imaginary straight line, as measured with a template based on the student’s average letter size.
Correct joins (cursive only)	Between two joined letters, a continuous line without changes in direction, undesirable loops, bends or breaks, that could be otherwise attributed to an individual style.

Note. SEMS = Systematische Erfassung motorischer Schreibstörungen.

Teacher evaluation

A teacher questionnaire was developed for this pilot project (online Supplement A), paralleling the SEMS’ scoring and focus. SEMS marking criteria for legibility items and speed were rephrased to apply to general handwriting skills rather than observation of one handwriting sample. As with the SEMS, scores are summed for an overall legibility score; higher scores represent poorer legibility. Teachers differentiated handwriting speed into three categories: score 0 for students who are mostly fast enough; score 1 for students who are sometimes fast enough; and score 2 for students who are rarely fast enough. Teachers were also asked to indicate (yes/no) if the student had handwriting difficulties. Two teachers (each with ≥10 years’ experience) provided feedback to ensure feasibility and comprehensibility on this questionnaire. No changes were suggested. At the time of the study, classroom teachers completed one questionnaire per student from their class, based on each student’s general handwriting performance.

Procedures

The SEMS primary author instructed the primary investigator on administration and scoring. Five handwriting samples were used to assess percentage agreement between the two raters and were 40% for total scores and 63.6% for item scores, whence the SEMS’ author further instructed the investigator. After additional practice, reevaluation of percentage agreement using the five initial handwriting samples was conducted. Percentage agreement for total scores was 80% and for item scores 81.8%.

The SEMS was administered and scored by the first author. Students were tested in the classroom simultaneously with their teacher present. Students without parental consent worked quietly at their desks. Classroom teachers rated each of their student’s typical handwriting. The first author and teachers were blind to one another’s scoring.

Data Analysis

Because Grade 2 students printed and Grade 4 students wrote cursive, the grades were analyzed separately. A significance level of .05 was predetermined. To evaluate ecological validity, each teacher rated the general handwriting of each student in his or her own class with those ratings compared to results on the SEMS as administered by the school occupational therapist.

Item-level agreement

As a screening tool, the SEMS attempts to categorize students into groups, those with and without handwriting difficulties. At the item level, the test assesses the presence, partial presence, or absence of a trait. Here the absolute mathematical similarity of scores between testers is important (Kim, 2013). Hence, at the item level, the analysis focused on the agreement between SEMS scores and teachers’ ratings, measuring how often the teachers and the SEMS scored students exactly the same. Cohen’s kappa is a popular measure of the agreement but sensitive to bias and prevalence of a condition, giving paradoxical results when prevalence is high or low (Wongpakaran et al., 2013). Prevalence of handwriting difficulties ranges from 6 to 35% (Karlsdottir & Stefansson, 2002; Overvelde & Hulstijn, 2011). However, the prevalence of difficulties at the item level is probably lower as not all students will have problems with all items used to identify handwriting difficulties.

A suggested alternative is Gwet’s alternative coefficient AC₁, a chance-corrected agreement coefficient that is more stable and less affected by prevalence and bias than kappa (Gwet, 2016; Wongpakaran et al., 2013). Gwet’s AC₂ is a linear weighted version for use with ordinal, interval, and ratio data. Interpretation of AC₂ in this study was performed with Altman’s benchmark (Gwet, 2014).

Therapy indications and prevalence

Comparisons between SEMS total scores and students identified by teachers as having handwriting difficulties appear in histograms (Figures 1 and 2). Receiver–operating characteristic (ROC) curves were used to define optimal cutoff points for identifying students with handwriting difficulties and to calculate preliminary prevalence values based on the SEMS. Values for sensitivity, specificity, positive predictive values (PPV), and negative predictive values (NPV) were calculated but should be viewed cautiously, as the required sample size was not calculated a priori and teacher evaluation of handwriting is considered to be subjective (Rosenblum, 2005).

Figure 1.

SEMS scores and teachers’ opinions of handwriting difficulties for Grade 2.

Note. SEMS scores (x-axis) compared to teacher opinion (yes / no) of handwriting difficulties per student. Frequency (on the y-axis) indicates number of students identified by teachers with (yes) or without (no) handwriting difficulties for each SEMS score (on the x-axis). Teacher response “no”: n = 56. SEMS total score: M = 2.93; SD = 1.68. Teacher response “yes”: n = 7. SEMS total score: M = 7.29; SD = 1.11. SEMS = Systematische Erfassung motorischer Schreibstörungen; SD = standard deviation.

Figure 2.

SEMS scores and teachers’ opinion of handwriting difficulties for Grade 4.

Note. SEMS scores (x-axis) compared to teacher’ opinions (yes / no) of handwriting difficulties per student. Frequency (on the y-axis) indicates the number of students identified by teachers with (yes) or without (no) handwriting difficulties for each SEMS score (on the x-axis). Teacher response “no”: n = 43. SEMS total score: M = 1.84; SD = 1.40.

Teacher response “yes” n = 7. SEMS total score: M = 4.14; SD = 2.41. SEMS = Systematische Erfassung motorischer Schreibstörungen; SD = standard deviation.

Results

Grade 2

Item-level agreement

Analyzing weighted averages for classrooms (Table 3), Items 1 (letter formation) and 6 (spacing) demonstrated good agreement between SEMS’ and teachers’ ratings. Items 2 (fluency), 4 (writing size), and 7 (alignment) showed moderate agreement, whereas Item 5 (regularity of size) demonstrated only fair agreement (see online Supplement B to view Gwet’s AC₂ for Grade 2 classrooms).

Table 3.

Item-Level Agreement SEMS Scores and Teachers’ Ratings.

		Grade 2	Grade 4
Item		AC2	AC2
1	Formation	0.66***	0.52**
2	Fluency	0.52**	0.77***
3	Joins (Cursive)	—	0.71***
4	Writing size	0.44***	0.79***
5	Size regularity	0.34	0.73**
6	Spacing	0.63**	0.88***
7	Alignment	0.50***	0.69**

Note. AC₂ = Gwet’s AC2 correlation coefficient. N = 63 Grade 2 students. N = 50 Grade 4 students. SEMS = Systematische Erfassung motorischer Schreibstörungen.

^*p < .05. **p < .01. ***p < .001.

Therapy indications

Grade 2 students identified by teachers as having handwriting problems tended to score >6 points on the SEMS, whereas students without problems scored <5 points (Figure 1). Only one student who scored 6 on the SEMS was identified by the teacher as not having handwriting problems; otherwise, all students scoring ≥6 were identified by their teachers as having difficulties. The ROC calculations identified an optimal SEMS cutoff score of 5.5; this was rounded up to 6 because only whole numbers are possible in scoring (95% confidence interval [CI], sensitivity 100.00%, specificity 98.21%, PPV is 87.50%, NPV 100.00%).

Prevalence

Prevalence of handwriting difficulty based on SEMS evaluation was 12.70%, whereas teachers identified 11.11% of students as having problems.

Grade 4

Item-level agreement

Weighted averages for classrooms are shown in Table 3. Item 1 (formation) showed moderate agreement between SEMS scores and teacher ratings. Good agreement was shown for Items 2 (fluency), 3 (joins), 4 (writing size), 5 (regularity of writing size), and 7 (alignment). Item 6 (spacing) demonstrated very good agreement. (See online Supplement C to view Gwet’s AC₂ for Grade 4 classrooms.)

Therapy indications

Figure 2 shows that Grade 4 students identified with handwriting problems by their teachers attained a wide range of SEMS scores. ROC calculations identified a cutoff score of 1.5 (95% CI, sensitivity 100%, specificity 39.53%, PPV 21.21%, NPV 100%). Although sensitivity was optimal, specificity and PPV were poor. Using this cutoff, numerous students would be falsely identified as having problems. Tests that assess serious illnesses tend to focus on sensitivity and NPV to ensure that all positive cases are identified and only true negatives ruled out, sometimes risking numerous false positives (de Vet et al., 2011). For the SEMS, the focus should be on specificity and NPV so as not to refer numerous students without handwriting difficulties to therapy. A cutoff of 5 demonstrated better specificity (97.67%) and PPV (66.67%), albeit at the cost of sensitivity (28.57%). A cutoff score of 5 is clinically appropriate, with the expectation that students in higher grades should achieve lower (i.e., better) scores.

Prevalence

The prevalence based on SEMS evaluation was 6%, whereas teachers identified 14% of students as having handwriting problems.

Discussion

Grade 2

Writing on blank paper for the SEMS, Grade 2 students printed with increasing size and spacing, often with difficulty in orientation. At school, students write on lined paper, supporting letter placement and sizing. Thus, it is not surprising that writing size, alignment, and regularity of writing size demonstrated only fair to moderate agreement (Table 3) as teachers rated writing used in school (on lined paper), whereas the SEMS rated writing on blank paper. The SEMS’ use of blank paper assesses students’ ability to size and place letters without external support. The good agreement for spacing could be because the SEMS only negatively rates words that are too close. The good agreement between teachers’ ratings and SEMS’ scoring of correct letter formation (Table 3) demonstrates that both rated students’ knowledge of letter formation similarly.

The SEMS was developed to identify students with handwriting difficulties. Looking at Figure 1, it is evident that, except for one student, all with scores of ≥6 were identified as having difficulties. With a SEMS cutoff score of 6, only 1 of the 63 Grade 2 students would be incorrectly identified. At this cutoff level, the SEMS has a specificity of 98.21% and a sensitivity of 100%, indicating with relative certainty that students with a score <6 do not have handwriting problems, and students with a score of ≥6 do have handwriting problems. Knowledge of letter formation contributes greatly to writing performance (Rodriguez & Villarroel, 2017). The SEMS appears valid in identifying Grade 2 pupils with or without handwriting problems as perceived by teachers.

The similarity in calculated prevalence rates between teacher opinion (11.11%) and the SEMS evaluation (12.70%) also supports the SEMS’ ecological validity in identifying Grade 2 students with handwriting difficulties as perceived by their teachers. These rates match well with those by Overvelde and Hulstijn (2011) and fall within the low range from Karlsdottir and Stefansson (2002).

Grade 4

The SEMS failed to identify several students with handwriting problems as classified by their teachers (Figure 2). Handwriting is variable (Graham, 1986), affected by energy level, attention, interest, and so on. However, identification errors were such that teachers identified more students with handwriting problems than the SEMS (only one student was identified by the SEMS and not by their teacher). The length of writing task the SEMS assesses may be of issue. The SEMS requires students to write for only 5 min and difficulties might occur with increased writing time at higher grades. Grade 4 students are expected to write more, longer, and compose more complex texts than those in Grade 2. For example, van Waelvelde and colleagues (2012) identified the 5-minute duration and use of only a copying task as limiting factors of the SOS (the original Dutch version).

The SEMS assesses only one handwriting task: copying. Handwriting, however, depends not only on its underlying dimensions but also on the activity of writing, whereas spelling (Kandel et al., 2017), composition, and visual memory impact the production and placement of letters (Graham et al., 2006). For example, composing a lengthy text involves greater demands on spelling, language use, and composition than a copying task. In fact, Graham and colleagues (2006) found that the difference in handwriting was largest between copying and composition, influencing both production and placement of letters.

The good item-level agreement for fluency, letter joins, writing size, regularity of writing size, and alignment as well as a very good agreement for spacing shows (Table 3) that the SEMS and teachers rated the underlying legibility dimensions similar to teachers. However, at the Grade 4 level, SEMS’ scores and teachers’ ratings failed to agree as to which students actually had handwriting problems. It appears that although teachers recognized that their students were able to execute the specific components required for legible handwriting, the students failed to consistently do so and, in their teachers’ opinions, wrote illegibly. By using a near-copying task, the influence of factors such as composition, spelling, and visual memory (Graham et al., 2006) is reduced. Worth noting is that all but one pupil identified by the SEMS as having handwriting difficulties in Grade 4 was identified also by teachers as having handwriting concerns. However, teachers identified five additional students not identified by the SEMS (Figure 2). Possibly the extra students identified by teachers have handwriting problems due to increasing demands of composition, spelling, or visual memory, which interfere with letter production and placement (Graham et al., 2006). This supposition is supported by the fact that SEMS showed strong validity in discriminating between the handwriting of students with and without motor difficulties (Vinçon et al., 2015), that is, indicating that it can identify students with handwriting difficulties due to motor problems. The SEMS appears valid in assessing underlying handwriting dimensions but may fail to identify students with problems when demands of the task are longer than 5 min or more complex than copying. This difference is also indicated in the values for specificity 97.67% and sensitivity 28.57% (cutoff of 5).

The use of only a near-copying task enables the SEMS, with relative certainty, to rule out handwriting problems but is less accurate in identifying students with handwriting problems as viewed by teachers. Students however do not use graphomotor skills only to copy texts. With increasing grade levels, complexity and length of handwriting tasks increase, and teachers assess handwriting used in all these different school activities. The exclusive use of a near-copying task appears sufficient to correctly identify students in Grade 2 but reduces the sensitivity in Grade 4. Although teachers know that letter formation, spacing, and writing on the lines are important components of legibility, they identify handwriting problems based on their ability to read students’ handwriting (Hammerschmidt & Sudsawad, 2004). Thus, when teachers identify Grade 4 students with handwriting difficulties, they may also be identifying those who display handwriting difficulties not due to inherent problems with the underlying dimensions of legibility but rather due to factors within the writing task that make the execution of these underlying dimensions more difficult. To improve the sensitivity of the SEMS to identify all Grade 4 students with handwriting problems as perceived by teachers, additional handwriting tasks would also need to be assessed.

At Grade 4, the prevalence rate calculated on SEMS identification (6%) was less than half than that identified by teachers (14%). Although the 6% SEMS’ rate matches well with findings by Overvelde and Hulstijn (2011), this would be expected as the SEMS is based on the SOS-2, the short form of the test used in that study. The prevalence rate in our pilot project based on teacher evaluation is closer to that found by Karlsdottir and Stefansson (2002) with Grade 5 students (13%). The SEMS’ use of only a 5-min copying task may lead to lower prevalence rates than if a variety of handwriting tasks and/or speed were included in estimates of handwriting difficulty.

Study Limitations

The generalizability of our results is limited by the lack of random selection and small class sizes. The relatively high nonconsent rate and exclusion of four Grade 4 handwriting samples may have influenced our prevalence findings. Our failure to assess either interrater or intrarater reliability for the SEMS scores or teachers’ ratings is another shortcoming. Finally, as the required sample size was not calculated a priori, cutoff scores and values of sensitivity, specificity, PPV, and NPV should be viewed cautiously (Thomas et al., 2015).

Conclusion

In this prospective, correlational study, we examined how well a handwriting-screening test administered by a school occupational therapist compared to handwriting performance assessed in everyday school activities by the students’ classroom teachers. In Grade 2, the SEMS and teachers identified the same students as having legibility difficulties. Handwriting demands at Grade 2 are less difficult than at Grade 4, and the SEMS’ exclusive use of a copying task appears valid for identifying Grade 2 students with handwriting difficulties as perceived by teachers. Consequently, either the SEMS or the teacher report could be used in clinical practice to screen for handwriting difficulties in Grade 2 students. Teachers unsure of their own rating of students’ handwriting skills could, with training, use the SEMS as a screen to support their observations, and therapists can be assured that clinical application of the SEMS does represent teacher opinion at the Grade 2 level.

The SEMS had poor specificity and PPV (<80%) when compared to teachers’ ratings in Grade 4 students, that is, two of the four indicators did not support the test’s ecological validity for use in Grade 4. However, the good and very good agreement for six of the seven dimensions of legibility in Grade 4 lend support to the SEMS’ ability to identify Grade 4 students with handwriting problems due to the underlying dimensions of legibility, letter formation, size, placement, and spacing (Graham & Weintraub, 1996; Stefansson & Karlsdottir, 2003). At this grade level, with training, either teachers or school therapists could use the SEMS to screen students for possible handwriting problems due to difficulties in the underlying dimensions of handwriting. Future studies could examine whether the SEMS is ecologically valid for use with higher grade students, that is, Grade 5 and above.