Assessing the readability of the self-reported Strengths and Difficulties Questionnaire

Abstract

The Strengths and Difficulties Questionnaire (SDQ) is one of the most widely used measures in child and adolescent mental health in clinical practice, community-based screening and research. Assessing the readability of such questionnaires is important as young people may not comprehend items above their reading ability when self-reporting. Analyses of readability in the present study indicate that the self-report SDQ might not be suitable for young people with a reading age below 13–14 years and highlight differences in readability between subscales. The findings suggest a need for caution in using the SDQ as a self-report measure for children below the age of 13, and highlight considerations of readability in measure development, selection and interpretation.

Declaration of interest

None.

Self-reported perspectives of mental health symptoms are being used more widely in screening, research and practice in child and adolescent mental health. ^{1 – 3} Measures have been developed and tested for use by young people to report on their mental health symptoms, and several such measures that are well validated in terms of psychometric properties exist. ⁴ Measure development has become increasingly sophisticated, and a range of psychometric and other properties of questionnaires are assessed before deciding they are fit for purpose. ^{5 , 6} However, while observing the criteria used to assess measure suitability, ⁶ we noted that a key criterion that is specifically relevant to child measures, but is routinely not accounted for, is the readability of the items in the scale. Readability has been defined in many ways, but in simple terms refers to the ease with which a reader can read and understand text. ^{7 , 8} The concern regarding the readability of psychological questionnaires is not new, with the readability of some adult psychopathology questionnaires ^{9 , 10} and some measures of child psychopathology, as reported by parents and children, having been assessed for their readability. ¹¹ However, the latter investigation does not include a readability assessment of one of the most widely used child mental health measures in the UK, the Strengths and Difficulties Questionnaire (SDQ). The self-report SDQ is used extensively in the UK in research, community settings and clinical practice ^{12 , 13} and is available to complete in different modes. ¹⁴ The developers recommend this self-report measure for young people aged 11–17 years. ¹⁵ However, it has since been psychometrically validated for use in younger children, including from 8 years ¹⁶ and as young as 6 years. ¹⁷ In this short paper, we investigate the reading age suitability of the self-report SDQ.

Method

Strengths and Difficulties Questionnaire

The SDQ is a 25-item questionnaire that comprises five five-item subscales: emotional symptoms, conduct problems, hyperactivity, peer problems and prosocial behaviour. ¹⁸ Participants respond to each item by selecting one of three responses: not true, somewhat true and certainly true.

Readability

In the current study, we used four standard methods that can be used to examine the readability of text.

Flesch–Kincaid reading grade (FK)

This method, ¹⁹ adapted from the Flesch Reading Ease score, ¹⁹ is one of the oldest and most widely used readability indices, and is based on average numbers of syllables per word and sentence lengths. Scores are estimated as a US grade level.

where ASL = average words per sentence; ASW = average syllables per word

Gunning Fog Index (GFI)

The GFI corresponds to the number of years of formal education required to understand text ²⁰ and uses the numbers of words, sentences and complex words, which are defined as having three of more syllables.

where A = number of words; N = number of sentences; L = number of words with three or more syllables (excluding -ing and -ed ends)

Coleman Liau Index (CLI)

The CLI ²¹ differs from the FK and GFI tests by focusing on the number of letters (rather than syllables) per word. It also results in a US grade-level score. The formula for estimating the CLI is:

where L = average letters/100 words; S = average sentences/100 words

Dale–Chall Readability Formula (DC)

The DC differs from the previous three indices by incorporating the level of difficulty of the words in the text into the formula for its estimation. ^{7 , 22} A list of words that up to 80% of fourth graders (children aged around 10) know is used as the basis for identifying words that can be considered difficult.

where DW = difficult words (i.e. words not in the list), A = number of words, ASL = average words per sentence

These indices mainly estimate readability as a US grade-level score. Grade levels can be translated to age levels by adding 6 to a grade-level score (children in US grade 1 are aged 6–7 years), and this was done in this study. The four methods were chosen not only because they are well established and widely used measures of readability, but also because they have differences in focus when estimating readability, which can lead to varying readability estimates. The unique elements as part of their estimation include that the FK focuses on syllables in words, the GFI on words with more than three syllables, the CLI on the number of letters per word and the DC on the presence of difficult words.

Procedure

The readability formulae described above were applied to the instruction section and the items of the SDQ (for the full set of items in the measure and for each of the subscales separately), resulting in four readability estimates for each of the examined components (Table 1). The estimates across the four readability estimates were averaged to provide a single readability estimate.

Table 1.

Readability estimates (in years) made using the four different approaches and the average estimate for the full measure, instructions and subscales of the Strengths and Difficulties Questionnaire

	FK	GFI	CLI	DC	Average readability age across the four indices
Full measure	10.94	12.74	11.63	11.69	11.75
Instructions	12.88	15.22	13.33	12.22	13.41
Subscales
Conduct disorder	9.40	10.74	11.27	10.44	10.46
Emotional symptoms	12.01	14.30	16.16	12.92	13.85
Hyperactivity	11.16	14.16	15.79	13.12	13.56
Peer problems	10.82	12.43	12.38	11.69	11.83
Prosocial behaviour	11.85	12.43	16.38	10.77	12.84

FK, Flesch–Kincaid reading grade; GFI, Gunning Fog Index; CLI, Coleman Liau Index; DC, Dale–Chall Readability Formula.

Results

For the full measure, age estimates for readability ranged from 10.94 to 12.74, with a mean estimate of 11.75 years (Table 1). For the subscales, the conduct problems subscale had the lowest mean readability age of 10.46 years, followed by peer problems (M = 11.83), prosocial behaviour (M = 12.84), hyperactivity (M = 13.56), with the emotional symptoms subscale (M = 13.85) having the highest average readability age. For the instructions, the average readability estimate was 13.41 years.

Discussion

The results indicate that while some of the SDQ subscales have a readability of around 11–12 years (peer and conduct problems), the instructions and some of the subscales (notably emotional difficulties and hyperactivity) have average readability estimates that are substantially higher (ranging up to 13.9 years). On the basis of these readability estimates, the SDQ would be considered unsuitable for 6- and 8-year-olds (despite psychometric validation studies at these ages) ^{16 , 17} . Moreover, these findings suggest that it might be difficult to understand overall for 11-year-olds (the recommended starting age for this measure). This difficulty may be further compounded when young people have lower reading ages relative to their developmental age. This might be of particular relevance in clinical settings, given that many children with mental health difficulties also have learning difficulties and special educational needs. ²³ It is important to note that this issue of unsuitably high readability is not unique to this self-report measure; for example, the Youth Self Report version of the Achenbach assessments of child mental health has a readability estimate of 12.5 years, ¹¹ although, as with the SDQ, it is meant to be suitable from age 11 years onwards.

We present the results from a range of readability indices to highlight the variation in the age estimates they provide. The inclusion of the DC is especially relevant here, as it is the only index to include word complexity in its estimate, an aspect that provides insight into possible difficulty in understanding the specific content of the items. Previous attempts to map the readability of psychopathology measures ⁹ have been criticised for not taking word complexity into account in their estimates. ²⁴ In this example, the hyperactivity and emotional symptoms subscales have eight and seven difficult words, respectively (e.g. squirming, fidgeting, down-hearted). This highlights the importance of also considering specific words and their suitability for the age group in question when designing questionnaires. These findings raise issues for interpretation of the SDQ self-report data derived from younger children. If younger children with a reading age lower than around 13 who are completing the SDQ are not comprehending items, key words or instructions, this may affect their responses and subsequently the derived scores used in analysis or to inform treatment.

Psychometric properties are not the only criteria that determine whether or not a measure is fit for purpose. There is a crucial prior step: assessing whether the target population can read and understand the items in a questionnaire comfortably. In this paper we describe and apply a non-resource-intense approach to assess the content of measures using four standardised measures of readability. Estimates of readability provide an overarching view of the complexity of the language used in a questionnaire and help identify words that might be difficult to understand. In addition, more intensive approaches to investigating measure comprehension, such as cognitive interviewing, can help illuminate how items are understood and interpreted by respondents. ²⁵

Advice in relation to measures and health-related materials for adults is that they should have a readability of around 12 years, although the average reading age of adults is around 14 years. ²⁶ Extrapolating from this advice, we recommend that child self-report measures should aim for a readability age of around 2 years below the target minimum age for the measure.

The implications of these findings for the use of the SDQ include reconsidering the age of the target group for the questionnaire, developing accompanying support materials and explanations to aid completion of the survey or developing alternative questionnaires with a lower readability age. The wider implications of the results are that they highlight that alongside psychometric properties, a key consideration for the selection and reliable use of any self-report measure with children (or adults) should be: can the target user understand this?

Ethical approval

Given that no data from human participants were used in this report, no ethical approvals were necessary.