Are Patient-reported Outcome Measures in Orthopaedics Easily Read by Patients?

Abstract

Background

Patient-reported outcome measures (PROMs) are commonly used by healthcare providers as means of assessing health-related quality of life and function at any given time. The complexity of PROMs can differ and when combined with varying degrees of adult literacy, error can be introduced if patients fail to understand questions. With an average adult literacy level of 11-year-old students in the United Kingdom, it is unclear to what degree PROMs can be read and understood by most patients (readability); to our knowledge, this has not been evaluated.

Questions/purposes

We wished to determine the readability of commonly used PROMs in orthopaedic surgery, as assessed by a validated tool that measures the complexity of the language in these surveys.

Methods

We performed a MEDLINE search to identify the most-commonly reported PROMs in orthopaedic research. One hundred twenty-one PROMs were identified and reviewed by 19 attending orthopaedic surgeons at our institution. Fifty-nine were selected as the most commonly used in our department. Of these, 52 (78%) were disease specific and included: 12 (20%) knee, 10 (17%) shoulder, seven (12%) spine, six (10%) hip, five (8%) foot and ankle, four (7%) elbow, three (5%) pelvis, three (5%) hand and wrist, and two (3%) lower limb. The remaining seven (12%) PROMs were general health questionnaires. The Flesch Reading Ease Score is a validated readability tool measuring average sentence length and syllables per word. It is expressed on a scale from 0 to 100 with higher scores indicating easier reading. We extracted the text from each PROM and inserted it in the same online Flesch Reading Ease Score calculator to generate a score.

Results

The mean readability score was 55 (range, 0–93), corresponding to text best understood by 16- to 18-year-old students (11^th–12^th grades). Twenty-nine PROMs (49%) scored less than 60, classifying them as at least fairly difficult to read. Eight (14%) scored less than 30, best understood by university graduates. Only seven of 59 PROMs analyzed scored greater than 79, corresponding to text that can be understood by the average UK adult.

Conclusions

The majority of PROMs analyzed are written at a level that is incomprehensible to the average UK adult.

Clinical Relevance

This issue needs to be addressed if we are to continue basing our research conclusions on outcome scores. The information obtained is useful for patients to understand their musculoskeletal health, governmental agencies allocating healthcare resources, provision of management guidelines, and as a link to other data sets, such as hospital episodes statistics. Accurate and reliable data can be obtained only if patients who complete these evaluations are able to read and understand the questions asked.

Introduction

Background

At least seven million adults are functionally illiterate in the United Kingdom (UK), with one in six having a literacy level less than that expected of a 9-year-old [29]; 36% of Americans (89 million) have less than basic health literacy skills [63], and almost ½ of all Australians have literacy skills less than the minimum level required to understand and use information from medical texts [6]. These subjects would have difficulty understanding healthcare questionnaires written at a level greater than their reading grade and have poorer health outcomes [7, 11, 93]. Currently an approach to measure healthcare performance is through patient-reported outcome measures (PROMs) [5, 57, 92]. The use of PROMs in the USA is gaining popularity, and is becoming standard practice in the UK, forming an essential part of treatment pathways and determining best practice [19, 51, 81, 94].

Rationale

Although we know that most online patient education material from the American Academy of Orthopaedic Surgeons is incomprehensible for the average patient, it is unclear whether PROMs are [32]. If we are to continue drawing research conclusions and base future healthcare strategies on PROMs, it is pertinent to identify if patients can comprehend these questionnaires. The Flesch Reading Ease Score (FRES) is a validated technique used to evaluate the ease by which text can be read and understood, defined as readability [2, 37, 53, 70, 99].

Study Purpose

We therefore sought to determine the readability of commonly used PROMs in orthopaedic surgery, as assessed by a validated readability tool that measures the complexity of the language in these surveys, and matching this to the readers’ level of understanding [70].

Materials and Methods

Many algorithms exist; the most widely used is the Flesch–Kincaid reading grade developed for the US Navy in 1975 [2, 53]. The FRES is a validated readability tool inversely related to the Flesch–Kincaid grade, which correlates this to the reader’s education level. It analyzes the average number of syllables per word and the average number of words in a sentence to calculate a numerical score presented as a number from 0 to 100 (Appendix 1). Higher scores indicate text easily read and understood by individuals from lower school years. Applying this to level of education, a score of 90 to 100 is easily understood by 9-year-old subjects (4^th grade), 80 to 89 by 11-year-old subjects (6^th grade), 70 to 79 by 12-year-old subjects (7^th grade), 60 to 69 by 13- to 15-year-old subjects (8^th–10^th grades), 50 to 59 by 16- to 18-year-old subjects (11^th–12^th grades), 30 to 49 by 18- to 22-year-old subjects (undergraduates), and 0 to 29 by postgraduate students (Table 1) [37].

Table 1.

Relationship between FRES and minimum level of education required to comprehend text

FRES	Readability level	Student age	Education level
90–100	Very easy	9 years old	4th grade
80–89	Easy	11 years old	6th grade
70–79	Fairly easy	12 years old	7th grade
60–69	Standard English	13 to 15 years old	8th–10th grades
50–59	Fairly difficult	16 to 18 years old	11th–12th grades
30–49	Difficult	18 to 22 years old	Undergraduates
0–29	Very difficult	Varies	Postgraduates

FRES = Flesch Reading Ease Score.

Among validated readability scores, the FRES was chosen for our study as it is the most commonly used in the assessment of medical material and survey readability [17]. It also is used in numerous other industries such as education and banking, and is the preferred readability score for Microsoft^® Office (Microsoft^® Corporation, Redmond, WA, USA); US courts accept its use in testimony, and insurance policies are required by federal law to have a FRES of 45 or greater [72, 76, 91]. Applying the FRES to published editorials, the Readers Digest scores greater than 65, so it is easily understood by 12-year-old subjects, whereas the Harvard Law Review scores less than 49, so it is best understood by university graduates [46].

To determine which PROMs are commonly used and reported in the orthopaedic literature, a comprehensive MEDLINE search was performed in February 2014 using the keywords: “patient reported” and “scoring system” in combination with “pelvis”, “hip”, “knee”, “ankle”, “foot”, “hand”, “elbow”, “shoulder”, and “spine”.

Abstracts were checked individually by the authors and 121 orthopaedic-specific PROMs were identified. These then were reviewed by 19 attending orthopaedic surgeons from all subspecialties at our institution (Level 1 trauma center) and 59 were selected as most commonly used. Of these, seven (12%) were general health questionnaires and the remaining 52 (78%) were disease specific. These included: 12 (20%) knee, 10 (17%) shoulder, seven (12%) spine, six (10%) hip, five (8%) foot and ankle, four (7%) elbow, three (5%) pelvis, three (5%) hand and wrist, and two (3%) lower limb.

Graphic scales or pictograms were removed from all questionnaires and text was edited to include only words as per guidelines in the readability literature [38]. The extracted text was inserted in the same online FRES calculator after a Google search (www.readability-score.com), generating a score [84].

Results

The mean readability score was 55 (range, 0–93) corresponding to text best understood by 16- to 18-year-old students (11^th–12^th grades). Twelve percent (seven of 59) of the PROMs we analyzed scored greater than 79, corresponding to text that can be understood by the average UK adult (Table 2). Of the seven, three were spine (Oswestry Low Back Pain Score, Roland-Morris Disability Questionnaire, Copenhagen Neck Disability Scale), two were knee (Hughston Clinic Knee Self-Assessment Questionnaire, Lysholm Score), one was a general health questionnaire (SF-36), and one was a hand and wrist PROM (Modified Mayo Wrist Score). Knee PROMs had the widest FRES range (range, 0-80); of those, two of 12 scored greater than 79 (Lysholm and Hughston Clinic scores), whereas four of 12 scored less than 29, corresponding to text best understood by postgraduates (Knee Outcome Survey, Cincinnati Knee Rating System, IKDC, Tegner Activity Scale). Two of three pelvic PROMs analyzed scored 0 (Iowa and Majeed pelvic scores) (Table 3). The Roland-Morris Disability Questionnaire had the highest readability score (FRES 93), while the University of California-Los Angeles (UCLA) Activity Scale, Knee Outcome Survey, Iowa Pelvic Score, and Majeed Pelvic Score had the lowest (FRES 0) (Table 4).

Table 2.

PROMs that can be understood by the average UK adult

PROMs	FRES greater than 79
Roland-Morris Disability Questionnaire	93
Oswestry Low Back Pain Score	83
SF-36	83
Lysholm Knee Score	80
Modified Mayo Wrist Score	80
Hughston Clinic Knee Self-Assessment Questionnaire	79
Copenhagen Neck Disability Scale	79

PROMs = patient-reported outcome measures; FRES = Flesch Reading Ease Score.

Table 3.

PROMs by region with corresponding Flesch Reading Ease Score

PROMs	FRES
Hip
WOMAC hip score [59]	33
Hip disability and Osteoarthritis Outcome Score [59]	35
Lequesne Index of Severity for Osteoarthritis of the Hip [65]	35
Oxford Hip Score [27]	54
Copenhagen Hip and Groin Outcome Score [98]	57
Hip disability and Osteoarthritis Outcome Score–Physical Function Short Form [22]	64
Knee
Knee Outcome Survey [69]	0
IKDC [44]	5
Tegner Activity Scale [96]	17
Cincinnati Knee Rating System [1]	27
Knee injury and Osteoarthritis Outcome Score [89]	41
WOMAC knee score [90]	41
Kujala [62]	58
Oxford Knee Score [28]	59
Anterior Cruciate Ligament Quality Of Life [73]	67
Western Ontario Meniscal Evaluation Tool [57]	75
Hughston Clinic Knee Self-Assessment Questionnaire [36]	79
Lysholm Knee Score [66]	80
UCLA Activity Scale [111]	0
Lower limb
Lower Extremity Functional Scale [12]	67
Elbow
The American Shoulder and Elbow Surgeons-Elbow score [54]	41
Mayo Elbow Performance Score [74]	46
Oxford Elbow Score [24]	62
Kerlan-Jobe Orthopaedic Clinic Shoulder and Elbow score [8]	63
DASH [47]	47
Quick-DASH [9]	36
Shoulder
Oxford Shoulder Score [25]	57
Oxford Instability Score [25, 26]	58
American Shoulder and Elbow Surgeons [54]	58
Western Ontario Osteoarthritis of the Shoulder [61]	61
Western Ontario Rotator Cuff Index [55]	62
Western Ontario Shoulder Instability Index [56]	65
Shoulder Pain and Disability Index [85]	70
Simple Shoulder Test [86]	76
Wrist and hand
Michigan Hand Outcome Questionnaire [19]	60
Patient-Rated Wrist/Hand Evaluation [67]	57
Modified Mayo Wrist Score [4]	80
Spine
Quebec Back Pain Disability Scale [60]	37
Oswestry Low Back Pain Score [35]	83
Roland-Morris Disability Questionnaire [87]	93
The Neck Disability Index [101]	62
Copenhagen Neck Disability Scale [49]	79
The Bournemouth Back Questionnaire [13]	76
The Bournemouth Neck Questionnaire [14]	74
Foot and ankle
Kaikkonen Functional Scale [50]	59
Foot and Ankle Outcome Score [88]	70
Foot and Ankle Disability Index [42]	72
AAOS Foot and Ankle Questionnaire [48]	73
Foot Function Index [15]	74
Pelvis
Iowa Pelvic Score [97]	0
Majeed Pelvic Score [68]	0
Short Musculoskeletal Function Assessment [95]	63
SF-12 [107]	78
Health questionnaire
Stanford Health Assessment Questionnaire [82]	78
SF-36 [108]	83
Physical Activity Scale For The Elderly [109]	65
Euroqol-5D [34] Pain questionnaire	53
McGill Pain Questionnaire [71]	47
Dallas Pain Questionnaire [64]	64

PROM = patient-reported outcome measure; FRES = Flesch Reading Ease Score; AAOS = American Academy of Orthopaedic Surgeons.

Table 4.

PROMs categorized by minimum level of education required to understand questionnaire based on FRES score

FRES	Readability level	Education level	PROMs
90–100	Very easy	9-year-old subjects (4th grade)	Roland-Morris Disability Questionnaire
80–89	Easy	11-year-old subjects (6th grade)	Oswestry Low Back Pain Score
			SF-36
			Hughston Clinic Knee Self-Assessment Questionnaire
			Lysholm Knee Score
			Modified Mayo Wrist Score
			Copenhagen Neck Disability Scale
70–79	Fairly easy	12-year-old subjects (7th grade)	Foot and Ankle Disability Index
			AAOS Foot and Ankle Questionnaire
			Foot Function Index
			Western Ontario Meniscal Evaluation Tool
			SF-12
			Simple Shoulder Test
			Stanford Health Assessment Questionnaire
			The Bournemouth Back Questionnaire
			The Bournemouth Neck Questionnaire
60–69	Standard English	13- to 15-year-old subjects (8th–10th grades)	Western Ontario Osteoarthritis of the Shoulder
			Western Ontario Rotator Cuff Index
			The Neck Disability Index
			Oxford Elbow Score
			Short Musculoskeletal Function Assessment
			Kerlan-Jobe Orthopaedic Clinic Shoulder and Elbow Score
			Dallas Pain Questionnaire
			Hip disability and Osteoarthritis Outcome Score–Physical Function Short Form
			Western Ontario Shoulder Instability Index
			Anterior Cruciate Ligament Quality Of Life
			Lower Extremity Functional Scale
			Shoulder Pain and Disability Index
			Foot and Ankle Outcome Score
			Oxford Knee Score
			Michigan Hand Outcome Questionnaire
			Physical Activity Scale For The Elderly
50–59	Fairly difficult	16- to 18-year-old subjects (11th–12th grades)	Oxford Hip Score
			Oxford Shoulder Score
			Kujala Score
			Copenhagen Hip and Groin Outcome Score
			Oxford Instability Score
			American Shoulder and Elbow Surgeons
			Kaikkonen Functional Scale
			Patient-Rated Wrist/Hand Evaluation
			Euroqol-5D
30–49	Difficult	Undergraduates	WOMAC Hip Score
			Hip disability and Osteoarthritis Outcome Score
			Lequesne Index of Severity for Osteoarthritis of the Hip
			Quick-DASH
			The Quebec Back Pain Disability Scale
			Knee injury and Osteoarthritis Outcome Score
			WOMAC Knee Score
			The American Shoulder and Elbow Surgeons-Elbow Score
			Mayo Elbow Performance Score
			McGill Pain Questionnaire
			DASH
0–29	Very difficult	Postgraduates	Knee Outcome Survey
			University of California, Los Angeles Activity Scale
			Iowa Pelvic Score
			Majeed Pelvic Score
			IKDC
			Tegner Activity Scale
			Cincinnati Knee Rating System

PROMs = patient-reported outcome measures; FRES = Flesch Reading Ease Score; AAOS = American Academy of Orthopaedic Surgeons.

Discussion

We found that a surprising number of PROMs were likely to be unreadable and potentially incomprehensible to most patients asked to complete them. As information from PROMs are helpful to patients in understanding their musculoskeletal health and to those assessing the health care patients receive [20, 31, 43], it is important that these data are accurate and reliable. Our study highlights the importance of readability, which may be overlooked in the design of commonly used orthopaedic PROMs, at least as assessed by the FRES.

This study had several potential limitations. Our study focused only on the readability of text determined through use of mathematical algorithms matching text to the readers understanding producing a score. We did not assess the readability of document design, such as use of color, font size or type, difficulty of concepts conferred, and level of readers’ interest toward the text [45]. PROMs using visual aids such as a series of boxes going from “I totally disagree” to “I totally agree” or a sliding bar from “good” to “bad” may affect the readability of our results. We also did not consider the reader’s familiarity with a subject, for example, patients who have long-term diabetes may be more familiar with disease-related terminology compared with patients with newly diagnosed diabetes. Furthermore our study tested only the readability of the entire document, as opposed to analyzing each question. This does not reflect variation in the readability of individual items. We did not assess whether this would apply to other language translations of these PROMs. FRES considers long polysyllabic words less readable. This may not be true for all terms; for example, “disability” has five syllables producing a low score but otherwise is easily understood. Other validated readability formulas use different mathematical algorithms to measure semantic (words and sentences) and syntactic elements (syllables and sentences) to determine the readability of text [70]. Applying different formulas to the same text may yield different results, whereas selecting a single formula could bias our results. For example, applying the Simplified Measure of Gobbledygook (SMOG) formula to the DASH questionnaire gives a score that indicates it would be easily understood by 12-year-old subjects (7^th grade), whereas using the FRES formula suggests it is best understood by 18- to 22-year-old subjects (undergraduates) [2, 46]. Because we did not compare the FRES with other readability scores, our results can apply only to FRES. Although we assessed the readability of commonly used PROMs at our institution, our selection process is open to selection bias and therefore numerous PROMs used by orthopaedic surgeons elsewhere may not have been included. In addition we made no attempt to rank PROMs in terms of how frequently they appear in the literature.

Non-English PROMs were not assessed as most readability scores are not validated for use in any other language. Despite these limitations, readability formulas are useful in providing a benchmark for comparison when revising text. A better perspective might have been achieved through an average readability score derived from several formulas, although this method has not been validated in the readability literature.

With the majority of medical negligence claims related to poor communication, it is essential that written healthcare material be easily understood [10]. In 1999 Tampa General Hospital paid USD 3.8 million in compensation because signed consent forms were written at a level that exceeded patients’ understanding [46, 99]. Medical organizations in the European Union are legally liable for text that is difficult to understand [16]. In the US, the Plain Writing Act of 2010 was introduced legislating that written healthcare information be understood by adults with average literacy skills [83].

Our results are consistent with those of previous studies that most healthcare information is written at a level greater than patient understanding [23, 33, 39, 41, 102, 104, 105]. However, there is no evidence to suggest that lower literacy levels affect the quality of PROMs data collected.

Only 12% of the PROMs analyzed could be read and understood by the average UK adult; ½ of these 12% were spine PROMS. Thirty-one percent are best understood by university graduates. None of the shoulder, elbow, pelvis, hip, foot, or ankle PROMs met the national average readability requirements. Two of three pelvic PROMs scored zero suggesting the highest literacy level required to complete these questionnaires. UK national standard PROMs (Oxford hip and knee scores) exceeded the average readability requirements [29], at best understood by adults with a literacy level of 13- to 15-year-old subjects (Oxford Knee Score = FRES 59), at worst by college graduates (Oxford Hip Score = FRES 54). Similarly, the preferred outcome measure for use by the Department of Health (EuroQol-5D) is best understood by patients with literacy levels of 17- to 18-year-old subjects (12^th grade) (FRES 53) [5, 30].

Stigma surrounding adults struggling to read often leaves them uncomfortable disclosing any literacy difficulties [80]. Because patients with low literacy skills ask fewer questions concerning information they do not understand, PROMs may be completed incorrectly, leading to unreliable data [52]. However, difficulties associated with comprehension are not exclusive to patients with low literacy. Patients with proficient literacy skills may not seek clarification to avoid embarrassment or appearing ignorant [110]. Equally it cannot be assumed that patients with difficulties understanding fail to seek clarification. For this reason clinicians should identify patients who may struggle, using validated screening tools (Appendix 2) [18, 75, 78, 103]. Despite this clinicians may fail to identify patients with low health literacy and should consider producing “health-literate” PROMs written at or below the level of an 11-year-old subject [21, 40, 106].

To achieve this, writers should determine which medical words are important for patients understanding and where possible, substitute with “plain language” alternatives, for example, swelling rather than edema, and break rather than fracture. Alternatives that change the meaning but improve the readability score should be avoided. Each medical word used should be carefully selected and where appropriate offer an easy to understand explanation or definition for patient reference. Applying these basic principles (Table 5) can facilitate patient-friendly questionnaires by concentrating on ensuring the general content, complexity, and format are easy to understand [100, 110].

Table 5.

Guidelines for writing easy-to-read health-related material

General content

Limit content to one or two key objectives

Avoid information overload

Avoid medical terms

Ensure information is appropriate for target audience age and culture

Text construct

Write at or below the level of 11-year-old students (FRES > 80)

Use one- or two-syllable words

Use active voice

Avoid tables and graphs

Fonts and type

Use a minimum 12-point font size with serifs

Avoid multiple font styles

Avoid uppercase text

Layout

Ensure adequate spacing

Use headings and subheadings

FRES = Flesch Reading Ease Score.

Although shortening sentences may decrease the readability score, it can make it more difficult to understand. Simplifying text using common words that are monosyllabic and bisyllabic, creating shorter more-simple sentences will improve readability while avoiding this limitation. For example, if we take an item from the UCLA Activity Scale that reads: “Wholly inactive, dependent on others, and cannot leave residence”, gives a FRES of 44, which corresponds to text that is difficult to read and is best understood by undergraduates. This can be rewritten to read: “In need of help from others and unable to leave the house”, producing a FRES of 89, corresponding to text that is easily read and understood by 11-year-old subjects (average adult).

Any method used to improve the readability of PROMs will require additional resources to validate this. To establish whether to invest in such resources, further studies are needed to determine whether PROMs with a low readability score would yield substantially different answers if they were rewritten with a higher readability score.

To ensure subsequent interventions are effective and avoid unreliable data, it is crucial that clinicians consider varying levels of health literacy when collecting PROMs data [3, 58, 77, 79]. This can be facilitated by ensuring PROMs are written at or below the average literacy level of the population they are meant to survey [1].

Appendix 1: The FRES Algorithm

The FRES algorithm used to produce a readability score analyzes the average number of syllables per word and the average number of words in a sentence to calculate a numerical score presented as a number from 0 to 100.

FRES = 206.835: (84.6s – 1.015w)

s = the average number of syllables per word; w = the average number of words per sentence.

Appendix 2: The Single Question Screen

The single question screen is a validated tool that can be used to identify patients with low health literacy who may struggle to read and understand PROMs.

“How often do you need someone to help you read instructions, leaflets, or other written material from your doctor or pharmacist?”

• Positive answers: “sometimes,” “often,” or “always”

“How confident are you in filling out medical questionnaires by yourself?”

• Positive answers: “somewhat,” “a little bit,” or “not at all”