Beyond Reading Level: A Systematic Review of the Suitability of Cancer Education Print and Web-based Materials

Abstract

Background

Consideration of categories related to reading comprehension—beyond reading level—is imperative to reach low literacy populations effectively. “Suitability” has been proposed as a term to encompass six categories of such factors: content, literacy demand graphics, layout/typography, learning stimulation, and cultural appropriateness. Our purpose was to describe instruments used to evaluate categories of suitability in cancer education materials in published reports and their findings.

Methods

We searched databases and reference lists for evaluations of print and Web-based cancer education materials to identify and describe measures of these categories. Studies had to evaluate reading level and at least one category of suitability.

Results

Eleven studies met our criteria.. Seven studies reported inter-rater reliability. Cultural appropriateness was most often assessed; four instruments assessed only surface aspects of cultural appropriateness. Only two of seven instruments used, the Suitability Assessment of Materials (SAM) and the Comprehensibility Assessment of Materials (SAM + CAM), were described as having any evidence of validity. Studies using SMOG and Fry reported higher average reading level scores than those using FK. Most materials failed criteria for reading level and cultural appropriateness.

Conclusions

We recommend more emphasis on the categories of suitability for those developing cancer education materials and more study of these categories and reliability and validity testing of instruments.

Background

In 2003, the National Assessment of Adult Literacy Survey reported that older adults, some minority groups, and adults with chronic health problems are more likely to have limited literacy skills¹. These groups are also at higher risk for cancer and are more likely to need cancer-related information. Therefore, it is important to assess factors that influence comprehension of cancer education materials.

Focusing exclusively on “reading level” to evaluate print and Web-based cancer education materials for U.S. adults is not enough; this strategy misses important factors that can influence reading comprehension ^2–6 and perpetuates the gap between what we know we should do and what we actually do when developing cancer education materials. A systematic review of “readability and comprehension” instruments found limitations to those most commonly used, specifically the inability to take into account sentence structure, prior knowledge, and the effects of illustrations ². Thus, materials meeting a recommended reading level, e.g., 6th grade, may not be as comprehensible as the developers might believe; alternatively, higher reading levels are not necessarily too difficult or low literacy audiences, should we use the arbitrary cutoff of 6th grade. Other researchers also suggest that layout,^3,7,8,9 use of graphics and illustrations,^3,8,10,11 learning stimulation and motivation,^3,6 and cultural appropriateness ^3,9 may improve reading comprehension and a patient’s ability to apply health information.

“Suitability” as defined by Doak, Doak, and Root covers six categories of these factors: content, literacy demand, graphics, layout/typography, learning stimulation and motivation, and cultural appropriateness. The purpose of this review is to describe the use of instruments to evaluate categories of suitability in cancer education materials in the published literature. This review will 1) describe the instruments used to assess at least one category of suitability beyond reading level, 2) identify which categories are most frequently measured, and 3) summarize the findings.

Methods

Inclusion Criteria

We used Doak, Doak, and Root’s 1996 categories of suitability³ to create the search strategy for published studies reporting assessments of cancer-specific education materials for categories of suitability. Studies were included if they reported original research, were published in English, and evaluated print or Web-based cancer-specific education material. Studies were included in the final sample if they measured reading level and at least one category of suitability.

Search Strategy

The search was designed by health sciences research librarians and last updated in June 2009. Ovid Medline, EBSCO CINAHL, and Ovid PsychInfo were searched for peer-reviewed articles published since 1996 using combinations of terms from the categories of suitability, measure, assessment, formula, or evaluate, patient education, forms of print materials, cancer, and limited to English-language (the full search strategy and flowchart of article selection are available from the corresponding author). We also reviewed reference lists of selected articles and the Harvard School of Public Health’s Literacy Studies website (http://www.hsph.harvard.edu/healthliteracy/materials.html) for additional assessment instruments and materials.

Study Selection

Titles and abstracts of studies found through the search were reviewed independently for inclusion by three authors (RF, TF, SKL), who then reviewed the full text of questionable articles which were deemed possible candidates for inclusion in the review. The authors discussed discrepancies until agreement was reached.

Coding

Studies were coded for study purpose, intended audience, study design, the number and cancer focus of materials assessed, Web or print form, and findings for reading level and suitability categories. One author (RF) searched references and sent email requests for evaluation instruments to the corresponding author of each included study. Instruments were coded for purpose, categories measured, results, and any reported information about reliability and validity. Findings for each instrument were abstracted based on instrument-specific terms. For instance, studies using the SAM reported results as “superior”, “adequate”, or “not suitable”. For studies not using the SAM, we summarized the score for each category of suitability as “not suitable” or “adequate” based on the original study results.

Two coders abstracted data from the full text of each eligible article. One author (RF) coded all of the selected studies and two additional coders (TF and VG) independently coded half of the selected studies. Discrepancies in coding were resolved by consensus; the data were entered into an Access database (Windows 97-2003).

Results

Identification and Description of Eligible Studies

The database searches yielded 636 unique potentially eligible articles. We discarded 618 after reviewing titles and abstracts. Two authors received the full text of the remaining 18 articles. Studies were excluded if they did not formally assess the written materials^12,13 or if they only evaluated reading level or categories outside the scope of this review, e.g., accuracy and usability.^14–18 Eleven articles met our inclusion criteria; together, they evaluated 432 pieces of cancer education material (262 print, 170 webpages, not necessarily unique) (Table 1).

Table 1.

Study Characteristics

Author, date	Purpose	Materials assessed Target Audiance	Instruments used (inter-rater agreement * - if given)	Findings Reading Level** and Suitability**
Web-based materials
Friedman & Kao, 2008	Assess content, readability and cultural sensitivity	70 prostate cancer prevention; Older minority men	SMOG FK (Flesch-Kincaid) FRE (Flesh Reading Ease) Cultural Sensitivity Assessment Tool (CSAT) Cultural Sensitivity Checklist (CSC)	Reading level: SMOG = 13 (95% CI:12.4 – 13.5) X FK = 11 (95% CI: 10.8 – 11.6) X FRE = 45 (95% CI: 42.0 – 48.1) Cultural approp.: Adequate X CSAT = 2.78 No mention of racial- or ethnic-specific perceptions of cancer risk, cultural beliefs about health, or alternative medicine
Kaphingst et al., 2006	Assess reading difficulty (readability, suitability)	19 colorectal cancer; Adults	SMOG (r=.88) SAM (r=.82)	Reading level: X SMOG = 13 (SD, 95% CI, or range not given) Overall suitability: Adequate = 47%; Not suitable = 53%
Thomson & Hoffman-Goetz, 2007	Assess reading-level and cultural appropriateness of web-based cancer decision-aids	23 cancer decision aids (9 assessed for cultural sensitivity); Cancer patients	SMOG CSAT CSC	Reading level: X SMOG = 9 (range 8 – 13) Cultural approp.: Adequate X CSAT 2.9 and 3.0 Only the two identified as culturally appropriate by the CSAT were also identified as culturally appropriate by the CSC
Wilson et al., 2000	Assess reading levels and cultural sensitivity	49 cancer (CancerNet); Lay people of various ethnic groups	FK Bloch’s Ethnic/Cultural Assessment Guide (100%)	Reading level: X FK = 12th grade (SD = 2.9) Cultural approp. = Not suitable
Web-based and Print Materials
Helitzer et al, 2009	Assess reading level, comprehensibility, suitability, and message design	9 Web; 60 print; cervical cancer and HPV prevention Adults	Fry SAM + CAM (Comprehensibility Assessment of Materials) (k = .77)	Reading level: X Fry = 11th grade Content = 51% adequate Lit. demand = 50% superior Numeracy = 73% superior Graphics = 55% adequate Layout = 66% superior Learning stim. = 60% not suitable
Print materials
Singh, 2003	Assess readability and estimate reading levels	10 cancer; General population	SMOG (100%) RAIN (94%, 97%)	Reading level: X SMOG = 12 (SD, 95% CI, or range not given) Content not suitable - signaling devices; adequate - unity Literacy demand not suitable - writing style; adequate - pronoun references, substitutions, connectives Graphics = not suitable Layout not suitable - print size or print style; adequate - color and highlighting of headers Learning stim.= adequate Cultural approp = not suitable
Weintraub et al., 2003	Assess suitability	29 prostate cancer; Men with prostate cancer	Fry SAM (100% negotiation)	Reading level: Fry score 7 – 17 Overall suitability: Superior = 6 (21%); Adequate = 22 (76%); Not suitable = 1 (3%)
Rees et al., 2003	Assess quality, readability, and suitability	31 prostate cancer; Men with prostate cancer	FRE SAM (individual items, K =.20–1.0)	Reading level: X FRE = 53 (SD = 8.7) Overall suitability: Superior = 19%; Adequate = 65%; Not suitable = 16%
Mohrmann, et al., 2000	Assess accuracy, readability, and cultural sensitivity	60 breast cancer; African American women	FK FRE CSAT	Reading level: X FK = 8 (SD = 2.5) X FRE = 63 (SD = 9.4) Cultural approp. = Not suitable
Guidry et al., 1998	Assess cultural sensitivity and readability	34 breast cancer and 12 prostate cancer; African American women and men	SMOG CSAT	Reading level: X SMOG = 9 (SD = 2.10) for breast and 9 (SD = 2.6) for prostate Cultural approp. = Not suitable X CSAT = 2.41 for breast; 1.91 for prostate
Masset, 1996	Assess linguistic appropriateness and cultural sensitivity	26 breast cancer prevention, Spanish; Hispanic women	SMOG (84%) Self developed coding instrument (82%)	Reading level: X SMOG = 10 (SD, 95% CI, or range not given) Literacy demand = Not suitable Graphics = Not suitable Typography = Not suitable Cultural approp. = Not suitable

^*reported as % agreement, correlation, or weighted kappa (k) coefficient

^***Reading level interpretation:

FRE: 0–30 understandable to college students; 31–59 understandable to high school graduate; 60–70 understandable to those 13–15 years of age

Fry: score equals grade level (17=college plus 1 year of graduate school)

SMOG and FK: score equals minimum grade level in U.S. needed to understand text; 12.8 = college freshman

Suitability score based on number of materials to which each category was applied (if a material did not assess that category it was not included in the count)

Overview of Instruments

Seven instruments were used to assess categories of suitability (Table 2). Four assessed surface structure elements of cultural appropriateness (appropriate graphics, language, physical appearance).^3,19–21 Three assessed use of illustrations or graphics outside of cultural appropriateness and literacy demand beyond reading level. ^3,19,22 Four assessed content (evident purpose, inclusion of behavioral guides, scope of the material, and inclusion of summary of information)^3,19,21,22 while one assessed an additional category: accuracy of the information.²⁰ Five assessed layout and typography, ^3,19–22 and four, learning stimulation and motivation (quizzes and suggested action). ^3,22,23 Seven studies assessed inter-rater reliability of instruments^{19,21,22,24–27}. Two of seven instruments were described having any evidence of validity^3,22. The following sections describe the results of each of the studies, grouped according to the instrument used.

Table 2.

Instrument Characteristics

Instrument (1st publication)	Purpose	Suitability Categories (# items)	Interpretation	Strengths	Weaknesses
Suitability Assessment of Materials (Doak, Doak, & Root, 1996)	To assess the suitability of print materials	Content (4) Lit. demand (4) Graphics (5) Layout (3) Learning stim. (3) Cultural approp. (2)	Items are scored 0 (not suitable) to 2 (superior) and divided by the total possible; %’s are converted to categories: 0%– 39% = not suitable 40%–69% = adequate 70%–100% = superior	Diverse group of healthcare professionals provided content validity; identifies specific areas that reduce suitability	Difficult to apply when there is little or no info about the intended cultural audience
Comprehensibility Assessment of Materials (Helitzer et al, 2009)	To assess comprehensibility, suitability, and message design of materials	Content (4) Lit. demand (5) Graphics (2) Layout (3) Learning stim. (6) Numeracy (2)	Same as the SAM	Expert reviewers provided content validation; adds criteria to evaluate communication factors	New instrument, will benefit from further use
Readability Assessment Instrument (Singh, 2003)	To determine the readability of texts in terms of 14 variables that affect comprehension.	Content (2) Lit. demand (5) Graphics (1) Layout (4) Learning stim. (1) Cultural approp. (1)	For each variable indicate whether the variable meets criterion by circling Yes or No (datasheet 3). Total the number of variables meeting criterion. There is no summary RAIN score.	Incorporates more variables into assessment of reading comprehension.	Has not been validated
Cultural Sensitivity Assessment Tool (Guidry et al, 1996)	To determine the degree to which individual printed cancer education materials are culturally sensitive.	Graphics (4) Layout (3) Cultural approp. (12) Other – accuracy and effectiveness of information (11)	Range for each question: Strongly Agree= absolutely will be accepted by the audience (4 points) to Strongly Disagree= absolutely will not be accepted by the intended audience (1point). Not Applicable= 0 points. For each category: adjusted question total = (total # of questions) minus (total # of 0 scores (N/A); response score total = sum of scores; category score = (response score total) divided by (adjusted question total). Category score≤ 2.5 is culturally insensitive	Included target audience in its development; only quantitative measure of the cultural sensitivity of cancer prevention resources	Has not been validated; scoring system allows generic materials to be rated as culturally sensitive
Cultural Sensitivity Checklist (CSC) (Friedman and Hoffman-Goetz, 2006)	To comprehensively assess cultural sensitivity of cancer information	Learning stim. (1) Cultural approp. (7)	Scores are calculated by answering yes or no to eight questions. There is no summary CSC score.	Assesses deep structure elements of cultural sensitivity	Has not been validated
Bloch’s Ethnic/Cultural Assessment (Bloch, 1983)	To evaluate appropriateness given ethnic/cultural differences in four categories: cultural, social, psychological, and biophysiological	Cultural approp.(4)	Not reported (NR)	Quick way to screen materials	Was not originally developed to assess print materials and has not been validated
Masset’s Checklist (Masset, 1996)	To assess the linguistic appropriateness and cultural sensitivity of print materials distributed to Hispanic women	Content (NR) Lit. Demand (NR) Graphics (NR) Layout (NR) Culturally appropriate (NR)	NR	Developed through lit review, interviews with experts, and focus group discussions	Cannot assess*

^*Did not receive a copy of Masset’s Checklist

Suitability Assessment of Materials (SAM)

The three studies using the SAM assessed all six categories of suitability. Two of these studies focused on prostate cancer print materials and found most materials to be “adequate” or “superior”^25,26; in contrast, the study of colorectal cancer webpages ²⁴ found a majority of webpages to be “not suitable”. The most commonly failed factors, for both print and Web-based materials, were 1) presentation of information in a behavior-related context (giving patients behavioral guides); 2) summary of key ideas; 3) use of illustrations; 4) use of interactive features (e.g. quizzes); and 5) use of culturally appropriate visuals (specific data not shown).

Suitability and Comprehensibility Assessment of Materials (SAM + CAM)

Helitzer and colleagues (2009)22 modified the SAM by adding other variables that address comprehensibility: Presentation of numerical information, use of behavioral theory, and use of framing and tone of messages and used this modified version to assess comprehensability of Webpages and newspaper and magazine articles, health education brochures, and revised insurance and health system forms. These reviewers found most cervical cancer and HPV materials were “adequate” (68%) or “superior” (20%). Webpages and health education brochures were ranked in the “superior” category more often than other materials.

Readability Assessment INstrument (RAIN)

The study using RAIN to evaluate cancer brochures¹⁹ reported that all ten assessed materials met RAIN criteria for pronoun references, connectives, unity, color, and highlighting of titles and subtitles. Brochures varied in terms of using signaling devices, print style, and adjunct questions. Few brochures used substitutions or illustrations. None of the assessed brochures met RAIN criteria for sentence structure, audience appropriateness, writing style, use of illustrations, or print size.

Cultural Sensitivity Assessment Tool (CSAT)

The CSAT was developed to assess the cultural appropriateness of materials intended for African Americans, which is still its primary use.²⁰ The four studies using CSAT evaluated materials intended for racial and ethnic populations. One of the reviews found that the print materials focused on breast cancer were not culturally sensitive.²⁸ Another found just over half of the print materials on breast cancer were culturally sensitive, whereas the majority of print materials on prostate cancer were not.²⁹ The third review found some prostate cancer prevention webpages scored in the culturally sensitive range despite “not mentioning high-risk racial or ethnic groups”.³⁰ The fourth study only found two materials (specifically written for African Americans) to be culturally sensitive. ³¹ Most materials were reported to have low scores in the visual category because they did not present images of the intended racial or ethnic group.

Cultural Sensitivity Checklist (CSC)

Two studies included in this review used the CSC in addition to the CSAT to assess what Resnicow and colleagues³² classify as deeper aspects of cultural sensitivity such as beliefs and perceptions about dying, symbolic represenations of health and illness, traditional medicine, latent messages, and themes²³. Both studies ^30,31 reported that less than 25% of reviewed materials assessed: 1) racial or ethnic perceptions of cancer risk, in the racial or ethnic group, 2) cultural beliefs about health, or 3) traditional or alternative medicine as methods of cancer prevention or treatment.

Bloch’s Ethnic/Cultural Assessment

One study ²⁷ adapted four questions from Bloch’s Ethnic/Cultural Assessment Guide³³ to assess the cultural appropriateness of information targeted to ethnic/racial groups on CancetNet’s webpages: 1) Did the written information identify or target particular groups? 2) Did the written information contain statements about the target groups’ beliefs toward life, death, and illness? 3) Was the written information presented in the language(s) of the target group(s)? 4) Did written information address cultural healing systems or practices? The results showed that although the materials targeted seven different ethnic groups, information for each ethnic group was presented in the same way without regard to culture. The only exception was that materials targeted to Hispanics were available in Spanish and English.

Masset’s Checklist

The study using Masset’s Checklist found that the majority of the 26 assessed materials did not meet recommended criteria in any area.²¹ Eighty-one percent of materials did not repeat key messages. None of the materials included a review section. Most materials (85%) did not define unfamiliar terms. Materials with graphics consisted of simple line drawings containing “extraneous background detail” (84%) and did not represent Hispanic persons (the intended audience) or settings. Typographic factors (font size, white space, bulleting format, all caps) were generally outside of recommended guidelines.

Discussion

The purpose of this systematic review was to describe instruments that measure categories of suitability and summarize their published results in evaluating print and Web-based cancer education materials. Among the eleven included studies, we found seven distinct instruments that were used to evaluate as many as 432 cancer education materials. These instruments most frequently assessed the cultural appropriateness of the materials, and most materials failed the criteria for this category. Most studies assessed inter-rater reliability. A surprising finding was that only two of the instruments included in this review (SAM and SAM+CAM) were described as having any evidence of validity, and this evidence was limited to content validity.^3,22 The RAIN and CSAT have been used to evaluate patient education materials in multiple studies, but no indication of validity of any kind was reported for these instruments.

We found differences in reading level scores that appeared to reflect the varying bases of the instruments used. For example, studies using the Simplified Measure of Goobledygook (SMOG) to assess reading level reported higher mean scores than studies using the Flesh-Kincaid (FK). One study that used both the SMOG and FK reported a difference of two grade levels between the two scores.³⁰ This is consistent with research which has reported that the FK (the “readability” formula used in Microsoft Word) tends to score written materials at a lower reading level than other reading-level formulas.² This variability in scoring provides further support for not using reading level alone to evaluate the literacy demand of cancer education materials.

Cultural appropriateness was frequently found to be only “adequate” or “not suitable” because materials did not present images of the intended group or lacked images in general.^21,26–29 One study using the CSAT reported that even webpages rated as culturally sensitive did not present images or mention the intended minority group – a limitation of CSAT’s scoring process – while results from the CSC point out deep structure characteristics (cultural, social, historical, and environmental factors) that were missing from webpages.³⁰ Similar to the CSAT, the SAM scores materials with “neutral” images or that do not present images of the intended minority group in the “adequate” category.³ To improve the usefulness of print and Web-based cancer education materials, health resources must be created with deep structure characteristics in mind. Cultural health beliefs, practices, and communication preferences differ among ethnic groups.³⁴ Therefore, it is important to consider these factors when designing cancer education materials.

Other researchers have suggested that some patients prefer formats such as audio or video.^8,35–37 These formats can be used alone or to supplement reading materials and can be personalized to the audience^38,39. Suitability measures need to be created or adapted before they can be used for audio-visual media, however.

In this review, we found that some cancer education materials might actually be adequate in terms of categories of suitability even though the reading levels are too high. This may be particularly true in specialized areas, including cancer education. In such areas, readability formulas may overestimate the difficulty of commonly used medical terms based on word length alone. Moreover, the most frequently used readability formulas do not take into account the use of glossaries when scoring a material. Thus, improving reading level alone will not guarantee that patients will understand or use education materials; other categories of suitability should be taken into consideration when developing or updating print or Web-based materials.

Strengths and Limitations

We believe that our review is strengthened by the use of Doak, Doak, and Root’s definition of categories of suitability. These categories include many of the elements that influence comprehension of written materials. A limitation of limiting our definition to six categories of suitability was that we missed assessing materials that evaluated other important aspects of suitability such as the quality (accuracy or up-to-dateness) of information and usability of websites.as Others have conducted studies to assess the quality of health education materials^18,40,41, and found that materials lacked information on treatment options, contained inaccurate or out-of-date information, or were missing information regarding effectiveness of treatment, among other issues; the U.S. Department of Health and Human Services has developed a guidebook for research-based approaches to Web design (http://usability.gov/pdfs/guidelines_book.pdf); the World Wide Web Consortium has also released a set of Web accessibility standards and guidelines (http://www.w3.org/WAI/guid-tech.html). Quality, usability and accessibility issues were outside the scope of this review, but are important points to consider when creating suitable print- or Web-based materials.

The seven instruments have not been compared to one another directly. And, our review is limited because we were unable to determine how much the scoring of categories of suitability differs between instruments. Using the studies to compare the instruments indirectly was hampered because we could not reliably determine the degree of overlap between the sets of materials evaluated across studies, although the search methods described by the authors do suggest that most studies assessed representative samples of materials were assembled from national organizations.

Conclusions

Assessment of the six categories of suitability has not been widely reported in the published literature evaluating cancer education print and Web-based materials. Findings from studies included in this review indicate that there are numerous shortcomings in materials related to suitability, the most frequently reported being in literacy demand and cultural appropriateness. Developers of cancer education materials need to be aware of these shortcomings and consider including an assessment of categories of suitability when reviewing current materials and when developing new materials. Future research is still needed to determine whether improving materials based on the categories of suitability also goes beyond increasing appeal among users to increasing knowledge and desired behavior change.

Finally, reliable and valid assessment tools are important to the accurate assessment of suitability. Five of the instruments reported any evidence of reliability (e.g., inter-rater reliability), and only two reported content validity. Therefore, further study of the categories suitability, including testing the measures for reliability and validity, is needed. Only then, can we develop, evaluate, and present materials that patients can dependably understand..