Abstract
Numerous breast cancer risk assessment tools that allow users to input personal risk information and obtain a personalized breast cancer risk estimate are available on the Internet. The goal of these tools is to increase screening awareness and identify modifiable health behaviors; however, the utility of this risk information is limited by the readability of the material. We undertook this study to assess the overall readability of breast cancer risk assessment tools and accompanying information, as well as to identify areas of suggested improvement. We searched for breast cancer risk assessment tools, using five search terms, on three search engines. All searches were performed on June 12, 2014. Sites that met inclusion criteria were then assessed for readability using the suitability assessment of materials (SAM) and the SMOG readability formula (July 1, 2014–January 31, 2015). The primary outcomes are the frequency distribution of overall SAM readability category (superior, adequate, or not suitable) and mean SMOG reading grade level. The search returned 42 sites were eligible for assessment, only 9 (21.4 %) of which achieved an overall SAM superior rating, and 27 (64.3 %) were deemed adequate. The average SMOG reading grade level was grade 12.1 (SD 1.6, range 9–15). The readability of breast cancer risk assessment tools and the sites that host them is an important barrier to risk communication. This study demonstrates that most breast cancer risk assessment tools are not accessible to individuals with limited health literacy skills. More importantly, this study identifies potential areas of improvement and has the potential to heighten a physician’s awareness of the Internet resources a patient might navigate in their quest for breast cancer risk information.
Electronic supplementary material
The online version of this article (doi:10.1007/s10549-015-3601-2) contains supplementary material, which is available to authorized users.
Keywords: Risk assessment, Readability, Breast cancer, Communication
Introduction
Breast cancer is the leading cause of cancer death in women worldwide. In the United States, it is the leading cause of death for women aged 20–59 years [1]. Numerous risk factors for the development of newly diagnosed breast cancer have been identified [2]. Factors such as age at menarche and family history are non-modifiable risk factors that significantly contribute to lifetime risk [3]. Modifiable risk factors, such as alcohol consumption and obesity, involve lifestyle choices that an individual could alter to reduce her personal risk of breast cancer [4]. Breast cancer risk assessment tools are used to give patients a sense of their level of risk to better individualize screening recommendations, as well as to inform women about modifiable risk behaviors [5].
Today, the Internet is an important source of risk information. A 2013 U.S.-based survey found that 59 % of people had searched the Internet for health information [6]. The risk knowledge gained by an Internet search is influenced not only by quality of information but also by an individual’s health literacy. Health literacy is “the degree to which individuals can obtain, process, and understand the basic health information and services they need to make appropriate health decisions” [7]. Limited health literacy is common, with 36 % of the U.S. population estimated to have limited health literacy [8]. When looking for health information, adults with limited literacy skills are more likely to use search terms that are broad, choose sites with an 11th grade or higher reading level, and click on advertisements [9]. For breast cancer risk assessment tools, health literacy can affect the user’s ability to comprehend the instructions to accurately complete the risk tool, as well as the user’s ability to comprehend the risk tool’s output.
Numerous studies have evaluated the importance of health literacy in cancer risk communication. Individuals with limited literacy skills are less likely to understand the purpose of cancer screening and less able to apply relative risk reduction information to their own personal cancer risk [10–12]. This has a significant impact on the health of individuals with limited health literacy, as they report higher distress about developing cancer and lower rates of breast cancer screening [7, 13].
One opportunity to improve health outcomes for individuals with limited health literacy is to improve the design of health information [7]. The format of risk communication in online cancer risk assessment tools has been previously profiled [14]; however, the readability of breast cancer risk assessment tools has not been assessed. Readability describes the difficulty or ease of reading informational materials and consists of many factors in addition to the reading grade level, such as content and typography.
This study describes the overall readability of website pages that host or link to a breast cancer risk assessment tool. To accomplish this goal, this study evaluated not only a site’s reading grade level but also a number of other formatting and content characteristics that contribute to comprehension as guided by the Suitability Assessment of Materials [15]. This study then looks to identify areas of improvement for these websites.
Methods
Search protocol
To complete an Internet search for websites containing or linking to a breast cancer risk prediction model, we entered the following search terms: calculate breast cancer risk, breast cancer risk calculator, estimate breast cancer risk, assess breast cancer risk, and breast cancer risk assessment. We searched each term on three different search engines: Google, Bing, and Yahoo. Searches were performed in a web browser that did not have any search history, as search engines can customize returns based on previous searches. All searches were completed on June 12, 2014 to ensure no variability in return based on date of search.
To track the websites returned, we gave each unique site an ID number. A website was considered unique if its base site had not yet been returned on any search engine for any term. For example www.brightpink.org/Risk-Factors and www.brightpink.org/knowledge-is-power/assess-your-risk/ were considered to have the same ID number because they have the same base site, which is brightpink.org and were therefore treated as a single site for analysis.
Inclusion/exclusion criteria
Inclusion criteria for the sample were any unique site that contained either (1) a breast cancer risk assessment tool or (2) a link to a breast cancer risk assessment tool. Sites that were developed with the intention of bringing patients into the physician’s office for risk assessment and did not offer the risk tool online were therefore not included. Exclusion criteria were sites that presented the risk tool in the context of research articles, news articles, blogs, forums, or sites that contained only links. Sites that were non-U.S. based were excluded on the basis that a risk assessment tool developed for a non-U.S. population might not be applicable to U.S. patients. Other excluded sites were inaccessible, featured non-relevant content, or required an app or software download. We did not exclude ads. As stated prior, individuals with limited health literacy skills have been shown to preferentially click on ads [9].
Upon evaluation of sites that met the inclusion criteria, three of the coded sites evaluated the user for Hereditary Breast and Ovarian Cancer Syndrome and two sites featured a tool that evaluated the user for multiple cancers: breast, prostate, colon, melanoma, and lung cancer. These sites were included as they featured a tool that brought the user’s attention to their personal risk of breast cancer.
Two of the coded sites featured a breast cancer risk assessment tool that did not give output, but rather featured a series of yes/no questions and stated that the patient may be at higher risk for breast cancer if they answered yes to any question. These sites were included because they provide the user with more personalized breast cancer risk information than the general population risk.
Assessment of readability
To assess the readability of the breast cancer risk assessment tools, we evaluated sites using the suitability assessment of materials (SAM) and the SMOG Readability Formula [15, 16]. The SAM evaluates materials based on 22 factors that fall into one of six categories: content, literacy demand, graphics, layout and typography, learning stimulation and motivation, and cultural appropriateness (see eText 1 in the Supplement). Each SAM factor is given a score of (2) for superior, (1) for adequate, or (0) for not suitable. The sum total of all ratings for each website yields an overall superior, adequate, or not suitable rating for the site. The SMOG Readability Formula generates a numerical reading grade level based on the number of polysyllabic words in 30 sentences of text. For sites that did not contain 30 sentences, the SMOG offers a conversion table based on assessment of available sentences (see eText 2 in the Supplement).
Two independent researchers coded the same sections of 10 of the sites to demonstrate inter-rater agreement of 80 % or greater. To maximize potential output from risk tools, researchers were instructed to input an increased risk profile (>60 years old, no births, high BMI, etc.). For those SAM items on which researchers did not agree, a meeting was held with a third researcher who is an expert in health literacy and guidelines for interpretation of SAM criteria were refined (see eText 3 in the Supplement). Another 4 sites were coded by both researchers achieving >0.8 agreement on all 22 factors of the SAM. The same procedure was applied to the SMOG readability formula, with 100 % agreement. Once good inter-coder reliability was achieved, one coder coded the remaining websites.
Within each site, coders evaluated the pages that contained the introduction to the breast cancer risk assessment tool, the tool itself and the output. If the site just linked to the breast cancer risk assessment tool, coders evaluated the page containing the link to the breast cancer risk assessment tool. We decided to code the entirety of a webpage, even if only one paragraph contained breast cancer risk assessment tool information because a user would not be able to find the risk tool information without reading the entire page. Website coding occurred between July 1, 2014 and January 31, 2015.
Statistical analysis
Site characteristics and each of the 22 SAM categories were described using simple frequencies. To facilitate useful discussion about substantially contributing factors to overall scores, results were stratified by overall SAM score. SMOG reading grade level was evaluated using a statistical mean.
To evaluate the effect of host organizations on readability, we stratified SAM categories, as well as SMOG reading grade level by host organization type. Cancer centers, hospitals, and private practices were listed as separate organization types because the target population and patient education goals for these institutions are likely different [14]. Other organization types were commercial industry, healthcare industry, online media, advocacy/non-profit, government, and research group. For SAM categories, χ2 test of independence was performed to compare stratified groups. For SMOG, one-way ANOVA was performed to compare mean reading grade level between stratified groups. All statistical analysis was performed using Stata 13.1 (College Station, TX: StataCorp LP).
Results
Our search returned 576 sites of which 42 met inclusion criteria and were ultimately coded (see Fig. 1). A complete list of the coded sites, sorted by overall SAM rating, can be found in Table 1. Only 21.4 % of sites achieved an overall superior rating, 64.3 % were deemed adequate, and 14.3 % were rated not suitable.
Fig. 1.
Flow chart of website collection
Table 1.
Complete list of websites hosting or linking to breast cancer risk assessment tools, sorted by rating
In terms of website content, 52.4 % of sites hosted a breast cancer risk assessment tool, while the remaining sites linked to one or more tools (see Table 2). The majority of risk assessment tools, 61.9 %, used a Gail-based model. The second most common tool was Krames Staywell (9.5 %). All sites that linked to (rather than hosting) a breast cancer risk assessment tool linked to the National Cancer Institute’s Gail-based model. Output of the risk assessment tools varied with 83.3 % of tools yielding a numerical output (i.e. “Your lifetime risk for breast cancer is 11.6 %”) as opposed to a word output (i.e., “Your lifetime risk for breast cancer is higher than average”).
Table 2.
Website characteristics
| Category | Characteristics | Example | n (%) |
|---|---|---|---|
| Risk tool location | Sites that host a breast cancer risk assessment tool | 22 (52) | |
| Sites that link to a single breast cancer risk assessment tool | 16 (38) | ||
| Sites that link to more than one breast cancer risk assessment tool | 4 (10) | ||
| Risk tool output | Numerical output | This woman (age 60): 3.1 % | 35 (83) |
| Verbal output | Your risk is much above average | 7 (17) | |
| Risk tool type | Gail | 26 (62) | |
| Krames Staywell | 4 (10) | ||
| Harvard Center for Cancer Prevention Risk Assessment Tool | 2 (5) | ||
| Other | 10 (24) | ||
| Website host affiliation | Online media | WebMD | 8 (19) |
| Hospital | Brigham and Women’s Hospital | 6 (14) | |
| Cancer Center | MD Anderson | 5 (12) | |
| Advocacy/non-profit | World Wide Breast Cancer | 5 (12) | |
| Healthcare industry | BlueCrossBlueShield | 5 (12) | |
| Private practice | Halls MD | 4 (10) | |
| Commercial industry | Myriad genetics | 3 (7) | |
| Government | National Cancer Institute | 3 (7) | |
| Research group | breastcancerprevention.com | 3 (7) |
Sub-analysis was used to compare the frequency of obtaining a superior rating, in a given SAM category, stratified by overall SAM rating. Contribution of an individual SAM factor to the overall score was considered substantial when >50 % of sites, with an overall superior SAM rating, received a superior score in a given factor, while overall adequate and not suitable rated sites received a superior score at a frequency that dropped by 25 % in that same factor. The 6 of 22 SAM factors that contributed substantially to the overall superior rating were Content, Writing style, Context, Layout, Subheadings, and Model behavior (see Table 3). Description of the components of these 6 factors can be found in eTable 1 in the Supplement. The factors that overall superior rated sites failed to achieve at a 50 % level were Summary, Reading grade, Vocabulary, Cover graphic, Relevance of illustrations, and Interaction. Description of the components of these 6 factors is found in Table 4. Graphic type, List/tables explained, Captions, and Culture image factors were not included in this list because most sites were scored as “not applicable.”
Table 3.
Frequency of obtaining a superior rating for an individual factor, stratified by overall ratinga
| SAM factor | Overall SAM rating: superior | Overall SAM rating: adequate | Overall SAM rating: not suitable |
|---|---|---|---|
| Purpose | 7 (78) | 23 (85) | 2 (33) |
| Content | 8 (89) | 5 (19) | 0 (0) |
| Scope | 9 (100) | 20 (74) | 5 (83) |
| Summary | 3 (33) | 6 (22) | 0 (0) |
| Reading grade | 1 (11) | 0 (0) | 0 (0) |
| Writing style | 8 (89) | 13 (48) | 0 (0) |
| Vocabulary | 1 (11) | 3 (11) | 0 (0) |
| Context | 7 (78) | 6 (22) | 0 (0) |
| Learning aid | 5 (56) | 19 (70) | 2 (33) |
| Cover graphic | 2 (22) | 6 (22) | 0 (0) |
| Graphic type | 4 (44) | 0 (0) | 0 (0) |
| Relevance of illustrations | 3 (33) | 0 (0) | 0 (0) |
| List/tables explained | 2 (22) | 0 (0) | 0 (0) |
| Captions | 2 (22) | 2 (7) | 0 (0) |
| Layout | 9 (100) | 16 (59) | 2 (33) |
| Typography | 8 (89) | 22 (81) | 4 (67) |
| Subheadings | 8 (89) | 9 (33) | 1 (17) |
| Interaction | 2 (22) | 0 (0) | 0 (0) |
| Model behavior | 9 (100) | 5 (19) | 0 (0) |
| Motivation | 9 (100) | 21 (78) | 3 (50) |
| Culture match | 9 (100) | 27 (100) | 6 (100) |
| Culture image | 1 (11) | 2 (7) | 0 (0) |
Bolded categories contributed to an overall superior SAM rating
aReported frequencies, n (%)
Table 4.
Areas of suggested improvement for all sites
| SAM factor | Description | Example |
|---|---|---|
| Summary | Retell the key messages at the conclusion |
“Based on the information provided (see below), the woman’s estimated risk for developing invasive breast cancer over the next 5 years is 4.6 % compared to a risk of 1.8 % for a woman of the same age and race/ethnicity from the general U.S. population ”
National Cancer Institute http://www.cancer.gov/bcrisktool/ |
| Reading grade | 5th grade level or lower | |
| Vocabulary | Common words Technical words are explained with examples Imagery words are used |
“If you find new changes like this that stay around, show your doctor”
Worldwide Breast Cancer http://www.worldwidebreastcancer.com/ |
| Cover graphic | Friendly Attracts attention Portrays purpose |
Follow the link below for an example of a superior cover graphic Midstate Medical Center http://www.midstatemedical.org/cancer_breast_risk.aspx |
| Relevance of illustrations | Illustrations present key messages visually | Follow the link below for an example of a superior illustrations: Worldwide Breast Cancer http://www.worldwidebreastcancer.com/ |
| Interaction | Problems or questions were provided for reader responses |
“You have
3
things you can do to lower your risk. To see what your risk could be, click on a box and watch your risk drop”
Your Disease Risk http://yourdiseaserisk.wustl.edu/YDRDefault.aspx?ScreenControl=YDRGeneral&ScreenName=YDRbreast |
The above are categories that fewer than 50 % of overall superior SAM-rated sites featured
The website’s host affiliation was broken down into several categories (see Table 2). Online media, hospitals, cancer centers, advocacy, and healthcare industry made up the majority of our host sites. Chi-squared analysis of the frequency distribution of achieving a superior rating, in a given SAM factor, stratified by website affiliation was statistically significant for 3 of the 22 SAM factors: Summary χ2 (16) = 27.25, p = 0.039, vocabulary χ2 (16) = 27.65, p = 0.035, and Interaction χ2 (8) = 15.54, p = 0.049. Online media and government sites were more likely to provide a summary, whereas all sites hosted by a private practice, in this study, did not provide a summary. Commercial industry sites were more likely to use common words instead of technical jargon. Only sites hosted by cancer centers made use of interaction to communicate risk. Overall SAM rating was not affected by host affiliation χ2 (16) = 19.45, p = 0.25.
The average SMOG reading grade level of the sites was grade 12.1 (SD 1.6, range 9–15). Comparison of the mean SMOG reading grade levels, stratified by website affiliation, using one-way ANOVA yielded no significant difference in means (p = 0.50).
Discussion
The readability of breast cancer risk assessment tools is an important component of effective risk communication. Since the uptake of Internet-based health information is so prevalent, quality, Internet-based risk communication has the potential to alert users to personal, modifiable risk behaviors. Sites with low readability, however, could potentially mislead limited health literacy users about their breast cancer risk or discourage limited health literacy users from using a breast cancer risk assessment tool.
This study is the first to evaluate the readability of online breast cancer risk assessment tools and the information accompanying those tools. Using a search engine, we identified 42 unique sites that hosted or linked to a breast cancer risk assessment tool. Sites were hosted by a variety of organizations, the most frequent being online media, such as WebMD or healthcare organizations such as cancer centers, public hospitals, and private practices. The tool most sites hosted or linked to were Gail based. The Gail model was developed for use by healthcare providers, while most sites were developed for general public use. The development of an applicable, validated risk tool does not always lead to easy-to-communicate risk information for the broader population.
Using the SAM to rate readability, only 21.4 % of sites achieved an overall superior rating. Factors that contributed most to an overall superior rating were Content, Writing Style, Context, Layout, Subheadings, and Model Behavior. This means that sites that were rated as superior overall were rated as such because they started with an intro that explained why breast cancer is important or what the user was about to read (Context), featured content focused on desired behaviors (Content), spoke with an active voice (Writing Style), modeled behaviors specifically, for example, “you should have no more than one drink a day” (Model Behavior), subdivided long lists with subheadings (Subheadings), and presented the material in a visually easy to follow format (Layout).
Factors that even overall superior rated sites were unlikely to feature were Summary, Reading grade, Vocabulary, Cover graphic, Relevance of illustrations, and Interaction. This means that the majority of sites, regardless of rating, did not use a reading grade level of 5th grade or lower, common words or explanations for technical jargon, a friendly purposeful opening image, any illustrations at all, interactive learning, or end with a review of key points. This indicates that these are areas that websites could improve to increase the readability of their site.
The recommended reading grade level for patient-directed health information is 5th or 6th grade [17]. The average SMOG reading grade level of our sites was 12.1 and did not differ significantly based on host affiliation. While valid and easily reproducible, the SMOG reading grade level is on average one or two grade levels higher than other reading grade levels because the SMOG reports grade level required for 100 % comprehension [17]. Even so, the average reading grade level is above the reading skills of many patients. We recommend developers of health information materials use common wording (i.e., doctor instead of physician), words with fewer syllables and shorter sentences.
Strengths and limitations
The SAM tool was not originally developed for evaluation of web-based information, but rather paper brochures, booklets or audiovisual materials. The SAM has been used to evaluate web-based materials by numerous studies and organizations; however, certain aspects of the SAM are not directly translatable to a web-based platform. The SAM cannot take into consideration how many pages a user must click on to find all of the information. A brochure has a single trajectory, whereas a web user may follow any number of links on a website to a number of ends. In addition, the scores of many SAM factors are subjective. We created our own breast cancer risk assessment tool-specific guidelines for interpretation of the SAM factors in order to reach inter-rater agreement (see eText 3 in the Supplement). Inter-rater agreement of the SAM has not been validated in the past.
Webpages are constantly in flux, so the content and accessibility of sites could have changed since the conclusion of our evaluation. Additionally, this mode of risk communication is limited to individuals with access to Internet. Individuals with limited literacy rely more on interpersonal sources of information than their counterparts [8]. However, there is no reason to assume that clearer materials benefit only individuals with limited health literacy. Most patients prefer easy-to-read materials and guidelines suggest that health communication take “universal precautions” to ensure that materials are accessible for all individuals across literacy levels [15].
Conclusion
The readability of breast cancer risk assessment tools and the information accompanying those tools is critical to effective risk communication. Many studies have validated the tools, but none have yet to address the importance of making those tools accessible to our most at-risk population, adults with limited health literacy. Our study has elucidated the factors that affect whether a website achieves a superior readability score. If sites altered their content based on these scores, a larger audience of users could more readily understand crucial breast cancer risk information.
Electronic supplementary material
Acknowledgments
We thank Mary Politi, Ph.D. for content advice as a health literacy expert. There was no financial compensation.
Author Contributions
Dr. Cortez had full access to all the data in this study and takes responsibility for the integrity of the data and the accuracy of the data analysis.
Study concept and design: Colditz.
Acquisition, analysis, or interpretation of data: Cortez, Milbrandt, Kaphingst, and James.
Drafting of the manuscript: Cortez.
Critical revision of the manuscript for important intellectual content: Kaphingst.
Statistical Analysis: Cortez.
Obtained funding: Colditz.
Administrative, technical, or material support: Colditz.
Study Supervision: Colditz.
Disclaimer
The content is solely the responsibility of the authors and does not necessarily represent the official views of the NICHD, NCCR, or NIH.
Funding/Support
This research was supported in part by the Eunice Kennedy Shriver National Institute of Child Health & Human Development (Award numbers K23HD070979). Drs. Cortez, Colditz, James, and Ms. Milbrandt are supported by the Division of Public Health Sciences, the Siteman Cancer Center, and the Barnes Jewish Foundation. Dr. Colditz is supported by the Breast Cancer Research Foundation. Dr. Kaphingst is supported by the Huntsman Cancer Institute.
Role of the Funder/Sponsor
The funding source had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and the decision to submit the manuscript for publication.
Compliance with ethical standards
Conflict of Interest
Drs. Cortez, Colditz, James, and Ms. Milbrandt are supported in part by the Siteman Cancer Center. Dr. Colditz is supported by the Breast Cancer Research Foundation.
Contributor Information
Sarah Cortez, Phone: (248) 701-2713, Email: scortez@med.wayne.edu.
Graham Colditz, Email: colditzg@wustl.edu.
References
- 1.Siegel R, Naishadham D, Jemal A. Cancer statistics, 2013. CA Cancer J Clin. 2013;63(1):11–30. doi: 10.3322/caac.21166. [DOI] [PubMed] [Google Scholar]
- 2.American Cancer Society . Breast Cancer Facts & Figures 2013–2014. Atlanta: American Cancer Society, Inc.; 2013. [Google Scholar]
- 3.Hsieh CC, Trichopoulos D, Katsouyanni K, Yuasa S (1990) Age at menarche, age at menopause, height and obesity as risk factors for breast cancer: associations and interactions in an international case-control study. Int J Cancer J international du cancer 46 (5):796–800 [DOI] [PubMed]
- 4.Chen WY, Rosner B, Hankinson SE, Colditz GA, Willett WC. Moderate alcohol consumption during adult life, drinking patterns, and breast cancer risk. JAMA. 2011;306(17):1884–1890. doi: 10.1001/jama.2011.1590. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Amir E, Freedman OC, Seruga B, Evans DG. Assessing women at high risk of breast cancer: a review of risk assessment models. J Natl Cancer Inst. 2010;102(10):680–691. doi: 10.1093/jnci/djq088. [DOI] [PubMed] [Google Scholar]
- 6.Pew Research Center (2013) Health Online 2013. http://www.pewinternet.org/2013/01/15/health-online-2013/. Accessed Mar 4 2015
- 7.Berman ND, Sheridan SL, Donahue KE et al. (2011) Health Literacy Interventions and Outcomes: an updated systematic review. evidence report/technology assesment no. 199. (Prepared by RTI International–University of North Carolina evidence- based practice center under contract no. 290-2007-10056-I). AHRQ Publication Number 11- E006. Agency for Healthcare Research and Quality, Rockville
- 8.Kutner M, Greenberg E, Jin Y, Paulsen C. The Health literacy of America’s Adults: results from the 2003 National Assessment of Adult Literacy (NCES 2006–483) Washington, DC: U.S. Department of Education, National Center for Education Statistics; 2006. [Google Scholar]
- 9.Birru MS, Monaco VM, Charles L, Drew H, Njie V, Bierria T, Detlefsen E, Steinman RA. Internet usage by low-literacy adults seeking health information: an observational analysis. J Med Internet Res. 2004;6(3):e25. doi: 10.2196/jmir.6.3.e25. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Davis TC, Williams MV, Marin E, Parker RM, Glass J. Health Literacy and Cancer Communication. CA Cancer J Clin. 2002;52(3):134–149. doi: 10.3322/canjclin.52.3.134. [DOI] [PubMed] [Google Scholar]
- 11.Brewer NT, Tzeng JP, Lillie SE, Edwards AS, Peppercorn JM, Rimer BK. Health literacy and cancer risk perception: implications for genomic risk communication. Med Decis Mak. 2009;29(2):157–166. doi: 10.1177/0272989X08327111. [DOI] [PubMed] [Google Scholar]
- 12.Implications of health literacy for public health: workshop summary (2014) The National Academies Press, Institute of Medicine, Washington, DC [PubMed]
- 13.Sharp LK, Zurawski JM, Roland PY, O’Toole C, Hines J. Health literacy, cervical cancer risk factors, and distress in low-income African–American women seeking colposcopy. Ethn Dis. 2002;12(4):541–546. [PubMed] [Google Scholar]
- 14.Waters EA, Sullivan HW, Nelson W, Hesse BW. What is my cancer risk? How internet-based cancer risk assessment tools communicate individualized risk estimates to the public: content analysis. J Med Internet Res. 2009;11(3):e33. doi: 10.2196/jmir.1222. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Doak CC, Doak LG, Root JH. Teaching patients with low literacy skills. 2. Philadelphia: Lippincott; 1996. [Google Scholar]
- 16.McLaughlin GH. SMOG grading: a new readability formula. J Read. 1969;12(8):639–646. [Google Scholar]
- 17.Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006;33(3):352–373. doi: 10.1177/1090198105277329. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.