Abstract
Introduction
Testicular cancer is the most common solid malignancy diagnosed in young men aged 15–29. This population is also the age group that searches most actively for health information online. This study systematically evaluates the quality of websites available to patients with testicular cancer.
Methods
The term “testicular cancer” was inputted into the search engines Google, Dogpile, and Yippy. The top 100 websites intended for patient education were compiled. A validated structural rating tool was used to evaluate the websites with respect to attribution, currency, disclosure, interactivity, readability, and content.
Results
Less than half of the websites (44) disclosed authorship. Sixty-one websites provided a last modified date, and of those, 46 were updated in the last two years. The average readability level was 11.01 using the Flesh-Kincaid grade level system. The most accurate topic was treatment, with 82 websites being completely accurate and containing all required information. The least accurate topic was prognosis, with 27 being completely accurate.
Conclusions
These results show that authorship and currency are lacking in many online testicular resources, making it difficult for patients to validate the reliability of information. The high average readability of testicular cancer websites can affect comprehension. Topics such as prognosis were incompletely covered although represent an area for which patients often seek more information. These results can be used to counsel patients on the strength and weaknesses of online testicular cancer resources.
Introduction
The continuous growth of the internet has allowed increasing amounts of information to be available and accessible to the public, including medical information. Statistics Canada reports that over 70% of Canadians who use the internet have searched for health-related information.1,2 Increasingly, patients are turning to “Dr. Google” as the first source of information regarding even serious diseases, such as cancer — one of the top searched health conditions on Google.3
The patient population that searches most actively for health information online is young adults.4–6 Testicular cancer is the most common cancer diagnosed in men aged 15–29.7 Given the young age of testicular cancer patients, there is a high likelihood that the internet is a powerful tool for this young, internet-savvy population.
Little is known about the quality of online information about testicular cancer. Information quality can differ greatly among sources and patients may not have the necessary skills to evaluate the facts and apply it to their own health circumstances.8,9 This can lead to misinformation and confusion.10 It is essential to evaluate the quality of currently available online testicular cancer resources and identify potential gaps in knowledge so that patients and healthcare providers are aware of these shortcomings.
This study provides information to assess the strengths and weaknesses of online testicular cancer resources. Ultimately, this information will be useful for the development of new educational material and to improve patient-physician communication and education.
Methods
An internet search using the term “testicular cancer” was performed using one search and two metasearch engines. The search was performed on April 30, 2018 with Google; May 4, 2018 with Dogpile; and April 30, 2018 with Yippy. The metasearch engines Yippy and Dogpile combine results from several search engines, including Google, Yahoo, Bing, Yandex, etc.11 This helps to account for all potential results patients may obtain through using their search engine of choice. This also weights Google results more heavily, as Google is the most popular search engine and its results are, therefore, the most relevant for this purpose.12 All searches were conducted on a computer running macOS High Sierra using Chrome, the most popular cross platform internet browser.13 Searches were conducted in Chrome Incognito mode to prevent any personalized settings from affecting search results. The URLs of all results were recorded in the order they were ranked by the search engines.
Inclusion and exclusion criteria were applied to the three lists of search results. Websites were included if their URL was functional and contained testicular cancer information directed to patients. Websites were excluded if they were not intended for the purposes of general patient education. This included websites that were duplicates of a previous search result, the URL was broken, intended purely for fundraising/advertising, primary journal articles or for medical professionals, solely based on blogs or discussion boards, news stories, required paid subscription, not in English, or contained only links to other websites. After applying the inclusion/exclusion criteria, the lists were combined to provide an average rank order of websites from all three search engines. The first 100 websites were then taken from this list and will be referred to as the “top 100” for the purpose of this paper.
A previously validated, structured rating tool was applied to each of the top 100 websites.14 This tool was developed by our research group and assesses a website’s accountability, currency, interactivity, structure, readability, content coverage, and accuracy. The various components of the tool are described in prior publications.14–17 The tool was based on a thorough review of the literature and existing frameworks to evaluate website quality. The tool developed by our research team is essentially a “meta-tool” that combines existing tools, with additional criteria to address strengths and limitations identified in the literature. It has been iteratively developed with the principles of design-based research and applied to more than 10 cancer sites to date, with attention to stability of ratings and inter-rater reliability. The tool is adapted based on the Health on the Internet Foundation code, DISCERN criteria, JAMA benchmarks, and Abbott’s scale. A variation of the HONcode and DISCERN criteria was used to assess accountability. Interactivity and structural criteria were based on adaptation of the Abbot’s scale. Readability was assessed by copying and pasting the introduction and treatment sections of text into Read-able.com to calculate the Flesch-Kincaid (FK) grade level and Simple Measure of Gobbledygook Index. If no treatment section was present, the risk factor section was used instead.
Content and accuracy were evaluated by comparison of website content to materials determined by content experts to be informative for a patient’s understanding of testicular cancer. Testicular cancer information from the National Comprehensive Cancer Network (NCCN) and UpToDate was reviewed and summarized.18–20 These sources were used to be consistent with prior studies conducted by the research team.15–17 A practicing radiation oncologist with experience treating genitourinary malignancies then reviewed the summary and, through iterative discussions with the research team, a consensus document was completed. This document outlined the essential components required for content to be complete and the level of detail required for each accuracy level, ranging from incorrect to mostly correct to completely correct.
Using the structured rating tool, a score for each section (accountability, currency, interactivity, structure, readability, content coverage, and accuracy) was assigned for every website. A total score of each website was also calculated by adding all the section scores together. Data was analyzed with descriptive statistics in Microsoft Excel.
Results
The internet search yielded approximately 3 250 000 hits on Google and 8 340 212 hits on Yippy. The metasearch engine Dogpile did not disclose a total number of hits. All URLs provided by each of the search engines were recorded: 670 from Google, 259 from Yippy, and 610 from Dogpile. After applying inclusion and exclusion criteria, there were 107 valid websites from Google, 66 from Yippy, and 251 from Dogpile. The breakdown of the reasons for website exclusion are listed in Table 1. In total, there were 328 unique websites.
Table 1.
Breakdown of the number of hits excluded for each exclusion criteria
| Exclusion reason | # of hits excluded |
|---|---|
| Duplicate | 428 |
| Broken link | 35 |
| Fundraising/advertising/campaigns | 13 |
| For professionals/primary journal articles | 99 |
| Blogs/discussion boards | 78 |
| Contains links only | 14 |
| Contains only links to other publications | 35 |
| Purely news stories | 154 |
| Search engine/directories | 0 |
| Unrelated/not providing information | 199 |
| Requires subscription | 5 |
| Not English | 3 |
| Not a website (directs to pdf, YouTube video, Word document etc.) | 52 |
| Total excluded | 1115 |
Website affiliation
The most common website affiliation was commercial (49). Some examples of commercial websites are www.webmd.com and www.verywell.com. The second most common affiliation was non-profit organizations (39), followed by academic (6) and government (6). Some examples of nonprofit organization websites are http://www.cancer.org and www.testicularcancersociety.org.
Accountability
Accountability was evaluated through disclosure of authorship, citations, website creation and modification dates, and presence of external links. Forty-four websites identified an author and, of those websites, the author’s affiliation was provided in 21 and author’s credentials provided in 26.
Fifty-eight websites did not cite any sources. Of the websites with citations, one had no reliable sources, eight had only one reliable source, and 33 had two or more reliable sources cited. Reliable sources were defined as either scientific journal articles, peer-reviewed material, government or academic websites, and textbooks.
The date of website creation was identified on 48 websites, while 61 identified the date of last modification. Of the websites with modification dates, 46 websites were updated less than two years ago from the date of analysis, 14 between 2–4 years ago, and one over four years ago.
External links, excluding advertising, were present in 61 websites, with 57 websites containing two or more external links. The external links were followed to ensure current functionality. More than 50% of all external links were accessible in 57 websites, while less than 50% of external links were accessible in two websites. Two websites had no external links accessible at all.
Interactivity
Five components were considered when evaluating interactivity: presence of a search engine, audio/video support, presence of a discussion board/forum, ability to send queries to a medical professional (via web form, call, etc.), and educational support (such as online workshops, tools, etc.). The most common interactive component was a search engine, present in 81 of the 100 websites. Audio/video support was seen in 37 websites, while a discussion board/forum was identified in 19. Ability to send queries to a medical professional was provided in 20 websites, while educational support was seen in 25.
Site organization
The websites were evaluated for five structural tools, which consisted of headings, subheadings, pictures/diagrams/tables, hyperlinks, and absence of advertisement. All the websites had headings, while the vast majority (97) had subheadings as well. Pictures/diagrams/tables were seen in 87 websites, while 99 websites contained hyperlinks; 70 websites did not have any advertising.
Readability
Readability was analyzed using the online readability test tool www.read-able.com. Eighty-two websites were written at a high school grade level (FK grade level 8.00–12.99); 13 websites were written at a university level (FK grade level ≥13.00); and five were written at an elementary school grade level (FK ≤7.99). The average FK grade level was 11.02. The average SMOG grade level was 10.34.
Coverage and accuracy
Websites were assessed for their coverage of eight testicular cancer topics: definition, incidence/prevalence, etiology/risk factors, symptoms, prevention, detection/workup, treatment, and prognosis (Fig. 1). The most common topic covered was symptoms (97), followed by incidence/prevalence (95) and etiology/risk factors (90). The least covered topic was prevention (25), followed by prognosis (66).
Fig. 1.
Coverage of testicular cancer topics.
Accuracy of each topic was assessed by classifying each section as “completely accurate and containing all required information,” “mostly accurate and/or missing some required information,” or “completely inaccurate and/or missing all required information” (Fig. 2). Accuracy was evaluated against predetermined criteria that was in agreement with Uptodate.com, NCCN, and expert opinion. Most websites contained information that was correct. Lower accuracy scores were mostly due to missing required information rather than containing incorrect information. The most accurate topic was treatment, with 82 of the 86 websites that covered treatment classified as completely accurate. The least accurate topic was prognosis, with only 27 of the 66 websites that covered prognosis classified as completely accurate.
Fig. 2.
Accuracy of testicular cancer topics.
Global accuracy was evaluated as rater judgment of the congruence between overall website material with UpToDate and NCCN. Seventy-nine websites were classified as completely accurate globally, 19 as mostly accurate, and two as mostly not accurate. In terms of objectivity, 89 of 100 websites did not express any bias/opinion.
Overall quality
The total score for each website was calculated by totaling the scores in all sections evaluated. The maximum possible score was 55. The highest scoring websites were www.healthlinkbc.ca and myhealth.alberta.ca at 49, while the lowest scoring websites were www.nfcr.org/cancer-types/testicular-cancer and www.rd.com/health/conditions/testicular-cancer-symptoms at 16. The average total score was 34. Table 2 lists the top nine websites by score.
Table 2.
Top 9 testicular cancer websites based on overall quality score, and their Google search rank
Discussion
The internet is an important source of information for testicular cancer patients, yet studies evaluating the quality of online English testicular cancer information are limited. To our knowledge, there have only been two studies to date evaluating these online resources.21,22 Our study builds on these previous studies by using a validated rating tool to evaluate a larger sample size of top 100 websites intended for the purpose patient education.
Many patients use search engines rather than specific websites recommended by healthcare professionals to find healthcare information. The most popular search engine, Google, uses link popularity as one way to rank its search results.23 However previous studies have shown link popularity is not always associated with high-quality healthcare information. It is important that patients are aware of reliable approaches to evaluate the credibility of websites, as the information they obtain can have a significant impact on their treatment decisions and outcomes.24–26 Metrics such as whether authorship is disclosed, how current the site is, and whether references are cited can help patients make judgements about the quality of resources. Unfortunately, these metrics are lacking in many online testicular cancer resources. Of the top 100 websites evaluated, less than half (44) disclosed authorship and even fewer stated author credentials and affiliation. Similarity, less than half (46) of the websites were updated in the last two years. Only 42 websites provided source citation. These findings are similar to the results of a prior study that evaluated 62 patient-orientated websites returned by Google, Yahoo, and Bing when the search terms “testicular cancer,” “testicular tumour,” “testicular tumor,” “testicular malignancy,” “germ cell tumour,” and “germ cell tumor” were inputted.16 The study found that only 32% of websites evaluated provided authorship, 39% gave the date of last update, and 58% cited reliable data sources. Missing information such as this can make it difficult for patients to validate the reliability of testicular cancer information they find online and attention into the development of new online resources in these areas is warranted.
Another hurdle patients must overcome when searching for online testicular cancer information is comprehension. Readability is an important component of health literacy, and the National Institutes of Health, United States Department of Health and Human Services, and American Medical Association guidelines all recommend that patient information materials should be written at or below the grade 6 level to optimize patient understanding.27,28 Out of the 100 websites evaluated, only one was written at an FK grade level of 6, and the average FK grade level was 11.02. These results are consistent with previous findings that online health information, including information about urologic oncology, are written well above the recommended reading level.15,29,30 The high readability of testicular cancer websites can make it difficult for patients to interpret the information available.
Websites can improve readability by using graphics, which have been shown to improve patient understanding and retention.31–33 In addition, the use of analogies that are familiar and culturally appropriate for the target audience can also aid in comprehension.34 Substituting complex multisyllabic words for single-syllable words can reduce grade level readability.28 Some such substitutions may not be feasible due to a certain level of technical language required to accurately explain complex concepts.35 In these cases, websites can provide glossaries to define terms that may be unfamiliar to patients to address this problem. Lastly, alterations to sentence structure can also improve the readability of patient education material.36,37
Most of the information available on testicular cancer was factually correct, however, many sites were missing key pieces of information. Accurate but incomplete information can be misleading for internet consumers.38 For example, for the topic of prognosis, 36 of the 66 websites that covered prognosis were classified as mostly accurate and/or missing some required information. While many sites correctly stated that earlystage testicular cancer has a very good prognosis (five-year survival rate of >95%), few websites mentioned later-stage prognosis, which has a significantly lower five-year survival rate of 74%. These findings support previous studies, which conclude that while general information about testicular cancer can be found easily online, many topics contain incomplete information.21,22 One such study, which looked at the top 51 websites returned by Google when the search term “testicular cancer” was inputted, found that on average, 60% of the topics contained incomplete information and 12% of the topics contained inaccurate information when compared to the European Association of Urology guidelines.22
In addition to incomplete information, completely absent topics can be problematic as well. In this study, the least covered topic was prevention, with 75 websites neglecting to mention prevention at all. It is known that patients often go to the internet to seek how to prevent cancer.39 Although there is no known way to prevent testicular cancer and there are no risk factors that can be modified to mitigate the risk of getting testicular cancer, it is still important to note this to clarify any myths/misconceptions patients might have. Awareness of the current deficits in online testicular cancer patient information can help healthcare providers tailor their discussions with patients to better address potential gaps in knowledge that the internet does not cover, as well as guiding patients to trustworthy, high-quality websites.
Based on our analysis, we found that the highest scoring websites were www.healthlinkbc.ca and myhealth.alberta.ca at 49 points each. Both websites are Canadian and contain very similar content. These websites are maintained by the BC and Alberta governments, respectively, in partnership with trusted health information services in the province, such as the regional health authorities in BC and Alberta Health Services in Alberta. Canadian urologists, uro-oncologists, and other health professionals could potentially use these websites for the purpose of patient education in their practice and as examples of websites with high-quality features to help facilitate discussion with patients regarding content that may be useful to look for in other sites.
There are several limitations to this study. First, only English websites were evaluated in the study, and the quality of information available for non-English-speaking patients may vary. Readability was evaluated using representative sections of the websites rather than the full text. It is possible that readability of websites may have changed if we evaluated the full text, however, it would have been unrealistic for all 100 websites. Lastly, evaluating the completeness of websites can be misleading, as some websites may have only intended to cover specific topics. Future research could modify the tool to take into account the intended purpose of the site, rather than assuming all sites aim to provide information on all areas of testicular cancer.
Conclusions
In this study we systematically applied a validated rating tool to evaluate the quality of online testicular cancer information intended for patient education. We found that authorship, currency, and references are lacking in most online testicular cancer websites, which can make it difficult for patients to validate the credibility and trustworthiness of these resources. Almost all the websites evaluated were written at a readability level beyond the recommended reading level, which may hinder patient comprehension. Most testicular cancer websites had factually correct information, however, some were missing key pieces of information.
The results of this study can be used to counsel patients on the strength and weaknesses of online testicular cancer resources. It can also be used to develop new web-based educational resources and to improve patient-physician communication and education.
Footnotes
Competing interests: Dr. Eigl has received honoraria and travel support from Astellas, AstraZeneca, Bayer, Janssen, Merck, and Roche. The remaining authors report no competing personal or financial interests related to this work.
This paper has been peer-reviewed
References
- 1.Fox S. Washington DC: Pew Research Center; c2019. [Accessed May 29, 2019]. [updated 2011 Feb 1]. Available at: https://www.pewinternet.org/2011/02/01/health-topics-2/ [Google Scholar]
- 2.Statistics Canada. Ottawa, Ontario: Statistics Canada; [Accessed Dec. 4, 2018]. [updated 2019 May 29] Available at: https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=2210006601. [Google Scholar]
- 3.Hale TM, Pathipati AS, Zan S, et al. Representation of health conditions on Facebook: Content analysis and evaluation of user engagement. J Med Internet Res. 2014;16:182. doi: 10.2196/jmir.3275. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Cotten SR, Gupta SS. Characteristics of online and offline health information seekers and factors that discriminate between them. Soc Sci Med. 2004;59:1795–806. doi: 10.1016/j.socscimed.2004.02.020. [DOI] [PubMed] [Google Scholar]
- 5.Dubowicz A, Schulz PJ. Medical information on the internet: A tool for measuring consumer perception of quality aspects. Interact J Med Res. 2015;4:8. doi: 10.2196/ijmr.3144. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Abara E, Narushima M, Abara EO. Patterns of computer and Internet usage among urology patients in two rural Northern Ontario communities. Can Urol Assoc J. 2010;4:37–41. doi: 10.5489/cuaj.772. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Canadian Cancer Society. Toronto, Ontario: Canadian Cancer Society; c2019. [Accessed Dec. 4, 2018]. Available at: http://www.cancer.ca/en/cancer-information/cancer-type/testicular/risks/?region=on. [Google Scholar]
- 8.Tan SS, Goonawardene N. Internet health information seeking and the patient-physician relationship: A systematic review. J Med Internet Res. 2017;19:9. doi: 10.2196/jmir.5729. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Berland GK, Elliott MN, Morales LS, et al. Health information on the Internet: Accessibility, quality, and readability in English and Spanish. JAMA. 2001;285:2612–21. doi: 10.1001/jama.285.20.2612. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Ahmad F, Hudak PL, Bercovitz K, et al. Are physicians ready for patients with Internet-based health information? J Med Internet Res. 2006;8:22. doi: 10.2196/jmir.8.3.e22. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.University of California Berkely. Berkely, CA: Regents of the University of California; c2010. [Accessed Sept 25, 2019]. [updated 2010 Aug 18]. Available at: https://laulima.hawaii.edu/access/content/group/8fc98f26-7b72-42c1-0041-108f739bef82/internet/ucb_metasearch.html. [Google Scholar]
- 12.Statcounter. c1999–2019. [Accessed Sept. 25, 2019]. Available at: https://gs.statcounter.com/search-engine-market-share.
- 13.Statista. New York, NY: Statista; c2019. [Accessed Dec. 4, 2018]. Available at: https://www.statista.com/statistics/186150/most-popular-browsers-in-canada-by-market-share/ [Google Scholar]
- 14.Ingledew P-A, Regehr G, Tekian A, et al. Navigating the Internet in patient-physician collaboration (Master’s thesis) UIC; 2015. [Accessed Dec. 4, 2018]. Available at: http://hdl.handle.net/10027/19489. [Google Scholar]
- 15.Kobes K, Harris IB, Regehr G, et al. Malignant websites? Analyzing the quality of prostate cancer education web resources. Can Urol Assoc J. 2018;12:344–50. doi: 10.5489/cuaj.5084. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Chang KL, Grubbs EG, Ingledew P-A. An analysis of the quality of thyroid cancer websites. Endocr Pract. 2019;25:1003–11. doi: 10.4158/EP-2018-0595. [DOI] [PubMed] [Google Scholar]
- 17.De Groot L, Harris I, Regehr G, et al. Quality of online resources for pancreatic cancer patients. J Cancer Educ. 2019;34:223–8. doi: 10.1007/s13187-017-1290-8. [DOI] [PubMed] [Google Scholar]
- 18.NCCN.org. Plymouth Meeting, PA: National Comprehensive Cancer Network; c2018. [Accessed May 16, 2018]. Available at: https://www.nccn.org/professionals/physician_gls/pdf/testicular.pdf. [Google Scholar]
- 19.UpToDate. c2018. [Accessed May 16, 2018]. Available at: https://www.uptodate.com/contents/clinical-manifestations-diagnosisand-staging-of-testicular-germ-cell-tumors.
- 20.UpToDate. c2018. [Accessed May 16, 2018]. Available at: https://www.uptodate.com/contents/testicular-cancer-beyond-the-basics.
- 21.Prasanth AS, Jayarajah U, Mohanappirian R, et al. Assessment of the quality of patient-oriented information over internet on testicular cancer. BMC Cancer. 2018;18:491. doi: 10.1186/s12885-018-4436-0. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Paffenholz P, Salem J, Borgmann H, et al. Testicular cancer on the web – an appropriate source of patient information in concordance with the European Association of Urology guidelines? J Cancer Educ. 2017;33:1314–22. doi: 10.1007/s13187-017-1249-9. [DOI] [PubMed] [Google Scholar]
- 23.Page L, Brin S, Motwani R, et al. The PageRank citation ranking: Bringing order to the web (Technical report) Standord InfoLab; 1999. [Accessed Dec. 4, 2019]. Available at: http://ilpubs.stanford.edu:8090/422/ [Google Scholar]
- 24.Kitchens B, Harle CA, Li S. Quality of health-related online search results. Decis Support Syst. 2014;57:454–62. doi: 10.1016/j.dss.2012.10.050. [DOI] [Google Scholar]
- 25.Zhang Y. Searching for specific health-related information in MedlinePlus: Behavioral patterns and user experience. J Assoc Inf Sci Technol. 2014;65:53–68. doi: 10.1002/asi.22957. [DOI] [Google Scholar]
- 26.Fahy E, Hardikar R, Fox A, et al. Quality of patient health information on the Internet: Reviewing a complex and evolving landscape. Australas Med J. 2014;7:24–8. doi: 10.4066/AMJ.2014.1900. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Rootman I, Gordon-El-Bihbety D. Ottawa, Ontario: Canadian Public Health Association; c2008. [Accessed Dec. 4, 2018]. Available at: https://www.cpha.ca/sites/default/files/uploads/resources/healthlit/report_e.pdf. [Google Scholar]
- 28.Dalziel K, Leveridge MJ, Steele SS, et al. An analysis of the readability of patient information materials for common urological conditions. Can Urol Assoc J. 2016;10:167–70. doi: 10.5489/cuaj.3578. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 29.Pruthi A, Nielsen ME, Raynor MC, et al. Readability of American online patient education materials in urologic oncology: A need for simple communication. Urology. 2015;85:351–6. doi: 10.1016/j.urology.2014.10.035. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Walsh TM, Volsko TA. Readability assessment of internet-based consumer health information. Respir Care. 2008;53:1310–5. [PubMed] [Google Scholar]
- 31.Houts PS, Witmer JT, Egeth HE, et al. Using pictographs to enhance recall of spoken medical instructions II. Patient Educ Couns. 2001;43:231–42. doi: 10.1016/S0738-3991(00)00171-3. [DOI] [PubMed] [Google Scholar]
- 32.Houts PS, Bachrach R, Witmer JT, et al. Using pictographs to enhance recall of spoken medical instructions. Patient Educ Couns. 1998;35:83–8. doi: 10.1016/S0738-3991(98)00065-2. [DOI] [PubMed] [Google Scholar]
- 33.Mansoor LE, Dowse R. Effect of pictograms on readability of patient information materials. Ann Pharmacother. 2003;37:1003–9. doi: 10.1345/aph.1C449. [DOI] [PubMed] [Google Scholar]
- 34.Centers for Disease Control and Prevention. Atlanta, Georgia: Centers for Disease Control and Prevention; 2009. [Accessed March 4, 2019]. Available at: https://www.cdc.gov/healthliteracy/pdf/simply_put.pdf. [Google Scholar]
- 35.Storino A, Castillo-Angeles M, Watkins AA, et al. Assessing the accuracy and readability of online health information for patients with pancreatic cancer. JAMA Surg. 2016;151:831–7. doi: 10.1001/jamasurg.2016.0730. [DOI] [PubMed] [Google Scholar]
- 36.Horner SD, Surratt D, Juliusson S. Improving readability of patient education materials. J Community Health Nurs. 2000;17:15–23. doi: 10.1207/S15327655JCHN1701_02. [DOI] [PubMed] [Google Scholar]
- 37.Sheppard ED, Hyde Z, Florence MN, et al. Improving the readability of online foot and ankle patient education materials. Foot Ankle Int. 2014;35:1282–6. doi: 10.1177/1071100714550650. [DOI] [PubMed] [Google Scholar]
- 38.Black PC, Penson DF. Prostate cancer on the Internet – information or misinformation? J Urol. 2006;175:1836–42. doi: 10.1016/S0022-5347(05)00996-1. discussion 42. [DOI] [PubMed] [Google Scholar]
- 39.Lee C-J, Niederdeppe J, Freres D. Socioeconomic disparities in fatalistic beliefs about cancer prevention and the Internet. J Commun. 2012;62:972–90. doi: 10.1111/j.1460-2466.2012.01683.x. [DOI] [PMC free article] [PubMed] [Google Scholar]