Abstract
Introduction
Prostate cancer patients are using more web resources to inform themselves about their cancer. However, patients may receive out-of-date or inaccurate information due to lack of regulation. The current study looks to systematically analyze the quality of websites accessed by patients with prostate cancer.
Methods
The term “prostate cancer” was searched in Google and the metasearch engines, Yippy and Dogpile, and the top 100 hits related to patient information were compiled from over 32 million hits. A standardized tool was used to examine 100 sites with respect to attribution, currency, usability, and content.
Results
Of the top 100 websites relating to prostate cancer information, only 27% identified an author, of which 16% had their credentials displayed. The majority of websites disclosed ownership (97%). Over half of the websites did not include the date of the last update and of those that did, only 66% were current within two years. According to the Flesch Kincaid grade level tool for readability, the majority (87%) of sites were found to be at a high school level, while 6% were at university level. Finally, content varied among websites; 90% of sites provided information on detection and workup and treatments, but only 14% of sites included information on prognosis.
Conclusions
The reliability of websites presenting prostate cancer information is questionable. There were noted deficiencies in attribution, currency, and readability. While information on detection and treatment is well-covered, information related to prognosis is lacking.
Introduction
Prostate cancer is the most prevalent cancer in Canada, excluding non-melanoma skin cancers. It constitutes up to 21% of newly diagnosed cancer cases annually and it is the third highest cause of cancer mortality in men.1 A diagnosis of cancer generates many questions and concerns that are often unique for each patient.2 To address their questions, patients commonly turn to the internet as one source of health information to make informed decisions about their healthcare.3 Accessing health information online is becoming increasingly popular.4 Statistics Canada reports that 70% of Canadians search medical or health-related information on the internet.5 With the increase in the use of the internet for health information, it is not surprising that urology patients also use the internet as a source of information for their health.6,7
Unlike peer-reviewed medical resources, the reliability and quality of healthcare information found on the internet is variable. There is little, if any, regulation of the quality of the content being published online.8 There is no single standardized and validated tool tailored for appraising internet-based health-related information and while some “stamps of approval” have been used, none of these are in routine use.9 As a result, it is up to the patient or the healthcare provider to make judgments with respect to the quality of a website. Prior studies have shown that the evaluation of several key features can help distinguish websites that may be of “higher quality or reliability.”9–11 These features can include declaration of website ownership, attribution (including clear authorship and citations), and a lack of commercial interest and sponsorship.10–12 With respect to attribution, more accurate information is often present in websites that contain references and citations compared to those that do not.13 Examining commercial interests is important, as websites may show a bias to support treatment with financial implications for their sponsors as distinguished from evidence-based data. For example, a study examining online information on proton therapy for prostate cancer showed a preferential bias of commercial websites to only discuss the benefits of proton-based treatments, with an apparent shift to marketing rather than patient education.14
In addition to key features that may help to distinguish the quality of websites, the readability of websites is critical for patient understanding. The American Medical Association (AMA) and National Institutes of Health (NIH) both recommend that patient materials be written at a grade six reading level.15,16 Despite this recommendation, many online patient education resources are written at a grade level far higher than grade six.17
The purpose of this study is to evaluate the quality of online prostate cancer patient education resources with respect to disclosure, attribution/currency, interactivity, readability, content, and accuracy. A previous study by Black and Penson a decade ago18 evaluated prostate cancer websites with respect to attribution and content. A more recent study evaluated the quality attributes of 13 prostate cancer websites with respect to congruence with established quality benchmarks (JAMA benchmark, DISCERN score, and HON code), as well as readability.19 This study expands on both of these studies, evaluating the “top 100” prostate cancer websites, with additional key areas relevant to quality, applying a validated rating tool,9 and taking into account inter-rater reliability. The results of this study will provide information with respect to both the strengths and weaknesses in the currently available online resources focused on prostate cancer, identify potential gaps in patients’ understanding of their cancer, and assist health professionals in recommending web-based resources to patients. Methods This study was conducted by applying a validated rating tool to evaluate the top 100 websites related to prostate cancer patient education. To develop a list of the top 100 websites, an internet search with the term “prostate cancer” was performed, with the meta-search engines Dogpile, Yippy, and Google in June 2016. We recorded up to the first 350 unique “hits” for each search engine to ensure that after the application of the exclusion criteria we would have enough websites to produce a list of 100. Websites were included in the analysis if they related to patient education and were accessible without a subscription. Exclusion criteria included websites not intended for the purposes of patient education, such as those intended specifically for healthcare professionals, or those dedicated only to advertisement without patient information. Inaccessible websites, as well as sites that only provided links to other sources and news articles, were also excluded. The list of the top 100 aggregate websites was assembled based on the average rank of each site within each result list produced by the search engines.
The top 100 websites were then evaluated using a structured rating tool9 to evaluate their disclosure, attribution/currency, interactivity, readability, content, and accuracy (Table 1). The structured rating tool was developed in 2009 and was adapted based on the Health on the Internet (HON) Foundation code,20 DISCERN scale,21 Abbott’s scale,22 JAMA benchmark criteria,23 and an evidence-based review of resources evaluating the quality of medical information on the internet. The tool has been used to evaluate the quality of resources for several cancers (colorectal, lung, breast, esophageal, pancreatic cancers, lymphoma, and melanoma) in an iterative approach and has been evaluated both for inter-rater reliability and usability.9 The various components of this tool have been validated elsewhere.20,21,23,24 Accountability criteria were derived from the HON code principles and the DISCERN scale, an instrument developed and implemented at Oxford University, intended to assist people without content expertise to evaluate the quality of a written health pub-lication.20,21 Interactivity criteria were based on adaptation of the Abbott’s scale22 evaluating the presence of audio or video support, a within-site search engine, education tools, discussion forums, and the possibility to send queries to the webmaster or authors. Readability was assessed using the Flesch-Kincaid (FK) grade level and readability ease, as well as the SMOG index, using the direct text input tool found on www.Read-able.com. For readability, sections from definition, diagnosis, and treatments were used, when possible, to generate the readability score. Content evaluation was done based on the presence or absence of material viewed as relevant to prostate cancer, as developed through consensus and consultation of evidence-based resources and content experts. For both accuracy and content, reference materials from the National Comprehensive Cancer Network (NCCN), as well as the peer-reviewed site UpToDate, were reviewed and summarized by the research assistant. Two oncologists then reviewed the summary of information from the NCCN and UpToDate, and through iterative discussion came to consensus on what essential areas should be included for complete patient information on prostate cancer (e.g., definition, staging, management) and similarly what level of detail was required under each heading to obtain a score of completely correct, mostly correct, or incorrect.
Table 1.
Components of evaluation tool
| Main component | Specifics areas assessed | Subcategory (as applicable) |
|---|---|---|
| Disclosure | Clarity of ownership | |
| Attribution | Website affiliation | |
| Authorship | Author identified | |
| Affiliations | ||
| Credentials | ||
| Attribution of sources | Sources cited | |
| Range of sources | ||
| Reliability of sources | ||
| Currency | Currency of website | Date of creation |
| Date of modification | ||
| Date of last update | ||
| Interactivity | Links | Number of links |
| Accessibility of links | ||
| Interactive components | Search engine | |
| Audio/visual support | ||
| Discussion board | ||
| Queries to webmaster | ||
| Educational support | ||
| Site organization | Headings | |
| Subheadings | ||
| Pictures/diagrams/tables | ||
| Hyperlinks | ||
| Absence of advertising | ||
| Readability | Flesch Kincaid | Flesch Kincaid grade level |
| Flesch Kincaid readability | ||
| Ease | ||
| SMOG | SMOG Index | |
| Coverage | Coverage congruent with predetermined criteria | Definition |
| Incidence/prevalence | ||
| Etiology/risk factors | ||
| Symptoms | ||
| Prevention | ||
| Detection/workup | ||
| Treatment | ||
| Prognosis | ||
| Accuracy | Accuracy congruent with predetermined criteria | Definition |
| Incidence/prevalence | ||
| Etiology/risk factors | ||
| Symptoms | ||
| Prevention | ||
| Detection/workup | ||
| Treatment | ||
| Prognosis | ||
| Global accuracy | ||
| Objectivity | Presence or absence of bias |
To determine the inter-rater reliability of website evaluation, the principal investigator and the research assistant used the structured rating tool to independently rate a random sample of 20 websites. The sample of websites was determined by assigning a number to each website and using an online random number generator to select the random set of sites. The inter-rater reliability for each item of the tool was compared, using the kappa statistic for the nominal ratings (e.g., yes/no), and the intraclass coefficient for continuous ratings (e.g., rating scales).25–27 For items on the tool having a reliability of <0.7, there was a review of the discrepancies in the ratings and a discussion among raters on how to improve consistency by modifying the rating scale and operationalizing definitions.28 Disagreements between raters was discussed and resolved by consensus. In this study, following the inter-rater reliability check after the first 20 websites, there were no items requiring revision and, as such, a single rater independently used the tool to evaluate the remaining 80 websites. The results were statistically evaluated using descriptive statistics.
Results
Searching “prostate cancer” on Google provided 31 900 000 hits, while Yippy yielded 109 527 and Dogpile an undisclosed total number. Over 1000 of these websites were recorded and after applying the inclusion and exclusion criteria, a list of the first 100 websites related to prostate cancer patient information was compiled.
Disclosure/affiliations
Affiliation of websites was analyzed by identifying both the type of site (e.g., educational, commercial, non-profit), as well as the disclosure of owner ship. Over half of the web-sites were commercially based, while non-profit sources made up 34% and government sites 12%. The remaining 2% were sponsored by academic or university sites. Almost all websites (97%) disclosed ownership or sponsorship. Overall, 86% of websites did not express an obvious bias.
Accountability
Accountability was evaluated with regards to disclosure of authorship, sources cited, links, and the dates of creation and last modification. Authorship of the websites was disclosed in only 27% of websites and only 16% provided the credentials of authors. Sixty percent of websites did not provide any references for their information and only 25% provided two or more reliable resources (e.g., scientific journals, peer-reviewed sites, academic or government sites, or textbooks). External links were provided on 38% of websites, and only 3% had 50% or fewer of them accessible. Finally, most websites (82%) provided a date of creation; however, only 39% indicated an update within the past four years.
Interactivity
The interactivity of websites was examined for patients’ ability to participate in learning within the website. The most common interactive tools were site-specific search engines, which were found in 80% of websites. Audio and visual support was supplied in 26% of websites, while only 8% provided a discussion board or forum, and only 6% included educational support, such as webinars. Thirty-three percent of sites included an opportunity for inquiries to the webmaster.
Site organization
Organization of each site was evaluated for including the following items: headings, subheadings, hyperlinks, pictures/diagrams/tables, and absence of advertising. All sites contained at least three of these items and 37% displayed all five of them.
Readability
The FK grade score level was measured, along with the FK readability ease and SMOG index. According to the FK grade score, most websites (93%) were at a high school level or higher. The greatest percentage of sites (87%) were written at the high school level (grade 8–12), while 6% were written at a university level (>13.00) and only 7% were written at an elementary school level. Similarly, the SMOG index indicated that 79% of websites were written at a high school or university level. Only 21% of sites were written at an elementary school level (<7.99) according to the SMOG index.
Content
Content was evaluated based both on coverage and accuracy. Coverage was evaluated based on the presence or absence of the following sections: a definition of prostate cancer, incidence and prevalence, etiology and risk factors, symptoms, prevention, diagnosis and workup, treatment, and finally prognosis. Diagnosis and workup and treatment were the most frequently covered topics, at 91% and 90%, respectively. Prognosis was the least covered topic, in only 14% of websites, with prevention being the second least, covered in only 38% of websites. Only 6% of websites covered all of the topics (Fig. 1).
Fig. 1.
Coverage by topic of the top 100 prostate cancer websites.
Accuracy of each section was evaluated and recorded as completely accurate, mostly accurate, or inaccurate or not present. As previously noted, accuracy was evaluated against predetermined criteria, as defined by the research team and congruent with NCCN guidelines, UpToDate, and expert opinion. Within each area of content, most sites that did provide information were completely accurate (Fig. 2). Missing information was the main cause for a lower accuracy scores. For example, with respect to information related to treatment, 72 of 90 websites were completely accurate, while 15 were mostly accurate, only three were mostly not accurate, and 10 contained no information (Fig. 2). With respect to incidence and prevalence, websites were required to present statistics relating to incidence and prevalence to be completely accurate. Fifty-five percent of websites were mostly accurate, with 28% providing all information needed to be considered completely accurate. Only 14% of websites covered prognosis and 8% provided information that was completely accurate. Global accuracy was a rater judgment of the consistency of all of the material on a website with reliable reference material on prostate cancer from UpToDate and NCCN. Most sites (89%) had a completely accurate global accuracy, and only 3% were globally mostly inaccurate.
Fig. 2.
Accuracy by topic of the top 100 prostate cancer websites.
Using the criteria for disclosure, attribution/currency, interactivity, readability, content and accuracy, an overall quality score can be generated for each website with a maximum value of 53. Of the top 100 websites relating to prostate cancer, the top website was www.bupa.co.uk, which had a score of 45. Table 2 shows the top 10 websites according to the overall quality. The lowest score was 11 from the website http://prostatitishome.com.
Table 2.
List of top 10 websites according to overall quality scores
Discussion
Patients use information to make their healthcare decisions and increasingly cancer patients are turning to the internet as a source of information about their disease and to make more informed decisions.3,29 Increasingly, prostate cancer patients are researching their disease online. However, the information presented to patients on the internet is difficult to regulate, and despite recommendations to require website accreditation, there are few standards for the quality of websites.30 Thus, it is generally left to the consumer or the health professional (i.e., urologist, oncologist, family physician) to navigate and evaluate the quality of the website resources. Our previous work led to the development and validation of a tool to evaluate the quality of online resources for cancer patients.9 The purpose of the present study was to apply this tool to evaluate the quality of prostate cancer websites intended for patient education, to evaluate accountability, accuracy, readability, and content covered.
Not surprisingly, the quality of prostate cancer websites evaluated in this study was variable. In this study, a minority of prostate cancer websites provided authorship (27%) or cited reliable sources (32%) for the information provided on their website. A lack of such markers of accountability within websites can be associated with a poorer quality of information compared to sites that do provide sources for their information.13 With new medical advances and the continuous release of new studies, it is critical that the information presented to patients is consistent with current standards and guidelines. In this study, fewer than 40% of websites had updated their information within the past four years. Studies indicate that such infrequent updating does not ensure that the information patients receive is current and may even be inaccurate.10,31,32 Previous studies, done over a decade ago, identified a similar lack of currency, attribution, and disclosure in prostate cancer web resources.18 Black and Penson found that 24 of 39 (61%) sites did not provide a date of last update and 28 of 39 (71%) did not provide references.18 This is similar to our findings that 56% of sites did not provide a date of last update and 68% of sites did not provide any reliable sources. It is remarkable that despite the passage of 10 years, there appears to be little improvement in the frequency of website updating and attribution. This situation provides the potential for the dissemination of out-of-date and low-quality information. Notably, in this study, the vast majority of sites declared ownership. This is consistent with prior studies where “the source of sponsorship was clearly indicated in most sites.”18 Sponsorship is often disclosed in the context of legal disclaimers. It is hypothesized that the high frequency of legal disclaimers leads to a high yield in sponsorship declaration.
Readability of websites is an important consideration for patient understanding the information presented. Almost half of Canadian adults (42%) have low literacy skills and this is predicted to increase in coming years with an aging population.33 Urology patients often represent an elderly population who may have even lower literacy skills.34 Using the FK grade level scoring and SMOG index, the vast majority of sites in this study have been shown to be written at a level more difficult than the average citizen can easily understand. In fact, by the FK grade level, only 7% of web-sites are written at the elementary grade level, which would not be in line with the recommendations of the NIH and the AMA for patient education materials. Inadequate clarity of written materials is a common finding among studies evaluating the readability of patient information materials and can lead to difficulties for patients in interpreting information.34–36 Readability is particularly important for prostate cancer patients, as there are many treatment options for patients and the choice of treatment is largely patient-driven.37 A recent study by Borgmann et al has also identified readability to be an area for improvement within popular prostate cancer websites accessed by patients.38 With most websites being written at an inappropriate reading level, steps will need to be taken to ensure that patients are accessing materials that are appropriate for their reading level and understanding. Online resources still have the advantage of providing individualized information for cancer patients, which is particularly important, as patient information needs depend on both the stage and trajectory of the disease.39 It is possible that a website tailored to individual patients could adjust the information presented based not only on tumour factors, such as disease stage, but also patient factors, such as literacy levels.
A previous study focused on the quality of information found on the internet, suggested that although the amount of information provided varied widely among websites, content was sufficient for patient decision-making.18 This is important, as Snow et al have reported that not all patients are receiving all of the information they think is required from their physicians to make treatment decisions.40 In these cases, it is possible that patients are then turning to the internet for additional information that their physicians did not provide. Fig. 2 indicates the percentage of websites that covered each area of content. Overall, websites provided information regarding aspects of prostate cancer, such as definition, diagnosis, symptoms, risk factors, and treatment options, which were described in over 70% of websites. However, coverage of both prevention and prognosis were markedly deficient, seen in only 38% and 14% of web-sites, respectively. This is an important inadequacy to recognize, as prognosis has previously been identified by cancer patients as an area of high interest.41–43
Aside from inclusion of content, it is important to evaluate the accuracy of content. The information presented on the 100 online resources was remarkably accurate and consistent with reference material in almost all cases, when it was covered. Over 89% of websites were completely accurate when rated for global accuracy, and only 3% were mostly not accurate. The content area most covered was diagnosis and workup; 76% of websites were completely accurate, 23% were mostly accurate, and only 1% was mostly not accurate. These findings are consistent with findings in the literature. Black and Penson found that over all categories of content, 94% of sites were “completely correct,” while 5% were “mostly correct” and only 1% was “mostly incorrect.”18 Not surprisingly, content areas more dependent on current information were more subject to inaccuracies. With respect to incidence and prevalence, websites were required to present current statistics to be completely accurate. In this context, 55% of websites were mostly accurate, with 28% providing all information needed to be considered completely accurate. This highlights the need for websites to be regularly maintained in order to provide accurate and current information.
This study has limitations. First, only websites in English were analyzed and information may vary for patients who do not speak English. Second, it is noted that for some search engines, geographic location can change list of hits. All searches were conducted from the same computer in the same geographic location on the same day. We attempted to minimize bias in the returned hits by using two meta-search engines and one search engine and then systematically combining all of the hits to develop a list of the “top 100 web-sites.” In the future, the same study could be repeated in different geographic locations to assess changes in the list of websites and their quality.
Our evaluation of completeness of the content covered may not reflect the purpose of the websites and what information they intended to provide, as some websites may have a narrow focus or intent. For the purposes of this study, we examined websites with “complete information” on all aspects of prostate cancer. Some patients may only seek information on a discrete subject area. As such, future research could modify the tool to evaluate the completeness and accuracy in the context of the information provided by the site and its intended purpose (i.e., management as a sole area of focus), acknowledging that not all websites are intended to provide a “complete picture.” Further, readability was measured only of sections relating to definition, diagnosis, and treatment, which may not be representative of the entire website.
When compiling our list of 100 websites, we excluded websites that were not intended to provide patient information (e.g., primary new articles, health professional sites, or sites for advertisement only). When recording our initial hits from all three search engines, we recorded approximately 350 hits per engine. After applying the exclusion criteria, we were left with 63% of the original hits from Google, 80% from Yippy, and 65% from Dogpile, which we then combined to a final list of 100 websites. Our results show that, on average, approximately a third of sites may appear in the search results but not be intended for patient information. Some of these websites appear in the first few pages of returned hits and many patients do not navigate beyond the first page of any search engine. Thus many patients may actually go to these excluded websites during their search process and future studies could use a modified tool to assess the accuracy of these “excluded” websites.
While these results appear to have been stable over the last decade since a similar study was conducted by Black and Penson,18 our study is a snapshot in time, and these results could change as websites are updated and new ones are formed. Finally, in the context of temporal change, patients are more commonly using social media to seek information. The intent of this study was to evaluate the quality of websites for patient information, but it is possible that our tool could be modified to adapt and evaluate the quality of social media sites.
Conclusion
Overall, the top 100 websites related to prostate cancer patient provide relatively consistent information relating to topics such as diagnosis and treatment; however, information relating to prognosis and prevention is lacking. While information on these sites is generally accurate, the content is generally found to be written at too difficult a level, which may hinder patient understanding and affect their treatment decisions. Almost all sites are written at a high school or greater level, which is above the standard guidelines for patient education. Urologists and oncologists caring for patients with prostate cancer need to be aware of the shortfalls in current online web resources and ensure that patients are receiving information that is tailored to their needs and adequate for their understanding to be able to aid in making informed decisions about their treatment.
Footnotes
Competing interests: The authors report no competing personal or financial interests.
This paper has been peer-reviewed.
References
- 1.Canadian Cancer Society’s Advisory Committee on Cancer Statistics. Canadian Cancer Statistics 2017. Toronto, ON: 2017. [Accessed Dec. 4, 2017]. Available at http://www.cancer.ca/~/media/cancer.ca/CW/cancer%20informa-tion/cancer%20101/Canadian%20cancer%20statistics/Canadian-Cancer-Statistics-2017-EN.pdf?la=enf. [Google Scholar]
- 2.Feldman-Stewart D, Brennenstuhl S, Brundage MD. The information needed by Canadian early-stage prostate cancer patients for decision-making: Stable over a decade. Patient Educ Couns. 2008;73:437–42. doi: 10.1016/j.pec.2008.07.008. [DOI] [PubMed] [Google Scholar]
- 3.Salonen A, Ryhä Nen AM, Leino-Kilpi H. Educational benefits of Internet and computer-based programs for prostate cancer patients: A systematic review. Patient Educ Couns. 2014;94:10–9. doi: 10.1016/j.pec.2013.08.022. [DOI] [PubMed] [Google Scholar]
- 4.Jacobs W, Amuta A, Jeon KC. Health information seeking in the digital age: An analysis of health information seeking behavior among US adults. J Cogent Soc Sci. 2017;3:e1302785. doi: 10.1080/23311886.2017.1302785. [DOI] [Google Scholar]
- 5.StatCan. Statistics Canada. Canadian Internet use survey. 2009. [Accessed Dec. 4, 2017]. Available at http://www.statcan.gc.ca/tables-tableaux/sum-som/l01/cst01/comm29a-eng.htm.
- 6.Abara E, Narushima M, Abara EO. Patterns of computer and internet usage among urology patients in two rural Northern Ontario communities. Can Urol Assoc J. 2010;4:37–41. doi: 10.5489/cuaj.772. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Phillips CA, Barz Leahy A, Li Y, et al. Relationship between state-level Google online search volume and cancer incidence in the United States: Retrospective study. J Med Internet Research. 2018;20:e6. doi: 10.2196/jmir.8870. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Tonsaker T, Bartlett G, Trpkov C. Health information on the internet: Gold mine or minefield? Can Fam Physician. 2014;60:407–8. [PMC free article] [PubMed] [Google Scholar]
- 9.Ingledew P-A, Regehr G, Tekian A, et al. Master’s thesis. 2015. Navigating the internet in patient-physician collaboration. Available at http://hdl.handle.net/10027/19489. [Google Scholar]
- 10.Martin-Facklam M, Kostrzewa M, Martin P, et al. Quality of drug information on the World Wide Web and strategies to improve pages with poor information quality. An intervention study on pages about sildenafil. Br J Clin Pharmacol. 2003;57:80–5. doi: 10.1046/j.1365-2125.2003.01954.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Martin-Facklam M, Kostrzewa M, Schubert F, et al. Quality markers of drug information on the Internet: An evaluation of sites about St. John’s wort. Am J Med. 2002;113:740–5. doi: 10.1016/S0002-9343(02)01256-1. [DOI] [PubMed] [Google Scholar]
- 12.Fogg BJ, Soohoo C, Danielson DR, et al. How do users evaluate the credibility of websites? A study with over 2500 participants. Proceedings of the Conference on Designing for User Experiences; 2003; [Accessed Dec. 4, 2017]. Available at https://dejanseo.com.au/media/pdf/credibility-online.pdf. [Google Scholar]
- 13.Ilic D, Risbridger G, Green S. Searching the internet for information on prostate cancer screening: An assessment of quality. Urology. 2004;64:112–6. doi: 10.1016/j.urology.2004.02.027. [DOI] [PubMed] [Google Scholar]
- 14.Sadowski DJ, Ellimoottil CS, Tejwani A, et al. Proton therapy for prostate cancer online: Patient education or marketing? Can J Urol. 2013;20:7015–20. [PubMed] [Google Scholar]
- 15.Weiss BD. Health Literacy and Patient Safety: Help Patients Understand (Manual for Clinicians, Second Edition) 2007. [Accessed Dec. 4, 2017]. Available at https://med.fsu.edu/userFiles/file/ahec_health_clinicians_manual.pdf.
- 16.MedlinePlus. How to Write Easy-to-Read Health Materials. US National Library of Medicine; 2012. [Accessed Dec. 4, 2017]. Available at https://medlineplus.gov/etr.html. [Google Scholar]
- 17.Byun J, Golden D. Readability of patient education materials from professional societies in radiation oncology: Are we meeting the national standard? IJORBP. 2015;91:1108–9. doi: 10.1016/j.ijrobp.2014.12.035. [DOI] [PubMed] [Google Scholar]
- 18.Black PC, Penson DF. Prostate cancer on the internet-information or misinformation? J Urol. 2006;175:1836–42. doi: 10.1016/S0022-5347(05)00996-1. [DOI] [PubMed] [Google Scholar]
- 19.Borgmann H, Wilm J-H, Vallo S, et al. Prostate cancer on the web-expedient tool for patients’ decision-making? J Cancer Educ. 2017;32:135–40. doi: 10.1007/s13187-015-0891-3. [DOI] [PubMed] [Google Scholar]
- 20.Health on the Net: HONcode Site Evaluation Form. Health on the Net Foundation; c:2017. [Accessed Aug. 31, 2018]. Available at http://www.hon.ch/cgi-bin/HONcode/Inscription/site_evaluation.pl?language=en&userCategory=individuals. [Google Scholar]
- 21.Charnock D, Shepperd S, Needham G, et al. DISCERN: An instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53:105–11. doi: 10.1136/jech.53.2.105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Abbott VP. Web page quality: Can we measure it and what do we find? A report of exploratory findings. J Public Health Med. 2000;22:191–7. doi: 10.1093/pubmed/22.2.191. [DOI] [PubMed] [Google Scholar]
- 23.Winker MA, Flanagin A, Chilum B, et al. Guidelines for medical and health information sites on the internet. JAMA. 2000;283:1600–6. doi: 10.1001/jama.283.12.1600. [DOI] [PubMed] [Google Scholar]
- 24.Silberg B, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the of medical information on the Internet: Caveant lector et viewor — let the reader and viewer beware. JAMA. 1997;277:1244–5. doi: 10.1001/jama.1997.03540390074039. [DOI] [PubMed] [Google Scholar]
- 25.Viera AJ, Garrett JM. Understanding interobserver agreement: The kappa statistic. Fam Med. 2005;37:360–3. [PubMed] [Google Scholar]
- 26.Cohen J. A coefficient of agreement for nominal scales. Educational and psychological measurement. Educ Psychol Meas. 1960;20:37–46. doi: 10.1177/001316446002000104. [DOI] [Google Scholar]
- 27.Rankin G, Stokes M. Reliability of assessment tools in rehabilitation: An illustration of appropriate statistical analyses. Clin Rehabil. 1998;12:187–99. doi: 10.1191/026921598672178340. [DOI] [PubMed] [Google Scholar]
- 28.Sagaram S, Walji M, Meric-Bernstam F, et al. Interobserver agreement for quality measures applied to online health information. Stud Health Technol Inform. 2004;107:1308–12. [PubMed] [Google Scholar]
- 29.Gwede CK, Pow-Sang J, Seigne J, et al. Treatment decision-making strategies and influences in patients with localized prostate carcinoma. Cancer. 2005;104:1381–90. doi: 10.1002/cncr.21330. [DOI] [PubMed] [Google Scholar]
- 30.Moretti FA, Oliveira VE, Silva EM. Access to health information on the internet: A public health issue? Rev Assoc Med Bras. 2012;58:650–8. doi: 10.1590/s0104-42302012000600008. [DOI] [PubMed] [Google Scholar]
- 31.Scholz-Crane A. Evaluating the future: A preliminary study of the process of how undergraduate students evaluate web sources. Ref Serv Rev. 1998;26:53–60. doi: 10.1108/00907329810307759. [DOI] [Google Scholar]
- 32.Chen LE, Minkes RK, Langer JC. Pediatric surgery on the Internet: Is the truth out there? J Pediatr Surg. 2000;35:1179–82. doi: 10.1053/jpsu.2000.8723. [DOI] [PubMed] [Google Scholar]
- 33.Reading the Future: Planning to Meet Canada’s Future Literacy Needs. Canadian Council of Learning; [Accessed Dec. 4, 2017]. Available at https://www.dartmouthlearning.net/wp-content/uploads/2013/02/Literacy-Reading-Future-Planning-to-Meet-Future-Literacy-Needs-in-Canada-Report.pdf. [Google Scholar]
- 34.Dalziel K, Leveridge MJ, Steele SS, et al. An analysis of the readability of patient education materials for common urologic conditions. Can Urol Assoc J. 2016;10:167–70. doi: 10.5489/cuaj.3578. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Johnson TV, Goodman M, Master VA. The efficacy of written screening tools in an inner city hospital: Literacy based limitations on patient access to appropriate care. J Urol. 2007;178:623–9. doi: 10.1016/j.juro.2007.03.118. [DOI] [PubMed] [Google Scholar]
- 36.Paasche-Orlow MK, Taylor HA, Brancati FL. Readability standards for informed-consent forms as compared with actual readability. N Engl J Med. 2003;348:721–6. doi: 10.1056/NEJMsa021212. [DOI] [PubMed] [Google Scholar]
- 37.Ellimoottil C, Polcari A, Kadlec A, et al. Readability of websites containing information about prostate cancer treatment options. J Urol. 2012;188:2171–5. doi: 10.1016/j.juro.2012.07.105. [DOI] [PubMed] [Google Scholar]
- 38.Borgmann H, Wolm JH, Vallo S, et al. Prostate cancer on the web — expedient tool for patients’ decision-making? J Canc Educ. 2017;32:135–40. doi: 10.1007/s13187-015-0891-3. [DOI] [PubMed] [Google Scholar]
- 39.Adams E, Boulton M, Watson E. The information needs of partners and family members of cancer patients: A systematic literature review. Patient Educ Couns. 2009;77:179–286. doi: 10.1016/j.pec.2009.03.027. [DOI] [PubMed] [Google Scholar]
- 40.Snow SL, Panton RL, Butler LJ, et al. Incomplete and inconsistent information provided to men making decisions for treatment of early-stage prostate cancer. Urology. 2007;69:941–5. doi: 10.1016/j.urology.2007.01.027. [DOI] [PubMed] [Google Scholar]
- 41.Nguyen SKA, Ingledew P-A. Tangled in the breast cancer web: An evaluation of the usage of web-based information resources by breast cancer patients. J Cancer Educ. 2013;28:662–8. doi: 10.1007/s13187-013-0509-6. [DOI] [PubMed] [Google Scholar]
- 42.Scali EP, Ingledew P-A. Melanoma patient information: Assessment of the quality of internet-based information resources. J Investig Med. 2011;59:183–4. [Google Scholar]
- 43.Hamilton SN, Scali EP, Yu I, et al. Sifting through it all: Characterizing melanoma patients’ utilization of the internet as an information source. J Cancer Educ. 2015;30:580–4. doi: 10.1007/s13187-014-0711-1. [DOI] [PubMed] [Google Scholar]