Skip to main content
NCBI home page
JBJS Open Access logoLink to JBJS Open Access
. 2020 Aug 19;5(3):e20.00054. doi: 10.2106/JBJS.OA.20.00054

Availability and Readability of Online Patient Information on Osteosarcoma

Assessment of Pediatric Hospital and National Cancer Institute-Designated Cancer Center (NCIDCC) Osteosarcoma Web Pages

Jason Young 1, Edward Christopher Dee 1, Collin May 1,2
PMCID: PMC7480967  PMID: 32984746

Background:

Online patient information (OPI) plays an important role in pediatric orthopaedic patient/caregiver education and decision-making. We assessed the availability and readability of OPI about osteosarcoma found at pediatric hospital and U.S. National Cancer Institute-designated cancer center (NCIDCC) websites.

Methods:

The websites of all NCIDCCs and the top 50 pediatric hospitals identified using U.S. News & World Report were included. The names of NCIDCCs and pediatric hospitals along with the terms “osteosarcoma,” “bone sarcoma,” and “sarcoma” were entered into Google.com, and were classified according to the availability of osteosarcoma-specific web pages. Unpaid monthly visits were assessed using the Ahrefs.com Organic Traffic Score (OTS) metric. Readability was assessed using 5 validated metrics and the composite grade level (CGL), the rounded mean of the 5 metrics.

Results:

Of the 71 NCIDCCs and 50 pediatric hospitals, 48 (67.6%) and 18 (36.0%), respectively, did not have at least 1 web page dedicated to osteosarcoma-specific OPI. The mean OTS for all 116 NCIDCC and pediatric hospital osteosarcoma-specific web pages assessed was 177 estimated visits per month, which was less than the mean OTS for the top 10 Google.com “osteosarcoma” search results (3,287.9; p < 0.001). The 52 NCIDCC web pages with osteosarcoma OPI (representing 23 centers) had an average CGL of 12.9, representing a readability level of at least a high school degree. The mean CGL for the 64 pediatric hospital web pages with osteosarcoma OPI (representing 32 hospitals) was 12.8, also representing readability of at least a high school degree. Only 8 (12.5%) of the 64 web pages were written at a seventh or eighth-grade level.

Conclusions:

Taken together, the majority of NCIDCCs and pediatric hospitals did not have a dedicated page of OPI for osteosarcoma. Of those that did, NCIDCC and pediatric hospital sites were visited much less frequently than sites visited through the most common Google.com searches. None of the osteosarcoma web pages offering OPI from NCIDCCs and pediatric hospitals met the American Medical Association readability recommendation (sixth-grade reading level). Therefore, greater effort must be made to create and direct patients and parents toward high-quality OPI of the appropriate level.

Clinical Relevance:

The clinical relevance of this study lies in its evaluation of OPI and its ability to impact the patient experience of clinical care.

The prevalence of the internet has democratized health information access1. Over three-quarters of Americans seek health information online2. Many patients turn to the internet as their initial source of online patient information (OPI) and can be influenced by what they find online before seeking providers1-4. This is also true in pediatric care, where the vast majority of parents use the internet to learn more about their children’s conditions5,6.

Despite the prevalence and influence of OPI use among patients and caregivers, studies in myriad fields across medicine7-10 have found the quality of OPI to be wanting. The findings of many studies suggest that providers circumvent this quality deficit by suggesting academic or hospital-associated websites to patients11,12. Specifically, some recommend websites created by educational institutions, often those ending in “.edu”13, or websites created by U.S. National Cancer Institute-designated cancer centers (NCIDCCs), as prior studies have identified them as providing some of the most accurate and evidence-based OPI14.

Despite the potential quality of these resources, less is known about the availability and readability of these OPI resources. OPI access is an important area of study, as only OPI that users encounter may influence and benefit them. An evaluation of readability, a metric that approximates the American grade level required to comprehend a text adequately, could identify OPI that may be too complex for most users14,15. The American Medical Association (AMA) recommends OPI written at the level of sixth grade to facilitate its utility for most U.S. users16.

Osteosarcoma is the most common primary bone malignancy in children and adolescents17, and among adolescents, it is one of the most common cancers18. Osteosarcoma care is often complex and multidisciplinary, highlighting the importance of appropriate patient information in treatment decisions. Therefore, we sought to assess the availability and readability of OPI about osteosarcoma made available by pediatric hospitals and NCIDCCs, not only as a test case for OPI that is accessible through NCIDCCs or pediatric hospitals, but also as one of the first assessments of OPI for this serious pediatric condition.

Materials and Methods

Institutional Identification

All NCIDCCs were included in the study and were identified using the “Find a Cancer Center” section of the NCI website, at: https://www.cancer.gov/research/nci-role/cancer-centers/find. Pediatric hospitals were identified using the U.S. News & World Report ranking system for cancer care; the top 50 pediatric cancer centers were identified using https://health.usnews.com/best-hospitals/pediatric-rankings/cancer. All institutions identified in the study are presented in Appendix Supplemental Tables 1 and 2.

Website Search Methodology

The names of NCIDCCs and pediatric hospitals along with the terms “osteosarcoma,” “bone sarcoma,” and “sarcoma” were entered into the Google search engine between December 29, 2019, and February 22, 2020. A single author (J.Y.) manually searched through website content provided by each NCIDCC and pediatric hospital to verify web page content, until a maximum of 5 pages per site were included, as done by Ammanuel et al.19 Findings were then verified by a second author (E.C.D.) for completeness. Pages that were not scientific papers were assumed to be patient information unless they explicitly specified that they were for providers. If an NCIDCC, a pediatric hospital, or its affiliated university or health-care system did not have at least 1 dedicated web page on the topic of osteosarcoma, it was excluded from further analyses. If an NCIDCC was affiliated with a pediatric hospital that was included in the analysis, or vice versa, the institutions were considered separately, given that—in all but 1 case—the NCIDCCs and pediatric hospitals hosted different websites despite having a common academic affiliation. Pages listing only osteosarcoma provider names or treatment locations were excluded. Patient stories and blog posts were also excluded. Web pages on osteosarcoma from websites associated with identified NCIDCCs and pediatric hospitals were evaluated for readability and website traffic.

OPI Availability Assessment

Websites of NCIDCCs and pediatric hospitals were assessed for the availability of osteosarcoma OPI. Websites were classified as 1 of the following: (1) having at least 1 osteosarcoma-specific web page, (2) having at least 1 paragraph but <1 dedicated page on osteosarcoma, (3) having at least 1 paragraph on sarcoma or bone cancer in general, but <1 paragraph on osteosarcoma, (4) having <1 paragraph of general sarcoma or bone cancer information, or (5) having no information at all on osteosarcoma, general sarcoma, or bone cancer.

Website Traffic Assessment

Website traffic was assessed using website analysis tools from Ahrefs, a website auditing and analytics company (ahrefs.com). Website traffic was assessed using the Organic Traffic Score (OTS) metric, which assigns an estimate of the number of organic (unpaid) page visits per month, based on a composite estimate of the traffic generated by the top 100 keywords associated with searches leading to the particular site over a 30-day average. By way of comparison, organic website traffic was also calculated for the top 10 web pages listed on Google.com from a search of the term “osteosarcoma” on March 5, 2020. The top 10 web pages were chosen because these represent the first page of Google.com results, and prior work has suggested that patients rarely search beyond the first Google.com page20. All explicit advertisements were excluded from this search.

Readability Assessment

Readability was assessed using the WebFX online tool (https://www.webfx.com/tools/read-able/), which assesses the readability of websites on the basis of the text presented. Readability was assessed using 5 validated metrics that make use of a combination of total words, sentences, and syllables to estimate the minimum grade level required to comprehend the text from each website. These included the Flesch-Kincaid Grade Level (FKGL), the Gunning Fog Score (GFS), the Simple Measure of Gobbledygook (SMOG) Index, the Coleman-Liau Index (CLI), and the Automated Readability Index (ARI). A higher grade level is correlated with an increased level of reading skill needed for text comprehension, where grade levels are equivalent to American grade levels16. Equations for each of these metrics are presented in Appendix Supplemental Figure 1.

To account for the variability among readability metrics, the mean of all 5 metrics, rounded to the nearest grade level, constituted the composite grade level (CGL), which has been reported previously8.

Statistical Analyses

OTS values were compared between NCIDCC and pediatric hospital websites, and between NCIDCC and pediatric hospital websites and the top 10 Google.com search results, using Student t tests. For a more conservative estimate of the OTS difference, NCIDCCs and pediatric hospitals that were also present in the top 10 Google.com search were included in both groups. T tests were also used to assess whether readability indices were at or below the sixth-grade level, the recommended OPI reading level per the AMA. We also used t tests to compare readability between NCIDCC websites and pediatric hospital websites across all 5 metrics and the CGL. Pearson correlation coefficients were calculated to assess the association between OPI readability metrics and OTS values for all NCIDCC and pediatric hospital websites. Statistical analyses were performed using Stata/SE (StataCorp), with 2-sided α = 0.05.

Results

Assessment of OPI Availability

Of the 71 NCIDCCs, 48 (67.6%) did not have at least 1 osteosarcoma-specific web page, and of the 50 pediatric hospitals, 18 (36.0%) did not have at least 1 web page dedicated to osteosarcoma OPI (Fig. 1). Of these 66 NCIDCCs and pediatric hospitals without a dedicated osteosarcoma-specific OPI web page, 8 (12.1%) had no information at all on osteosarcoma, general sarcoma, or bone cancer; 24 (36.4%) had some (<1 paragraph) general sarcoma or bone cancer information; 26 (39.4%) had at least 1 paragraph on sarcoma or bone cancer; and 8 (12.1%) had at least 1 paragraph but <1 page on osteosarcoma.

Fig. 1.

Fig. 1

Open in a new tab

The availability of online patient information (OPI) for NCIDCCs and pediatric hospitals. “With osteosarcoma OPI” = houses at least 1 page dedicated to osteosarcoma, “At least one paragraph on osteosarcoma” = houses at least 1 paragraph but <1 page on osteosarcoma, “At least one paragraph for sarcoma or bone cancer” = houses at least 1 paragraph for sarcoma or bone cancer, but <1 paragraph on osteosarcoma, “General sarcoma information” = houses <1 paragraph on sarcoma or general bone cancer, and “No information” = no information provided.

Website Traffic Assessment

The mean OTS for all 116 NCIDCC and pediatric hospital osteosarcoma-specific web pages assessed was 177.1, suggesting that approximately 177 visits per month were made to each web page, on average. The mean OTS for the NCIDCC websites was 207.9, and the mean OTS for the pediatric hospital websites was 152.2; there was no significant difference between the two (p = 0.69). In contrast, the mean OTS for the top 10 Google.com “osteosarcoma” search results was 3,287.9, which was significantly greater than that for the NCIDCC and pediatric hospital websites (p < 0.001) (Fig. 2). Of these 10, four were from NCIDCCs or pediatric hospitals (number 2: https://www.mayoclinic.org/diseases-conditions/osteosarcoma/symptoms-causes/syc-20351052, OTS = 6,500; number 4: https://www.stjude.org/disease/osteosarcoma.html, OTS = 2,800; number 9: https://www.hopkinsmedicine.org/health/conditions-and-diseases/sarcoma/osteosarcoma, OTS = 1,200; and number 10: https://www.dana-farber.org/bone-cancer-osteosarcoma/, OTS = 479).

Fig. 2.

Fig. 2

Open in a new tab

Comparison of mean monthly traffic estimates (and standard deviation) for the top 10 Google.com “osteosarcoma” search results and National Cancer Institute-designated cancer center (NCIDCC) and pediatric hospital (PH) web pages dedicated to osteosarcoma. Asterisks indicate a significant comparison (p = 0.05).

Of the NCIDCC web pages, 39 (75%) of 52 had an OTS of <10. Of the pediatric hospital web pages, 48 (75%) of 64 had an OTS of <10.

Readability Assessment

A total of 116 osteosarcoma-specific web pages representing 55 NCIDCCs and pediatric hospitals were assessed. The 52 NCIDCC web pages (from 23 centers) had a mean CGL of 12.9, representing at least a high school degree. The lowest reading level was tenth grade, for 8 (15.4%) of the 52 pages. For the 64 pediatric hospital web pages (from 32 hospitals), the mean CGL was 12.8, also representing at least a high school degree. Only 8 (12.5%) of the 64 pages were written at a seventh or eighth-grade level, and 65.6% required at least a high school degree to comprehend (Fig. 3). Among the websites identified in our top 10 Google.com search, the average CGL was 11.8. Reading metrics did not differ significantly from those of the NCIDCCs or pediatric hospitals (see Appendix Supplemental Table 3).

Fig. 3.

Fig. 3

Open in a new tab

Grade-level distributions of osteosarcoma-specific web pages. NCIDCC = National Cancer Institute-designated cancer center, and PH = pediatric hospital.

Across all 5 metrics and the CGL, the readability of the NCIDCC websites and the pediatric hospital websites was significantly greater than the AMA-recommended sixth-grade level (p < 0.001 for all) (Fig. 4).

Fig. 4.

Fig. 4

Open in a new tab

The readability of osteosarcoma online patient information found on National Cancer Institute (NCI)-designated cancer center and pediatric hospital (PH) web pages. Mean values (and standard deviation) are shown. The asterisk indicates a significant difference (p = 0.05). The bolded horizontal line indicates the AMA-recommended sixth grade reading level.

Only for the Coleman-Liau index (CLI) was a significant difference between NCIDCC and pediatric hospital websites demonstrated (NCIDCC CLI = 14.8, and pediatric hospital CLI = 13.7; p = 0.008); there were no significant differences across all other metrics (see Appendix Supplemental Table 4). Pearson correlation coefficients did not demonstrate significant associations between OTS and NCIDCC or pediatric hospital web page readability for any metric (see Appendix Supplemental Table 5).

Discussion

OPI plays an important role in patient education, and helps patients and their families to learn more about their conditions and make treatment decisions4-6,11. However, the availability and readability of OPI for pediatric cancers, including OPI provided by health-care institutions, remain poorly understood11. Consequently, we sought to assess the availability and readability of OPI provided by NCIDCCs and pediatric hospitals for osteosarcoma, given the complexity of care plans and the condition’s importance in pediatric orthopaedic oncology.

OPI Availability and Utilization

Our results suggest that OPI provided by NCIDCCs and pediatric hospitals remains limited in both availability and utilization. The websites of fewer than half of all NCIDCCs and pediatric hospitals provided at least 1 dedicated web page of osteosarcoma OPI; approximately half of the NCIDCCs and pediatric hospitals provided, at most, <1 paragraph of OPI specific to osteosarcoma. Additionally, organic traffic to the web pages of NCIDCCs and pediatric hospitals (OTS, 207.9 and 152.2, respectively) was significantly lower than traffic to sites populating the first page of search results on Google.com (3,287.9; p < 0.001), with 75% of the web pages from NCIDCCs and pediatric hospitals attracting <10 estimated organic visits per month. Prior studies have identified a wide variability in OPI quality and degree of bias but have found that information from academic and nonprofit websites is more accurate than OPI from media or private websites3. However, our results suggest that dedicated osteosarcoma OPI is not widely available from NCIDCCs or pediatric hospitals, and that when it is available, it is not frequently utilized compared with popular sites identified through a search on Google.com.

As noted in prior studies, the algorithms dictating what information appears on Google.com searches are not well understood by either physicians or patients11, and patients do not routinely search beyond the first page of results provided by a Google.com search20. Consequently, the current OPI that patients search for and access most frequently may not be screened for clinical accuracy or impartiality and may prove to be misleading or harmful to the patient’s treatment goals and expectations3. Web pages associated with NCIDCCs, pediatric hospitals, and their affiliated academic centers represent a privileged space where high-quality OPI can be provided in a way that bridges the care system, physicians, and clinical care received through information available online. Our findings suggest that one method of addressing this challenge would be for health-care institutions and physicians to expand access to, and increase utilization of, quality OPI provided on institutional websites in order to better empower patients in seeking care.

OPI Readability

Our study also demonstrated that, for institutions with at least 1 dedicated osteosarcoma web page, the level of OPI readability for both NCIDCCs and pediatric hospitals is too high, often requiring a reading level equivalent to a high school degree (mean CGL, 12.9 for NCIDCCs and 12.8 for pediatric hospitals). None of the web pages evaluated met the AMA-recommended sixth-grade reading level16. This finding is consistent with prior studies that demonstrated that OPI across many specialties and sources is too complex in terms of readability3,8,14,15. Additionally, the findings of this study are concordant with a prior evaluation of osteosarcoma OPI that demonstrated a high level of OPI complexity in the most popular web pages identified via search engines21; our study builds on this and other studies in highlighting that OPI from osteosarcoma treatment centers is also overly complex.

Health literacy in the U.S. remains low, with only about 1 in 10 Americans possessing proficient health literacy22. Prior studies have indicated that health information remains at a level that exceeds the literacy levels of most adults15. Understanding appropriate OPI is integral to its utility for patients; patients trust and adhere more to recommendations that they comprehend23,24. In contrast, poor understanding engenders dissatisfaction and compromises outcomes23,25, and OPI that is overly complex can negatively impact patients’ care-seeking and treatment decisions26,27 and may also perpetuate disparities8,28,29. Our results indicate that the challenges posed by overly complex OPI are relevant to, and affect, online information provided by hospitals and academic centers, suggesting that addressing this problem will require, at least in part, solutions coming from clinical and academic institutions.

Our findings highlight the possibility that the need for improvement in OPI availability and readability may extend to other pediatric conditions. The onus lies with health-care providers to point patients and parents toward OPI that is at the appropriate level and of good quality11. Additional studies should ascertain whether or not these findings apply to other relevant conditions.

Limitations

Our study is not without limitations. First, our reported OTS represents an estimate based on the Ahrefs algorithm and not actual counted page visits. It remains unclear how its keyword-based algorithm may or may not bias estimated page visits; however, we are not aware of any directional bias. Furthermore, we would not expect differential bias in our comparison of NCIDCC and pediatric hospital OPI and OPI identified in our Google.com search, and consequently, any bias would not change our findings that OPI provided by NCIDCCs and pediatric hospitals was much less viewed compared with that of the sites listed on the first Google.com page. Second, readability is an imperfect proxy for information comprehension15. In the absence of other validated metrics, however, readability remains a well-studied, objective measure of reading difficulty that can inform OPI design and identify areas of further study. Third, we did not perform an assessment of the accuracy of OPI provided by the NCIDCCs and pediatric hospitals. However, prior work has indicated that OPI provided by hospitals and academic institutions is among the highest quality available3. Fourth, comparing the OTS of the top 10 Google.com results and that of the NCIDCCs and pediatric hospitals is an imperfect measure, as the former is subject to change on the basis of targeted advertising as well as date, prior search history, and saved browser cookies30. However, we present the comparison between NCIDCC and pediatric hospital OPI and that of the top 10 Google.com sites to demonstrate that most NCIDCC and pediatric hospital OPI is not being utilized frequently relative to what is commonly being accessed by patients. We did not seek to assess the quality or origins of the top 10 Google.com sites themselves. Finally, it remains unclear whether findings related to osteosarcoma OPI translate to other conditions, and additional work will need to assess how these findings fit within a broader context of orthopaedic OPI.

Conclusions

In conclusion, our study highlights several challenges that educational and health-care institutions face in providing OPI on pediatric osteosarcoma. While there have been calls for educational and health-care institutions to play a more active role in providing quality OPI13,14, our study indicates that such OPI is not readily available, is poorly utilized, and is not easily readable by patients. Bolstering the availability and utilization of institutional OPI will be critical to properly addressing this problem, either through individual institutional efforts or through a coordinated, national-level program. However, additional work is needed, not only to evaluate the impact of such an intervention but also to improve the quality of OPI accessed by patients and caregivers and ultimately help better empower them to make informed decisions about their care.

Appendix

Supporting material provided by the authors is posted with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A200).

Footnotes

Investigation performed at Harvard Medical School, Boston, Massachusetts

Disclosure: The authors indicated that no external funding was received for any aspect of this work. The Disclosure of Potential Conflicts of Interest forms are provided with the online version of the article (http://links.lww.com/JBJSOA/A199).

References

  • 1.Tan SSL, Goonawardene N. Internet health information seeking and the patient-physician relationship: a systematic review. J Med Internet Res. 2017. January 19;19(1):e9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 2.Fox S. Health topics: 80% of internet users look for health information online. Pew Internet & American Life Project; 2011. [Google Scholar]
  • 3.Storino A, Castillo-Angeles M, Watkins AA, Vargas C, Mancias JD, Bullock A, Demirjian A, Moser AJ, Kent TS. Assessing the accuracy and readability of online health information for patients with pancreatic cancer. JAMA Surg. 2016. September 1;151(9):831-7. [DOI] [PubMed] [Google Scholar]
  • 4.Koch-Weser S, Bradshaw YS, Gualtieri L, Gallagher SS. The internet as a health information source: findings from the 2007 Health Information National Trends Survey and implications for health communication. J Health Commun. 2010;15(Suppl 3):279-93. [DOI] [PubMed] [Google Scholar]
  • 5.Wong MKY, Sivasegaran D, Choo CSC, Nah SA. Parental internet use and health information seeking behavior comparing elective and emergency pediatric surgical situations. Eur J Pediatr Surg. 2018. February;28(1):89-95. Epub 2017 Jun 29. [DOI] [PubMed] [Google Scholar]
  • 6.Hand F, McDowell DT, Glynn RW, Rowley H, Mortell A. Patterns of internet use by parents of children attending a pediatric surgical service. Pediatr Surg Int. 2013. July;29(7):729-33. Epub 2013 Apr 25. [DOI] [PubMed] [Google Scholar]
  • 7.Dee EC, Varady NH. Radiation oncology online: quality, strategies, and disparities. J Cancer Educ. 2019. June 3. Epub 2019 Jun 3. [DOI] [PubMed] [Google Scholar]
  • 8.Sha ST, Perni S, Muralidhar V, Mahal BA, Sanford NN, Nguyen PL, Dee EC. Trends, quality, and readability of online health resources on proton radiotherapy. Int J Radiat Oncol Biol Phys. 2020. May 1;107(1):33-8. Epub 2020 Jan 24. [DOI] [PubMed] [Google Scholar]
  • 9.Lawrentschuk N, Sasges D, Tasevski R, Abouassaly R, Scott AM, Davis ID. Oncology health information quality on the internet: a multilingual evaluation. Ann Surg Oncol. 2012. March;19(3):706-13. Epub 2011 Dec 7. [DOI] [PubMed] [Google Scholar]
  • 10.Lawrentschuk N, Abouassaly R, Hackett N, Groll R, Fleshner NE. Health information quality on the internet in urological oncology: a multilingual longitudinal evaluation. Urology. 2009. November;74(5):1058-63. Epub 2009 Sep 16. [DOI] [PubMed] [Google Scholar]
  • 11.Dee EC, Varady NH, Katz JN, Buchmiller TL. Disparity in online health information in pediatric vs. adult surgical conditions. Pediatr Surg Int. 2019. July;35(7):813-21. Epub 2019 Feb 15. [DOI] [PubMed] [Google Scholar]
  • 12.Hesse BW, Greenberg AJ, Rutten LJF. The role of internet resources in clinical oncology: promises and challenges. Nat Rev Clin Oncol. 2016. December;13(12):767-76. Epub 2016 Jun 7. [DOI] [PubMed] [Google Scholar]
  • 13.Chang DTS, Abouassaly R, Lawrentschuk N. Quality of health information on the internet for prostate cancer. Adv Urol. 2018. December 4;2018:6705152. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 14.Rosenberg SA, Francis D, Hullett CR, Morris ZS, Fisher MM, Brower JV, Bradley KA, Anderson BM, Bassetti MF, Kimple RJ. Readability of online patient educational resources found on NCI-designated cancer center web sites. J Natl Compr Canc Netw. 2016. June;14(6):735-40. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 15.Tran BNN, Ruan QZ, Epstein S, Ricci JA, Rudd RE, Lee BT. Literacy analysis of National Comprehensive Cancer Network patient guidelines for the most common malignancies in the United States. Cancer. 2018. February 15;124(4):769-74. Epub 2017 Nov 27. [DOI] [PubMed] [Google Scholar]
  • 16.Weiss BD. Health literacy: a manual for clinicians. American Medical Association Foundation and American Medical Association; 2003. [Google Scholar]
  • 17.Lindsey BA, Markel JE, Kleinerman ES. Osteosarcoma overview. Rheumatol Ther. 2017. June;4(1):25-43. Epub 2016 Dec 8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 18.Misaghi A, Goldin A, Awad M, Kulidjian AA. Osteosarcoma: a comprehensive review. SICOT J. 2018;4:12. Epub 2018 Apr 9. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Ammanuel SG, Edwards CS, Alhadi R, Hervey-Jumper SL. Readability of online neuro-oncology-related patient education materials from tertiary-care academic centers. World Neurosurg. 2020. February;134:e1108-14. Epub 2019 Nov 27. [DOI] [PubMed] [Google Scholar]
  • 20.Morahan-Martin JM. How internet users find, evaluate, and use online health information: a cross-cultural review. Cyberpsychol Behav. 2004. October;7(5):497-510. [DOI] [PubMed] [Google Scholar]
  • 21.Lam CG, Roter DL, Cohen KJ. Survey of quality, readability, and social reach of websites on osteosarcoma in adolescents. Patient Educ Couns. 2013. January;90(1):82-7. Epub 2012 Sep 5. [DOI] [PubMed] [Google Scholar]
  • 22.Cutilli CC, Bennett IM. Understanding the health literacy of America: results of the National Assessment of Adult Literacy. Orthop Nurs. 2009. Jan-Feb;28(1):27-32, quiz: :33-4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Bains SS, Bains SN. Health literacy influences self-management behavior in asthma. Chest. 2012. December;142(6):1687. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Rosas-Salazar C, Apter AJ, Canino G, Celedón JC. Health literacy and asthma. J Allergy Clin Immunol. 2012. April;129(4):935-42. Epub 2012 Feb 11. [DOI] [PubMed] [Google Scholar]
  • 25.American Medical Association; Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs. Health literacy: report of the Council on Scientific Affairs. Ad Hoc Committee on Health Literacy for the Council on Scientific Affairs, American Medical Association. JAMA. 1999. February 10;281(6):552-7. [PubMed] [Google Scholar]
  • 26.Castleton K, Fong T, Wang-Gillam A, Waqar MA, Jeffe DB, Kehlenbrink L, Gao F, Govindan R. A survey of internet utilization among patients with cancer. Support Care Cancer. 2011. August;19(8):1183-90. Epub 2010 Jun 18. [DOI] [PubMed] [Google Scholar]
  • 27.Shim M, Kelly B, Hornik R. Cancer information scanning and seeking behavior is associated with knowledge, lifestyle choices, and screening. J Health Commun. 2006;11(Suppl 1):157-72. [DOI] [PubMed] [Google Scholar]
  • 28.Hirschberg I, Seidel G, Strech D, Bastian H, Dierks ML. Evidence-based health information from the users’ perspective—a qualitative analysis. BMC Health Serv Res. 2013. October 10;13(1):405. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 29.Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011. July 19;155(2):97-107. [DOI] [PubMed] [Google Scholar]
  • 30.Hillyer GC, Beauchemin M, Garcia P, Kelsen M, Brogan FL, Schwartz GK, Basch CH. Readability of cancer clinical trials websites. Cancer Contr. 2020. Jan-Dec;27(1):1073274819901125. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from JBJS Open Access are provided here courtesy of Wolters Kluwer Health

RESOURCES