Abstract
Purpose:
To assess content, readability, and accountability of online information for patients regarding epiretinal membranes (ERMs).
Methods:
Cross-sectional study evaluating nine major medical websites on ERMs. Fifteen questions assessed patient-relevant content. Four indices estimated U.S. grade literacy level of the text. JAMA benchmarks (authorship, attribution, disclosure, currency) evaluated website accountability.
Results:
Average content score was 36.78 (SD 13.91, CI 95% ±0.64) from a possible maximum of 60, with significant variability between websites (H=22.68, p=0.004). Mean reading grade level was 12.29 (SD 2.30, CI 95% ±1.50). No website achieved all JAMA benchmarks; only one website fulfilled three of the four. Content score did not correlate with Google rank (order of listed websites, r=−0.23, p=0.55) or JAMA benchmarks (r=0.19, p=0.62) but significantly correlated with mean reading grade (r=0.67, p=0.05).
Conclusion:
Online information regarding ERMs varies significantly, may not adequately answer common patient questions, and is written at too complex a literacy level for the average patient.
Keywords: consumer health informatics, epiretinal membrane, Internet, patient education
Introduction
Epiretinal membranes (ERMs) are common retinal conditions, affecting nearly 15% of the general population in the United States and increasing in prevalence with age.1 Advanced ERMs can cause macular edema, metamorphopsias, and central vision impairment, at times necessitating surgical intervention. Surgical treatment for ERMs, which includes pars plana vitrectomy and peeling, has been common since the 1970s.2 However, in many cases, ERMs do not significantly affect vision, and 65% are either nonprogressive or spontaneously regress over five years.3
The main indication for ERM surgery is decreased visual acuity, but specific indications are not standardized, and clinical outcomes vary. Many studies have examined predictive factors for postoperative visual acuity (VA) following ERM surgery, and a review of these studies found that only preoperative VA was consistently associated with postoperative VA, while other potential factors, like central foveal thickness, inner segment/outer segment integrity on optical coherence tomography, and severity of metamorphopsia were not.4 As such, the decision to surgically intervene depends on the potential benefits of reduced visual impairment from removal of the ERM versus the risks of surgery, which include retinal detachment and endophthalmitis. Particularly in cases of ERMs, the patient’s subjective judgement heavily determines the course of action.5
According to a 2013 study conducted by the Pew Research Center, 72% of U.S. adults who used the internet reported searching online for health information within the past year,6 and the numbers continue to increase.7 Studies have demonstrated the internet can affect patient decision-making for a variety of diseases.8–10 Patients with ERMs may choose to consult medical websites in order to supplement information provided by their ophthalmologist. This may be done for educational reasons or to help with clinical decision-making.
While information on the internet can increase patient education, no standard exists across online resources, and online information can vary widely.11 No studies have examined online resources regarding ERMs, and given the subjectivity and variability in the decision to treat ERMs in many cases, it is important to be aware of the resources that may be influencing patient decision-making regarding ERMs. Our study evaluated freely available information on ERMs by analyzing the content, readability, and accountability of commonly used medical websites.
Methods
Website Selection
“Epiretinal membrane” was entered into the Google search engine, and major medical websites were selected for analyses (the American Academy of Ophthalmology (AAO), All About Vision, the American Society of Retinal Surgeons (ASRS), EyeWiki, Mayo Clinic, Medical News Today, Medscape, the National Eye Institute (NEI), and Wikipedia), similar to a prior study.11 Google rank, as determined by placement of website in the resultant search list, was noted. Each website was independently assessed by three evaluators: two vitreoretinal surgeons (JS and AEK) and one vitreoretinal surgery fellow (NP).
Content
Fifteen questions were composed by the authors to address clinical content most relevant to patients (Table 1). A grading scale of 0–4 points was used for the website’s accuracy, completeness, and clarity for addressing each question – with 4 indicating an accurate and thorough response that was focused and organized and 0 indicating no information relevant to the question was provided. Total scores were determined by adding points for all questions, yielding a minimum possible score of 0 and maximum of 60.
Table 1:
Mean Content Scores by Question and Website Regarding Epiretinal Membranes (ERMs)
| AAO | All About Vision | ASRS | EyeWiki | Mayo Clinic | Medical News Today | Medscape | NEI | Wikipedia | |
|---|---|---|---|---|---|---|---|---|---|
| What is an ERM? | 3.33 | 3.67 | 4.00 | 4.00 | 3.00 | 3.67 | 4.00 | 3.33 | 4.00 |
| Which age group most commonly presents with ERMs? | 2.00 | 0.33 | 4.00 | 3.67 | 0.00 | 4.00 | 4.00 | 1.33 | 3.33 |
| What is the prevalence of ERMs? | 0.00 | 0.33 | 3.67 | 2.67 | 0.00 | 3.67 | 4.00 | 0.00 | 3.67 |
| What are the causes of ERMs? | 3.00 | 3.00 | 4.00 | 3.33 | 0.00 | 3.33 | 4.00 | 3.00 | 4.00 |
| What are risk factors associated with the development of ERMs? | 3.00 | 4.00 | 3.67 | 3.67 | 0.00 | 4.00 | 4.00 | 1.33 | 3.00 |
| What are the symptoms of ERMs? | 3.67 | 3.67 | 4.00 | 3.33 | 2.33 | 3.67 | 4.00 | 4.00 | 3.67 |
| How are ERMs diagnosed? | 3.00 | 1.00 | 4.00 | 3.67 | 0.00 | 3.67 | 3.67 | 0.67 | 2.67 |
| What is an Amsler grid? | 0.33 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 2.67 |
| What is optical coherent tomography, and how is it used in the diagnosis of ERMs? | 2.00 | 0.00 | 3.67 | 3.33 | 0.00 | 3.67 | 3.67 | 0.00 | 2.00 |
| What are the treatment options for ERMs? | 3.33 | 3.33 | 3.67 | 3.67 | 0.00 | 3.00 | 4.00 | 3.33 | 3.00 |
| What is pars plana vitrectomy? | 3.67 | 3.67 | 3.67 | 2.00 | 0.00 | 3.00 | 4.00 | 2.67 | 3.33 |
| What are the complications associated with vitrectomy? | 2.67 | 3.67 | 3.67 | 3.33 | 0.00 | 3.33 | 4.00 | 2.33 | 3.67 |
| What are the visual outcomes after removal of ERMs? | 1.00 | 4.00 | 2.33 | 3.33 | 0.00 | 2.00 | 4.00 | 3.00 | 3.00 |
| Are ERMs progressive? | 1.33 | 0.67 | 3.00 | 1.67 | 0.00 | 3.00 | 1.67 | 2.67 | 1.67 |
| What is involved in recovery after vitrectomy? What are the indications for special positioning after it? Are there travel restrictions? | 0.67 | 2.33 | 0.00 | 0.33 | 0.00 | 3.00 | 1.00 | 1.00 | 0.33 |
| Total Points | 33.00 | 33.67 | 47.33 | 42.00 | 5.33 | 47.00 | 50.00 | 28.67 | 44.00 |
| Percentage (Out of Max 60) | 55% | 56% | 79% | 70% | 9% | 78% | 83% | 48% | 73% |
| Mean Points | 2.20 | 2.24 | 3.16 | 2.80 | 0.36 | 3.13 | 3.33 | 1.91 | 2.93 |
| SD | 1.21 | 1.58 | 1.31 | 1.20 | 0.91 | 0.97 | 1.26 | 1.32 | 0.95 |
| 95% CI | 0.61 | 0.80 | 0.66 | 0.61 | 0.46 | 0.49 | 0.64 | 0.67 | 0.48 |
Scores of individual questions scaled from 0–4. AAO = American Academy of Ophthalmology, ASRS = American Society of Retinal Surgeons, NEI = National Eye Institute.
Readability
Readability reflected the level of complexity of language and was measured using an online tool (Readable).12 Four readability measurements were used to determine the United States reading grade level of the text: Flesch Kincaid Grade Level, Gunning Fog Index, Coleman Liau Index, and Simple Measure of Gobbledygook (SMOG) Index. A mean reading grade represented an average grade of all indices.
Accountability
Accountability of websites was determined by The Journal of the American Medical Association (JAMA) benchmarks.13 These benchmarks evaluated provision of authorship (authors, contributors, their affiliations, and relevant credentials), attributions (references and relevant copyright information), currency (dates of when content was last updated and posted), and disclosure (website ownership, sponsorship, advertising, underwriting, commercial funding arrangements or support, and potential conflicts of interest). Benchmarks were scored as fulfilling the criteria (score of 1) or not fulfilling the criteria (score of 0), with total possible scores of 0–4.
Statistical Analyses
Statistical analyses were performed with the SPSS V.26.0 statistical package. Content scores were compared between websites using Kruskal-Dunn tests. Spearman’s coefficient was used for interrater agreeability and correlations between scores and Google rank. Statistical significance was set at p≤0.05.
Results
Website Grader Reproducibility
Nine websites were analyzed in the study. A Spearman correlation analysis was used to assess the interobserver reproducibility between each website grader. There were statistically significant correlations between the three graders with high agreeability (r=0.75, p=0.02 between JS and AEK; r=0.92, p<0.001 between JS and NP; and r=0.84, p=0.004 between NP and AEK), indicating consistent grading between the three observers.
Content Analysis
The average content score for all websites was 36.78 (SD 13.91, CI 95% ±0.64) out of 60 possible points using the scoring rubric created to assess the quality of website information. The scores ranged from 5.3 points to 50 points. Medscape.com scored the highest with 50 points. Kruskal -Wallis post hoc Dunn analysis was used to assess for variability in content score between websites. There was statistically significant variability between the content scores of websites (H=22.68, p=0.004). Table 1 shows the mean content scores by question for each website, and Table 2 shows website rankings based on total content score. There was no significant correlation between rank on the search engine Google.com and content score (r=−0.23, p=0.55) using Spearman correlation analysis.
Table 2:
Total Content Scores by Website Regarding Epiretinal Membranes
| Website | Total Score | % of Maximum (60 points) | Google Rank* |
|---|---|---|---|
| Medscape | 50 points | 83% | 14 |
| ASRS | 47.3 points | 79% | 1 |
| Medical News Today | 47 points | 78% | 3 |
| Wikipedia | 44 points | 73% | 18 |
| EyeWiki | 42 points | 70% | 9 |
| All About Vision | 33.7 points | 56% | 86 |
| AAO | 33 points | 55% | 36 |
| NEI | 28.7 points | 48% | 12 |
| Mayo Clinic | 5.3 points | 9% | 6 |
ASRS = American Society of Retinal Surgeons, AAO = American Academy of Ophthalmology, NEI = National Eye Institute.
Google rank determined by placement of website in resultant search list after “epiretinal membrane” was entered into the Google search engine.
Readability Analysis
The mean reading grade for all websites was 12.29 (SD 2.30, CI 95% ±1.50). The reading grade levels ranged from 9.2 to 15.6 reading grade level. Using Spearman correlation analysis, there was a significant correlation between content score and mean reading grade (r=0.67, p=0.05), indicating that higher reading difficulty was associated with higher content score. Table 3 shows readability scores by website.
Table 3:
Reading Grade by Website Regarding Epiretinal Membranes
| AAO | All About Vision | ASRS | EyeWiki | Mayo Clinic | Medical News Today | Medscape | NEI | Wikipedia | |
|---|---|---|---|---|---|---|---|---|---|
| Flesch Kincaid Grade Level | 8.3 | 12.2 | 11.2 | 12.9 | 8.8 | 8.6 | 14.5 | 10 | 11.8 |
| Gunning Fog Score | 11.4 | 15.1 | 14.2 | 16 | 7.3 | 11.7 | 17.9 | 12.8 | 14.6 |
| Coleman Liau Index | 11.2 | 14.5 | 13 | 14.6 | 10.3 | 10.2 | 15.9 | 11.3 | 13.3 |
| SMOG Index | 11.3 | 14.2 | 14.3 | 13.7 | 10.4 | 7.5 | 14 | 9.3 | 14.3 |
| Mean Reading Grade | 10.6 | 14 | 13.2 | 14.3 | 9.2 | 9.5 | 15.6 | 10.9 | 13.5 |
AAO = American Academy of Ophthalmology, ASRS = American Society of Retinal Surgeons, NEI = National Eye Institute, SMOG = Simple Measure of Gobbledygook.
Accountability Analysis
No website achieved the full four JAMA benchmarks, and only one of nine websites achieved three of the four JAMA benchmarks, which was Medscape.com (Table 4). Authorship and currency were the two most commonly achieved JAMA benchmarks. Table 4 also demonstrates JAMA accountability by specific benchmark. There was no statistical correlation between website score and number of JAMA benchmarks achieved (r=0.19, p=0.62) using Spearman correlation analysis.
Table 4:
JAMA Accountability by Number of Benchmarks Achieved and Specific Benchmark for Websites Regarding Epiretinal Membranes
| N | % | |
|---|---|---|
| 4 Benchmarks | 0 | 0% |
| 3 Benchmarks | 1 | 11% |
| 2 Benchmarks | 2 | 22% |
| 1 Benchmark | 1 | 11% |
| 0 Benchmarks | 5 | 56% |
| Attribution | 1 | 11% |
| Authorship | 4 | 44% |
| Currency | 3 | 33% |
| Disclosure | 0 | 0% |
N = number of websites, % of all websites. Journal of American Medical Association (JAMA) benchmarks for website accountability: authorship, attributions, currency, disclosures.
Discussion
This study demonstrated significant variations and limitations in content, readability, and accountability of major medical websites covering ERMs. Important content related to surgical intervention (e.g., complications associated with surgery, postsurgical visual outcomes) yielded scores from 0 to 4 across websites, highlighting the importance of advising patients to use certain resources over others when seeking digital information to help with surgical decision-making. In addition, all but two websites covering ERMs failed to mention Amsler grids. Amsler grids may be particularly effective for patient education and autonomy, as they can easily be used by the patient for self-monitoring of visual progression and may affect a patient’s decision to pursue treatment. Remarkably, no correlation was found between content score and Google rank, emphasizing the need for judicious selection of online resources, as higher Google ranking may be mistakenly associated with higher reliability by patients and providers.
Readability of the websites, as determined by an estimated U.S. grade reading level, was demonstrated to be of high complexity. The average twelfth-grade reading level of these websites far exceeded that of the eighth-grade reading level of the average adult American.14 Furthermore, it is recommended by the U.S. Department of Health and Human Services that medical text be written at a sixth-grade reading level.15 Interestingly, reading grade level and content score demonstrated a statistically significant positive correlation, raising a particular concern for compromised readability with higher quality content.
Finally, website accountability was low according to JAMA benchmarks, with no websites fulfilling all four benchmarks, and only one website fulfilling three. Five websites did not fulfill any benchmarks. Website accountability is necessary to demonstrate reliability as online resources available to patients are unregulated, and some websites may be outdated, biased, or purposefully misleading for financial gain.16
A limitation to this study is the self-composed questionnaire used to assess content accuracy, completeness, and clarity. However, no validated questionnaire specific to ophthalmic disease, particularly ERMs, exists to date. A future validated questionnaire could be used to verify our study’s findings. Another limitation is that patient comprehension of ERMs from reading these materials was not directly measured; rather, the rubric was composed to match presumed common questions patients have about ERMs based on the experience of the senior authors. Future studies may directly assess patient understanding, satisfaction, and resultant behavior regarding clinical decision-making.
Despite these limitations, our findings demonstrate that online information available to patients regarding ERMs varies significantly by source. Available information may not adequately answer common patient questions and, in terms of readability, may also be too complex for some patients. Additionally, many websites lack transparency regarding website accountability according to JAMA standards. There is significant room for improvement in online patient resources regarding epiretinal membranes. Standardized information that answers the most common patient questions regarding ERMs, is easier for the average patient to comprehend, and increases transparency in reliability may help improve patient education and decision-making on ERMs.
Acknowledgments
Financial Support: Bascom Palmer Eye Institute received funding from the NIH Core Grant P30EY014801, Department of Defense Grant #W81XWH-13-1-0048, and a Research to Prevent Blindness Unrestricted Grant. The sponsors or funding organizations had no role in the design or conduct of this research.
Footnotes
Disclosure of Interest: Dr. Jayanth Sridhar is a consultant for Alcon, Alimera Sciences, Inc., Regeneron Pharmaceuticals, Inc., and Oxurion. Dr. Ajay E. Kuriyan is a consultant for Allergan plc, Alimera Sciences, Inc., Bausch Health Companies, Inc., Genentech, Inc., Novartis International AG, Regeneron Pharmaceuticals, Inc., and Second Sight. Dr. Nicolas A. Yannuzzi is a consultant for Genentech, Inc. and Novartis International AG. None of the other authors report any disclosures.
References
- 1.Xiao W, Chen X, Yan W, et al. Prevalence and risk factors of epiretinal membranes: a systematic review and meta-analysis of population-based studies. BMJ Open. 2017;7(9):e014644. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Michels RG. Surgical management of epiretinal membranes. Trans Ophthalmol Soc U K. 1979;99(1):54–62. [PubMed] [Google Scholar]
- 3.Fraser-Bell S, Guzowski M, Rochtchina E, et al. Five-year cumulative incidence and progression of epiretinal membranes: the Blue Mountains Eye Study. Ophthalmology. 2003;110(1):34–40. [DOI] [PubMed] [Google Scholar]
- 4.Scheerlinck LM, van der Valk R, van Leeuwen R. Predictive factors for postoperative visual acuity in idiopathic epiretinal membrane: a systematic review. Acta Ophthalmol. 2015;93(3):203–212. [DOI] [PubMed] [Google Scholar]
- 5.Miliatos I, Lindgren G. Epiretinal membrane surgery evaluated by subjective outcome. Acta Ophthalmol. 2017;95(1):52–59. [DOI] [PubMed] [Google Scholar]
- 6.Fox SD, M. Health Online 2013. Pew Research Center Internet & Technology. https://www.pewresearch.org/internet/2013/01/15/health-online-2013/. Published 2013. Updated January 15, 2013. Accessed July 30, 2020. [Google Scholar]
- 7.Amante DJ, Hogan TP, Pagoto SL, et al. Access to care and use of the Internet to search for health information: results from the US National Health Interview Survey. J Med Internet Res. 2015;17(4):e106. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Zhang J, Marmor R, Huh J. Towards Supporting Patient Decision-making In Online Diabetes Communities. AMIA Annu Symp Proc. 2017;2017:1893–1902. [PMC free article] [PubMed] [Google Scholar]
- 9.Feler J, Tan A, Sammann A, et al. Decision Making Among Patients with Unruptured Aneurysms: A Qualitative Analysis of Online Patient Forum Discussions. World Neurosurg. 2019;131:e371–e378. [DOI] [PubMed] [Google Scholar]
- 10.Baker DM, Marshall JH, Lee MJ, et al. A Systematic Review of Internet Decision-Making Resources for Patients Considering Surgery for Ulcerative Colitis. Inflamm Bowel Dis. 2017;23(8):1293–1300. [DOI] [PubMed] [Google Scholar]
- 11.Kloosterboer A, Yannuzzi NA, Patel NA, et al. Assessment of the Quality, Content, and Readability of Freely Available Online Information for Patients Regarding Diabetic Retinopathy. JAMA Ophthalmol. 2019. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Readable. https://readable.io/. Accessed February 1, 2020.
- 13.Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor--Let the reader and viewer beware. JAMA. 1997;277(15):1244–1245. [PubMed] [Google Scholar]
- 14.Doak CC, Doak LG, Friedell GH, et al. Improving comprehension for cancer patients with low literacy skills: strategies for clinicians. CA Cancer J Clin. 1998;48(3):151–162. [DOI] [PubMed] [Google Scholar]
- 15.Weiss B Health literacy: A manual for clinicians. In: Association AM, ed. Chicago, IL: 2003. [Google Scholar]
- 16.Lau D, Ogbogu U, Taylor B, et al. Stem cell clinics online: the direct-to-consumer portrayal of stem cell medicine. Cell Stem Cell. 2008;3(6):591–594. [DOI] [PubMed] [Google Scholar]