Consulting Dr. Google: Quality of Online Resources About Tympanostomy Tube Placement

Abstract

Objective

Tympanostomy tube placement is common in children, however family-centeredness and utility of online information used for decision-making and understanding is unknown. We evaluate the quality of leading internet resources describing tympanostomy tube placement.

Study Design

Cross-sectional descriptive design.

Methods

We performed a Google™ search for terms related to tympanostomy tubes. We defined quality using scaled readability measures (Flesch Reading Ease and Flesch-Kincaid Grade-Level), understandability and actionability (Patient Education Materials Assessment Tool), shared-decision making centrality (Center for Medicare and Medicaid Services informed consent guidelines) and Clinical Practice Guideline compatibility. Three reviewers coded each measure. Fleiss κ interrater reliability analysis was performed.

Results

Ten most frequently encountered websites were analyzed. 1/10 met national health literacy standards (mean 10^th grade-level reading, median 9^th, range 6–15^th). All sites were understandable, (mean understandability 81.9%, range 73–92%). Most had low actionability scores (7/10, median 47%, mean 44.6%, range 0–80). Shared-decision making centrality was high (mean 5, range 4–6), but most did not list alternative treatment options. While clinical practice guideline compatibility was high (mean 3.4, range 1–4), many websites contained inconsistent recommendations about tube duration, follow-up, and water precautions. There was inter-rater agreement for understandability scoring (κ = 0.20; p = 0.02).

Conclusion

Internet resources about tympanostomy tube placement vary in quality pertaining to health literacy, principles of shared decision-making, and consistency with practice guidelines. With growing emphasis on patient/family-centered engagement in healthcare decision-making, standardization of content and improved usability of educational materials for common surgical procedures in children, such as tympanostomy tube placement, should be a public health priority.

Introduction

Tympanostomy tube (TT) placement is the most common surgical procedure in young children.¹ The American Academy of Otolaryngology-Head and Neck Surgery Foundation (AAO-HNSF) has published guidelines regarding patient selection for TT placement and management of patients who have received tubes, though adherence to these guidelines through dissemination and implementation efforts remains largely uncertain.¹ TT placement is often performed in otherwise healthy children who will be undergoing their first surgical procedure. Even if considered a minor procedure, parents may experience a great deal of decisional conflict.² Active participation by parents in decision-making and the use of decision aids may reduce decisional conflict over surgery in children. The internet is a readily available and often used source for patients to find health-related information and may be the first resource they consult. A 2013 study performed by Pew research center revealed that 35% of Americans use the internet to obtain information about diagnosis and treatment of medical conditions.³ Since there are no standard guidelines regarding how online patient education materials are written, online patient education materials vary in quality and readability. Most American adults read at an eighth-grade level.⁴ However, patient education materials found online are routinely written at grade levels above which the average American can read efficiently.⁴ Inadequate health literacy leads to poor adherence to prescribed treatments, poor follow-up, with resulting potential for increased risks and complications.⁴ When we educate patients and families about treatment risks, benefits and alternatives, we facilitate patient/family centered care, shared decision-making and improved outcomes.^5–7

There have been several studies in the otolaryngology literature evaluating online resources which have found wide heterogeneity in information regarding surgical procedures including tonsillectomy^8–10, Zenker’s diverticulum,¹¹ and thyroidectomy.¹² We seek to evaluate the online resources and educational materials available for the most commonly performed otolaryngologic surgery in children, TT placement.

Materials and Methods

Data Sources

This study involved nonhuman research and was exempt from Institutional Review Board approval. We performed an online search using Google™ search engine on August 25, 2016. The following search terms were used: “ear tubes”,” tympanostomy tubes” and “PE tubes”. Search terms were selected to mirror terms commonly used by lay people regarding TT placement. A Google ™ search was performed for each term. Search engine selection was based on data ranking Google ™ as the most commonly used search engine.¹³ Five pages of results with 20 results per page for each search term were reviewed and the ten websites that were found in the results for each search term were analyzed. The goal of this study was to simulate a google search performed by a parent/care-giver. Each sitelink was selected as it appeared in the search results. Websites that were written in English, had no access restrictions, and were published in the United States were evaluated. Resources that were primarily audio and/or video resources without written words were excluded as this study was focused on written material. Target audience was determined by either information listed on the website or the manner in which the reader was addressed (e.g., “your child’s doctor may suggest…”).

Outcome Measures

We evaluated each of the selected sites for readability, understandability, actionability, shared decision-making, and clinical practice guidelines (CPG). Analyses were performed with Stata statistical software (version 14; Stata Corp; College Station, TX).

Readability Evaluation

For readability evaluation, the text from each site was edited in Microsoft Word ®. Headings, bulleted items and other formatting were removed, in order to achieve accurate reading scores.¹⁴ The text was evaluated for readability using the Flesch Reading Ease test (FRE), and Flesch-Kincaid Grade Level (FKGL), via the online readability calculator found at https://readability-score.com. Reading scores calculated using FRE are based on four elements: average sentence length in words, average word length in syllables, average percentage of “personal words” (e.g., neutral gender pronouns), and average percentage of “personal sentences”, (i.e., sentences or statements where the reader is directly addressed).¹⁵ Higher scores are associated with easy to read material and lower scores indicate that the text is difficult to understand (Table 1). Readability scores were then used to calculate reading grade level based on a formula established by Kincaid in 1975.¹⁶ In addition to reading ease score, grade level is also determined by total words, sentences and syllables.¹⁶

Table 1.

Pattern of Reading Ease Scores

Reading Ease Score	Description of Style	Typical Magazine	Syllables per 100 words	Average Sentence Length in Words
0 to 30	Very difficult	Scientific	192 or more	29 or more
30 to 50	Difficult	Academic	167	25
50 to 60	Fairly difficult	Quality	155	21
60 to 70	Standard	Digests	147	17
70 to 80	Fairly easy	Slick-fiction	139	14
80 to 90	Easy	Pulp-fiction	131	11
90 to 100	Very easy	Comics	123 or less	8 or less

Reprinted from “A new readability yardstick,” by Rudolph Flesch, 1948, Journal of Applied Psychology, 32, p. 230. Copyright 1948 by the American Psychological Association. Reprinted with permission.

Understandability and actionability

Understandability and actionability were evaluated with the Patient Education Materials Evaluation Tool (PEMAT). PEMAT is a validated tool designed to be completed by professionals to assess understandability and actionability of patient education material.¹⁷ Materials are deemed understandable when individuals with varying levels of literacy are able to understand the central message.¹⁷ Understandability scores are calculated based on content, word choice and style, use of numbers, organization, layout/design and use of visual aids. Actionability refers to the intended audience’s ability to identify “next steps” or potential actions based on the information provided.¹⁷ Materials with scores of 70% or more are deemed to be adequately understood and actionable.¹⁷ PEMAT scoring was performed by three individuals (two physicians, and one non-clinical researcher). Fleiss κ interrater reliability analysis was performed using Stata software to determine level of agreement amongst raters.

Shared Decision-making

The Center for Medicaid and Medicare Services informed consent guidelines were used to determine six key factors central to shared decision-making. The six factors we deemed essential to be conveyed to patients and their families in order for them to make a well-informed decision are the description of procedure, indications, short-term risks, long-term risks, benefits of the procedure, and alternatives.¹⁸ Each site was scored on a six-point scale: one point was given for each factor that was included, with scores ranging from zero (no shared decision-making) to six. Websites were reviewed independently by two physician reviewers.

Clinical Practice Guideline compatibility

Websites were evaluated for adherence to the published AAO-HNSF Clinical Practice Guideline: Tympanostomy Tubes in Children.¹ Each site was reviewed to determine inclusion of information regarding CPG. The following four guideline points which the American Academy of Otolaryngology-Head and Neck Surgery (AAOHNS) CPG state should be addressed in perioperative education were used: 1) duration of tube function, 2) follow-up schedule, 3) detection of complications, and 4) water precautions. According to CPG “Short-term tubes generally last 10 to 18 months, but long-term tubes typically remain in place for several years.” Emphasis is also placed on informing the caregiver about the unpredictable nature of duration of tube function and the possibility of premature extrusion. There is no explicit follow-up schedule recommended in CPG but the importance of post-operative follow-up in order to decrease complication risks should be discussed with caregivers. Concerning the detection of complications, caregivers should be given information regarding presentation of ear infections while the tubes are in place and how the infections should be treated. CPG recommend against prophylactic water precautions with the following exceptions: it causes the child discomfort, active and/or recurrent TT otorrhea, middle ear infections with P. aeruginosa or S. aureus, children with immune dysfunction who may be at increased risk for infection, deep diving, and to avoid exposure to contaminated water.¹ Each site was scored on a four-point scale: one point was given if there was correct information for each of the above-mentioned guidelines, with scores ranging from zero to four. CPG compliance was assessed by two different physician coders.

Results

Ten websites were analyzed. Readability scores ranged from 28.8 to 70.6 with a mean of 50.4 (SD = 11.6) and a median of 51.85. Reading Grade levels ranged from 6^th to 15^th grade with a mean of 10^th grade (SD = 2.5) and a median of 10^th grade. Nine out of ten websites had readability scores above the NIH recommended seventh-eighth grade level.⁷ All websites were understandable with understandability scores ranging from 73 to 92% with a median of 81.5 and a mean of 81.9 (SD = 6). Actionability scores were more variable and generally low. Scores ranged from 0 to 80% with a median of 47 and mean of 44.6 (SD = 28). Most sites did not provide tangible tools and visual aids to help the reader take action. Table 2 summarizes PEMAT scores and grade levels for each site. Shared decision-making centrality scoring ranged from 4 to 6 with a mean and median of 5 (SD =0.5). Nine out of ten websites sufficiently describe the procedure and its risks, however only three mentioned anesthesia risks. Only two sites listed alternatives to surgery. Most sites had information that complied with the assessed components of AAOHNS CPG for TT placement. Adherence scores ranged from 1 to 4 with a mean of 3.4 (SD = 1) and median of 4. See Table 3 for representative text illustrating the variability in CPG adherence. Fleiss κ analysis showed slight inter-rater agreement for PEMAT understandability scoring (κ = 0.20; p= 0.02). Values for κ are interpreted as: 0 poor agreement, 0.01–0.20 slight, 0.21–0.40 fair, 0.41–0.60 moderate, 0.61–0.80 substantial, 0.81–1.00 almost perfect.¹⁹ Fleiss κ analysis for inter-rater agreement for PEMAT actionability scoring showed poor agreement but was not statistically significant (κ = 0.07; p = 0.13).

Table 2.

Readability and PEMAT Scoring.

Website	Setting type	FRE† score	FKGL‡	Understandability§	Actionability
http://www.entnet.org/content/ear-tubes	Society website	45.9	12	80%	47%
http://www.mayoclinic.org/tests-procedures/ear-tubes/home/ovc-20199999	Academic health system	53.3	9	85%	73%
http://kidshealth.org/en/parents/ear-infections.html	Community health system	55.5	10	80%	47%
http://www.webmd.com/cold-and-flu/ear-infection/tubes-for-ear-infections	Health & Wellness website	70.6	6	83%	80%
http://www.medicinenet.com/ear_tubes/article.htm	Health & Wellness website	53.6	9	92%	73%
http://www.emedicinehealth.com/ear_tubes/article_em.htm	Health & Wellness website	44.5	12	86%	33%
http://www.earcentergreensboro.com/medical-education/ear_tubes.php	Physician group	50.4	10	77%	53%
https://en.wikipedia.org/wiki/Tympanostomy_tube	Wikipedia	40.4	12	73%	0%
http://www.childrenshospital.vanderbilt.org/services.php?mid=9847	Academic health system	61.8	9	88%	40%
http://emedicine.medscape.com/article/1890757-overview	Health & Wellness website	28.8	15	75%	0%

^†Flesch Reading Ease score interpretation: 0–60 difficult to read, 60–70 standard, 70–100 easy to read.¹⁷.

^‡Flesch-Kincaid grade level converts reading score to a U.S. grade level.¹⁸ The NIH recommends patient education material be written at a 7^th to 8^th grade level to subvert inadequate health literacy.²⁵

^§Materials with scores of 70% or more are deemed to be adequately understood and actionable.²²

Table 3.

Clinical Practice Guideline Variation

Clinical Practice Guideline	Representative text	Corresponding Guideline Information
Duration of tube function	“Short- term tubes … typically stay in place for six to eighteen months…. Long-term tubes are larger and have flanges that secure them in place for a longer period of time.”32 “These ventilating tubes remain in place for six months to several years.”33 “Tympanostomy tubes generally remain in the eardrum for six months to two years, with T-tubes lasting up to four years.”34	“Parents/caregivers of children with tympanostomy tubes should be given information regarding longevity of the tympanostomy tubes. This will vary depending on the type of tube that is placed (short-term versus long-term tubes). Short-term tubes generally last 10 to 18 months, but long-term tubes typically remain in place for several years.”
Follow-up schedule†	“An initial follow-up appointment will be scheduled within the first two to four weeks after the procedure. Other follow-up appointments…will be scheduled at four- to six-month intervals”.35 “Follow-up visits…. Are very important. The doctor checks to see whether the tubes are working and whether the child’s hearing has improved.”36 Your doctor may recommend a follow-up examination 7–14 days after the procedure. Further appointments are typically scheduled every 3–6 months…”.37	“Generally, the child should be evaluated periodically by an otolaryngologist while the tympanostomy tubes are in place. After extrusion, an additional follow-up appointment with the otolaryngologist should occur to ensure the ears are healthy and to identify any need for further surveillance or treatment. “
Detection of complications‡	“Medical attention may be necessary… If the child has experienced several ear infections…The child has persistent ear drainage after using the drops as ordered. The child has increasing ear pain without ear drainage…. If any significant change of hearing is noted”.37 “If the drainage persists or if there is fever greater than 102F, an office visit may be necessary…”.38 “Otorrhea…- This is treated initially with antibiotic ear drops; occasionally, children experience persistent ear tube drainage that necessitates prompt removal of the tube”.39	‘…. parents/caregivers should be counseled that TTO may occur, responds to topical antibiotic ear drops, does not usually require oral antibiotics, and benefits from water precautions until the discharge is no longer present.”
Water precautions §	“Your surgeon might recommend earplugs for regular bathing or swimming.”40 “…usually you don’t have to worry about protecting the ears with an earplug unless your child is dunking their head deeply (over a couple of feet below the surface) or the water is not thought to be clean.”41 “Current guidelines do not recommend routine water precautions”.32	“Water precautions are unnecessary for most children with tympanostomy tubes but should be implemented for children who develop TTO or experience discomfort upon exposure to water. Protection with earplugs, headbands, or water avoidance may be necessary during periods of active TTO.”

^†Not mentioned on two websites.

^‡Not mentioned on two websites.

^§Not mentioned on two websites.

Discussion

Clinical experience, medical training, and published CPG, as well as an evaluation of the specific needs of each patient and family, guide an otolaryngologist in decision-making and counseling for TT placement. While published CPG and similar materials may be accessible to patients, they are usually targeted to clinicians and are not designed to inform patients and families.

Previous studies have illustrated the heterogeneity of online patient education materials, similar to our findings in this study.^{8–9,11, 20–21} The median readability grade level was at the 10^th grade level, well above the 7^th–8^th grade level recommended by the NIH health literacy guidelines for health materials.²² This may indicate that many patients, particularly patients with low socioeconomic status and with low literacy, are at a disadvantage for successful shared decision-making when using internet resources. Inadequate health literacy further widens health disparities experienced by those of low socioeconomic status.²³ As we formulate strategies to reduce health disparities emphasis should be placed on providing patient education materials that are written at appropriate grade levels to promote health literacy.²⁴ Of note readability formulas should be used with caution as they do not assess other factors such as the overall context of the material.

Due to limitations in readability scoring, we also used the Patient Education Materials Assessment tool to assess understandability and actionability. Most websites were understandable with an average understandability score of 81.9% and a of range 73–92%. Seven of the ten websites had low actionability scores with a median of 47% and a mean of 44.6%. PEMAT can be a useful guide to help authors of patient education material determine if the information provided is understandable and if patients will be able to act on what they learn. However, the PEMAT does not assess quality of materials because it is does not evaluate accuracy of information.¹⁷ Another limitation is differing interpretation of items being evaluated amongst raters which is what we found in this study. Raters consisted of two physicians and one non-clinical investigator. Discrepancies in ratings could be due to a difference in definition. For example, raters may have separate qualifications for a website being “clear” or “distracting,” or a different definition of what constitutes a “summary.” While examples are provided in the PEMAT user guide there is no set criteria for scoring. Also, raters may have different perspectives when material is subjective.

Shared decision-making centrality was high with a mean of 5 and a range of 4 to 6. Eight out of ten websites failed to list alternative treatment options or discuss the risk/benefits of surgery versus observation. One reason for this may be that the sites were geared more to families that had already decided to undergo surgery. However, in a prior study regarding parental experience with decision making for management of sleep disordered breathing, parents who were provided with several therapeutic options had greater satisfaction with decision making.²⁵ Only three out of ten sites addressed anesthesia risks, which many clinicians and families deem perhaps the most significant worry with TT. Shared decision-making emphasizes the patient’s/family’s involvement in deciding on a treatment plans and is most successful when all reasonable options are discussed.²⁶

While these websites are not explicitly called decision aids, 28% of people use the internet to help make medical decisions.²⁷ This may especially be the case when parents have to make health decisions for their children. Parents can have high decisional conflict concerning surgical decision making for their children.²⁸ Increasing their knowledge and being informed allows them to decrease this conflict and anxiety in order to participate in the SDM process.²⁹ Easily accessible, comprehensive decision aids specifically designed for TT placement may help with SDM.

Accuracy of information was defined as the degree of concordance with clinical practice guidelines and the absence of inaccurate or misleading information. Clinical Practice Guideline compatibility and accuracy of information was generally high with a mean of 3.4 and a range of 1 to 4. All ten of the websites explained the variable duration of tube function. While recommendations for follow-up intervals were not seen in these sites, no such recommendation exist in the CPG either. All sites mentioned the detection of TT otorrhea, a frequently encountered complication.³⁰ Information concerning water precautions was most variable with three websites suggesting that ear plugs may need to be used and two websites providing no mention of water precautions. A systematic review of randomized controlled trials regarding the effectiveness of water precautions in preventing ear infections showed no clinical significant decrease in ear infections when water precautions were taken.³¹ These findings are not indicative of physician compliance but may provide a starting point to evaluate CPG adherence in clinical practice and standardization of practice.

Study limitations

In an effort to include websites that were found using all three search terms we only evaluated ten sites. While a previous study by Eysenbach& Köhler showed that consumers routinely only review a small number of the many websites listed in search results¹⁰; limiting our study to ten sites led to low statistical power. The Google search algorithm alters search results based on the type of device used to perform the search, personal search history, geographic location and browser type. Therefore, the websites chosen for this study may not be representative of search results others may encounter. The education materials reviewed are not necessarily representative of what physicians provide for patient education, but the aim of this paper was to evaluate publicly available online education materials. The scales to evaluate SDM and CPG compliance were created for this study and are not validated. There was also no formal training for using the PEMAT. In addition, the PEMAT was designed to be completed by professionals, however it is possible that “understandability” may be better evaluated by patients, families, or other non-healthcare professionals. Despite these limitations, this is the first study to evaluate online patient education materials regarding TT placement. Future efforts to standardize information for parents and families about common elective procedures would prove to be a key quality improvement initiative.

Conclusion

Patient-centered care and shared decision making are important components for elective procedures. Patient/family education can be a key component to successful decision making in TT placement. Commonly used internet resources about TT placement vary in quality pertaining to health literacy, principles of shared decision-making, and consistency with practice guidelines. Overall, easily accessed online educational materials for TT placement are understandable but are written at inappropriately high reading levels and have low actionability. Shared decision making centrality and adherence to CPG were good for the websites evaluated. Clinicians should recognize that the available online educational materials may be inadequate for successful shared decision-making and reduced decisional conflict, and should be prepared to supplement this with in-person counseling and well-constructed decision-aids.

Table 4.

Key term glossary

Term	Definition
Understandability	Materials are deemed understandable when individuals with varying levels of literacy are able to understand the central message.17
Actionability	Actionability refers to the intended audience’s ability to identify “next steps” or potential actions based on the information provided.17
Shared decision-making centrality	Principles critical to ensure shared-decision making
Clinical Practice Guideline Compatibility	Information and practices supported by clinical guidelines
Decisional Conflict	Uncertainty about what course of action to take when choosing between options involve regret, risk, or challenge to personal values.42.