Evaluating asthma websites using the Brief DISCERN instrument

Abstract

Purpose

The primary purpose of this study was to examine the quality of sponsored and unsponsored asthma websites using the Brief DISCERN instrument and to evaluate whether the Health On the Net Code of Conduct (HONcode) logo was present, thereby indicating that the site met the criteria. The Internet is an important source of health information for patients and their families. The primary purpose of this study was to examine the quality of sponsored and unsupported asthma websites. A secondary aim was to determine the readability and reading ease of the materials for each website along with the grade level.

Methods

We queried seven Internet search engines using the keyword “asthma.” The websites were evaluated using the six-item Brief DISCERN instrument and by ascertaining whether the HONcode quality label was present. The websites were also evaluated for readability employing Flesch-Kincaid grade level and Flesch reading ease tools using Microsoft Office Word 2013 software.

Results

A total of 22 unique websites were included in the study. Approximately 68% of the websites reviewed had a Brief DISCERN cutoff score of ≥16. The overall Brief DISCERN scores ranged from 6 to 30, and the mean score was 17.32 (SD =6.71). The Flesch-Kincaid grade level scores ranged from 2.9 to 15.4, and the average reading grade score was 9.49 (SD =2.7). The Flesch reading ease scores ranged from 17 to 82.7, with a mean reading ease score of 53.57 (SD =15.03). Sites with a HONcode quality label had significantly higher Brief DISCERN scores than those without one (t=2.3795; df=20; p=0.02).

Conclusion

Brief DISCERN scores revealed that there is quality asthma information for children and their families available on the Internet. The grade level ranged between 2.9 and 15.4 among the websites. However, the mean grade level scores were 9.3–9.89, which is high for the average consumer. Access to accurate information via the Internet, with appropriate readability, may enable pediatric asthma patients and their caregivers to better control and manage asthma.

Introduction

According to the most recent data from the Centers for Disease Control and Prevention, asthma affects 8.6% of children under the age of 18.¹ Asthma is one of the most common chronic diseases in pediatrics with the highest rates among non-Hispanic blacks, Puerto Ricans, Hispanics, mixed race, males, and persons living below the poverty level.¹ Poorly controlled asthma has been associated with adverse behavioral and psychological outcomes, poor quality of life, and a significant impact on daily activities in children.^2,3 Controlling and managing pediatric asthma requires the collaboration of the patient, family caregiver, and primary care provider (PCP). In the past, asthma management plans were discussed in the PCP’s office, and written educational materials of the PCP’s choice would be distributed to the pediatric patients and their families. Currently, however, patients with asthma do not rely on PCPs alone to provide them with written educational materials on the disease. The Internet allows patients with asthma and their families unfettered access to information about asthma.

In 2012, Pew Research Center conducted a survey and found that 72% of Internet users searched the Internet for health information.⁴ Of the 72% who searched for health information, 77% used a popular search engine like Google or Bing, and 13% used a health-specific website like WebMD.⁴ This suggested that the Internet can be a powerful tool for accessing information about health topics such as asthma, but its use also presents the risk that inaccurate, harmful, and perhaps obsolete information will be disseminated to children and their families. Because successful treatment of asthma necessitates considerable understanding of the condition by pediatric patients and their caregivers, access to high-quality and accurate information is required. Access to such information may enable greater self and family management of asthma.⁵

Because a majority of patients are searching for health information on the Internet, researchers have evaluated various aspects of health information websites available to patients. Some studies have assessed the readability and grade level of the websites, while others have examined the quality of information available by noting whether a website had the Health On the Net Code of Conduct (HONcode) label or by using instruments such as the Brief DISCERN and full DISCERN questionnaires.^5–11 The following tools were used for this study.

The HONcode developed by the Health On the Net (HON, Geneva, Switzerland) Foundation was used to evaluate the quality of websites. HON is a nongovernmental organization that encourages quality health information on the Internet.¹² There are eight principles that a health care website has to fulfill in order to apply for certification to place the HONcode logo on their site. The eight principles include authority, complementarity, confidentiality, attribution, justifiability, transparency, financial disclosure, and advertising.¹² The HONcode foundation evaluates websites on the basis of these eight principles of the HON Foundation.¹² Websites can apply for the HONcode logo and if approved can place the logo on their sites as an indication that the information provided is transparent, reliable, and credible.¹² The current study noted the presence or absence of the HONcode label on the included websites.

The Brief DISCERN instrument was developed to evaluate the quality of websites about treatment choices.⁶ This instrument is a shorter version of the DISCERN, developed in 1999, to aid consumers and health care providers in assessing websites for quality information.^13,6 The Brief DISCERN instrument is a valid and reliable instrument based on six questions extracted from the DISCERN instrument focusing on the source of the information, including references and dates of publication, description of treatments, short-term and long-term benefits of the treatments including risks, and the effects of a treatment of choice on quality of life (Box 1).⁶ The instrument is scored on a Likert scale of 5 points (5 = yes, meaning that the criteria for quality have been met, 2–4 = the criteria for quality have been partly met, 1 = no, meaning that the criteria for quality have not been met). Cronbach’s α coefficient was acceptable for the questionnaire (α=0.74).⁶ A score of ≥16 on the Brief DISCERN demonstrated good content quality or evidence-based information.⁶

Box 1. Brief DISCERN questions.

1. Is it clear what sources of information were used to compile the publication (other than the author or producer)

   • Check whether the main claims or statements made about treatment choices are accompanied by a reference to the sources used as evidence, eg, a research study or an expert opinion.
   • Look for a means of checking the sources used such as a bibliography/reference list or the addresses of the experts or organizations quoted.
2. Is it clear when the information used or reported in the publication was produced?

   • Dates of the main sources of information used to compile the publication.
   • Date of any revisions of the publication (but not dates of reprinting).
   • Date of publication (copyright date).
3. Does it describe how each treatment works?

   • Look for a description of how a treatment acts on the body to achieve its effect.
4. Does the publication describe the benefits of each treatment?

   • Benefits can include controlling or getting rid of symptoms, preventing recurrence of the condition, and eliminating the condition, both short term and long term.
5. Does it describe the risks of each treatment?

   • Risks can include side effects, complications, and adverse reactions to treatment, both short term and long term.
6. Does it describe how the treatment choices affect the overall quality of life?

   • Look for: description of the effects of the treatment choices on day-to-day activity.
   • Description of the effects of the treatment choices on relationships with family, friends, and caregivers.

The Flesch-Kincaid tool was used to evaluate the readability of the materials on each website with the use of Microsoft Office Word 2013 software (Microsoft Corporation, Redmond, WA, USA). This tool has been widely used to examine the reading grade level of health education materials. The Flesch-Kincaid tool rates text on a US school grade level. A reading grade level score between the fifth and twelfth grade levels is provided. For instance, a score of 9.0 means that a ninth grader should be able to read and understand the document. If a document is rated as more difficult than the twelfth grade reading level, the results are reported as the twelfth grade reading level.

The Flesch reading ease tool rates text on a 100-point scale. The higher the score, the easier it is to comprehend the document. A score on the Flesch reading ease between 90 and 100 implies easy comprehension by an 11-year-old student, a score of 60–70 indicates that average 13- to 15-year-old students can understand the presented material, and a score of 0–30 is comprehended by college-level students.

The purpose of this study was to examine the quality of asthma websites on sponsored and unsponsored websites using Brief DISCERN and ascertaining whether the HON-code logo was present on the site. The secondary aim was to determine the readability of the website with the Flesch-Kincaid grade level tool and Flesch reading ease score.

Methods

Using the Brief DISCERN instrument, a descriptive study design was employed to evaluate the quality of information regarding asthma on the Internet. We searched seven English-language search engines using the keyword “asthma.” The search engines used for this study were Google (www.google.com), AOL (www.aol.com), Yahoo (www.yahoo.com), Ask (www.ask.com), Lycos (www.lycos.com), Bing (www.bing.com), and Blekko (www.blekko.com). The strategy was to search for information in the manner of a patient or a patient’s caregiver. Therefore, both sponsored and unsponsored Internet sites were reviewed. Unsponsored websites provide information and usually do not advertise products or services (eg, an academic institution with a website ending in “.edu”). Sponsored websites are provided by paid advertisers of products and/or services (eg, a website that sells nebulizer machines or spacers).

Searching the Internet for “asthma” yielded 87–118 million results. The search was limited to the first page of each search engine because research has found that the majority of Internet users searched only the first 10 web links that were displayed in search results.¹⁴ Inclusion criteria included English-language websites on asthma. Websites excluded were duplications, Wikipedia, videos, invalid addresses, non-English-language websites, external links, books, articles, and discussion or open forums. Both unsponsored and sponsored websites were used in the review. A total of 112 sponsored and unsponsored websites were initially reviewed. After narrowing down the websites based on the above exclusion criteria, 15 unsponsored websites (Table 1) and seven sponsored websites (Table 2) were used in the study. The websites were evaluated for quality content using the Brief DISCERN and the presence of the HONcode. The websites were also evaluated for readability using Flesch reading ease and Flesch-Kincaid grade level using the Microsoft Office Word 2013 software.

Table 1.

Uniform resource locator addresses of the unsponsored websites and their Brief DISCERN scores, reading ease, and reading grade level scores

Website	HON*	Reading ease	Grade level	Brief DISCERN
www.aaaai.org/conditions-and-treatments/asthma.aspx	No	42	11.2	16
www.aafa.org	Yes	54	9.5	20
www.webmd.com/asthma/default.htm	Yes	60.7	7.7	16
www.aanma.org	No	51.9	10.2	6
www.epa.gov/asthma/index	No	82.7	2.9	10
www.mayoclinic.com/health/asthma/DS00021	Yes	51.4	9.5	27
www.medicinenet.com/asthma/article.htm	Yes	39.1	11.8	26
www.ncbi.nlm.nih.gov/pubmedhealth/PMH0001196	No	58.6	8.4	20
www.nhlbi.nih.gov/health/health-topics/topics/asthma	Yes	62.7	8	26
www.cdc.gov/asthma	No	69.5	7.1	18
www.emedicine.medscape.com/article/296301-overview	No	17	15.4	30
www.lung.org/lung-disease/asthma/	No	60.3	8.3	22
www.asthma.com/	No	53	10.1	20
http://health.yahoo.net/channel/asthma.html	No	59.4	8.8	13
http://www.ask.healthline.com/health/asthma	No	48.1	10.6	19
Mean score		54.02	9.3	19.27

Note:

^*HON, presence of the Health On the Net Code of Conduct (HONcode) logo on the website.

Table 2.

Uniform resource locator addresses of the sponsored websites and their Brief DISCERN scores, reading ease, and grade level scores

Website	HON*	Reading ease	Grade level	Brief DISCERN
www.asthma.com	No	60.2	10	16
www.dailyasthmamedication.com	No	57	9.1	22
www.asthma-control-treatment.com	No	24.7	14.4	15
www.xpansions.com	No	50.4	10.9	13
www.caallergy.com	No	42	11.1	6
www.merckengage.com	No	79.1	4.8	12
www.massgeneral.org/children	No	54.8	8.9	8
Mean score		52.6	9.89	13.14

Note:

^*HON, presence of the Health On the Net Code of Conduct (HONcode) logo on the website.

Results

A total of 22 unique websites were reviewed after the application of the exclusion criteria and removal of duplicate websites. For the overall sample of sponsored and unsponsored websites, the Brief DISCERN scores ranged from 6 to 30 with a mean score of 17.32 (SD =6.71). Approximately 68% of the websites reviewed had a Brief DISCERN score of ≥16. The Flesch-Kincaid grade level scores ranged from 2.9 to 15.4, with an average reading grade score of 9.49 (SD =2.7). Of these 22 websites, only four (18%) had reading grade levels below the eighth grade reading level. The Flesch reading ease scores ranged from 17 to 82.7, with an average reading ease score of 53.57 (SD =15.03).

When the findings were divided into subsets of sponsored and unsponsored websites, the results revealed that there were seven sponsored websites that met the criteria for inclusion in the study. The Brief DISCERN score for sponsored websites in this study ranged from 6 to 22, with an average of 13.14 (SD =5.30). Forty-two percent of the sponsored websites had a Brief DISCERN score of ≥16. The mean Flesch-Kincaid grade level score was 9.89 (SD =2.897) with a range of 4.8–14.4. Only one of the seven sponsored websites (14%) had a reading grade level below the eighth grade reading level. The average Flesch reading ease score was 52.6 (SD =16.73) with a range from 24.7 to 79.1.

Fifteen unsponsored websites were evaluated. The Brief DISCERN scores for this group ranged between 6 and 30 with an average score of 19.27 (SD =6.79). Approximately 80% of the unsponsored websites had a Brief DISCERN score of ≥16. The average Flesch-Kincaid grade level was 9.3 (SD =2.782) with a range of 2.9–15.4. Four of the 15 (27%) of the unsponsored websites had a reading grade level below the eighth grade reading level. The Flesch reading ease scores ranged from 17 to 82.7 with an average of 54.02 (SD =15.24).

A comparison of the sponsored and the unsponsored websites showed that the mean Brief DISCERN scores were not statistically significant, although the results showed a trend toward the unsponsored sites having higher Brief DISCERN scores than the sponsored sites (19.27 and 13.14 respectively). A comparison of the Flesch reading ease scores showed they did not differ significantly (52.6 and 54.02), nor did the mean reading grade level scores (9.89 and 9.3).

Five unsponsored websites (22.72%) had the HONcode quality label with an average Flesch-Kincaid reading grade level of 9.3 (SD =1.63); four had a reading grade level below the eighth grade. No sponsored websites had the HONcode quality label.

The mean score of the Flesch-Kincaid reading grade level for the websites with and without the HON label was 9.3 and 9.51, respectively. Similarly, the mean score of the Flesch reading ease scores for websites with and without the HON label was 53.69 and 53.43, respectively. Sites with a HONcode quality label had significantly higher Brief DISCERN scores than those without the HONcode quality label (23 and 15.65, respectively).

Discussion

A review of the medical and nursing literature identified five studies researching Internet sites regarding asthma.^5,7–10 These studies examined readability and quality of asthma websites, as determined by adherence to national asthma standards set forth by the National Asthma Education and Prevention Program (NAEPP).¹⁵ None of the studies on Internet websites on asthma determined quality of asthma websites using a standardized instrument such as the Brief DISCERN or HONcode. Despite the quantity of information available on asthma, only one study examined the use of the DISCERN instrument on asthma and atopic dermatitis pamphlets.¹¹ The DISCERN instrument (a Japanese translation) was used to determine the interrater agreement between health care professionals and patients when examining the asthma pamphlets and websites on atopic dermatitis.¹¹ The authors concluded that the DISCERN instrument was useful in evaluating the reliability of information both in written pamphlets and on the Internet.¹¹ They also found that medical providers were better prepared to evaluate the reliability of websites than patients.¹¹

This is one of the first studies that have looked at the quality of asthma websites using the Brief DISCERN tool and HONcode quality label. A Brief DISCERN score of ≥16 out of 30 is a potentially good indicator of good quality content websites. In the current study, 68% of websites had a Brief DISCERN score of ≥16, showing that the majority of these asthma sites had good quality information. However, when broken down into sponsored and unsponsored, less than half of the sponsored sites reached the cutoff score, while 80% of the unsponsored sites had scores ≥16. This finding is supported by previous studies that found that sponsored websites contained less accurate content than unsponsored ones,^5,16,17 but differed from Kaicker et al,¹⁸ which suggested that content quality was not associated with the origin of the site or rank order on the search engine. Pediatric patients and their caregivers who are looking for information regarding asthma on the Internet may be more likely to find accurate information on unsponsored rather than sponsored sites because the goal of the latter may be to advertise a product rather than to provide accurate information.

The current study found that 22.7% of sites had a health-related quality seal such as HONcode. This is lower than what was previously reported in the literature – 36%.¹⁹ While the numbers from the current study are lower, neither of these are suitable percentages – with roughly one-third or fewer sites containing the HON label. Comparable to the findings of Bruce-Brand et al,²⁰ the results of the current study indicated differences in scores between websites without the HON label and websites with the HON label.²⁰ Several other researchers also found that websites that bore the HON seal had significantly higher DISCERN scores.^19,21–23 Developers of health-related websites should strive to meet the criteria set forth by the HON Foundation to ensure that the health information disseminated on the Internet meets quality standards. Caregivers of children with asthma, as well as pediatric patients themselves, who seek health information on the Internet should be instructed to look for the HON seal to confirm that the website being used contains quality information.

When evaluating the readability of asthma websites with the Flesch reading ease test and the Flesch-Kincaid tool, the authors found that the websites assessed in their study had an average reading ease level of 54.33 (100-point scale) and a mean reading grade level of 9.73 – above the recommended reading grade level for health information materials.⁷ The National Institutes of Health recommends that the reading grade level for written health materials be between the seventh and eighth grade reading levels for the average adult.²⁴ It was concluded that the materials on asthma on the Internet are written at a reading grade level that may be too high for the average consumer to comprehend even when the information on the websites is deemed to be accurate.⁷ In the current study, the reading ease and reading grade level scores were not significantly different for the websites carrying the HONcode logo compared with those without the HONcode logo. Similarly, Khazaal et al²¹ found that those items with the HONcode designation did not have significantly higher Flesch-Kincaid reading grade level scores than those without the HONcode designation.²¹ Their study findings did reveal significantly higher reading ease scores. Only four of the websites in the current study had reading grade levels below the recommended eighth grade reading level. None of those sites had the HONcode designation. So while the HONcode designation may indicate quality, the materials may be of little to no benefit for pediatric patients and their caregivers if they are too difficult to read. These varied results indicate the need for further research to determine the importance of the presence of the HONcode on reading scores.

The overall Flesch-Kincaid reading grade level scores of 9.49 in this current study are similar to what has been reported in the literature in the past regarding reading grade levels of asthma websites^7,9 and slightly higher than what has been reported for written asthma materials.^25–27 The average reading ease score of 53.57 is consistent with the findings of previously published literature.⁷ These reading ease scores correspond to the high school level of reading, and the level is deemed to be too high for the average health consumer whose mean reading level is eighth grade.²⁸ Many pediatric patients who are striving toward self-management may find many asthma websites difficult to comprehend if the reading level is too high. Many adolescents, even those in high school, may not have a ninth grade reading ability and may not be able to fully understand the information available on the Internet – even if it is accurate information.

Limitations

Potential limitations of this study include the cross-sectional design and the small number of websites included. Future research may focus on repeating this study with a larger number of websites while also noting the interrater reliability of the website examiners. In addition, findings from the Brief DISCERN instrument should be compared with those of the full-length DISCERN instrument.

Conclusion

Asthma websites have great potential to provide educational materials for pediatric patients and their caregivers. However, because the quality of the information can vary so widely, there is a risk that inaccurate information that is not easily readable or comprehensible may be disseminated to pediatric patients and their families. Our study provided evidence that the Brief DISCERN with a score >16 and the HON quality indicator are predictors of quality websites. In this study, the reading level was too high for the average consumer. The high grade levels of health-related materials can impede pediatric patients and their caregivers from effective self- and family management of chronic illnesses such as asthma. Access to high-quality, accurate, and comprehensible information may enable pediatric patients with asthma and their caregivers to better control and manage asthma.