Abstract
Background/Aims
The Internet is the main resource for health-related information. The incidence of inflammatory bowel disease (IBD) is rapidly increasing in Asian countries. However, the quality of websites for IBD available in this region has not been evaluated. We aimed to evaluate the quality of the information on IBD obtained from Korean websites.
Methods
Using the terms “Crohn’s disease” or “ulcerative colitis,” websites were selected from those obtained with the three most renowned search engines in Korea; 60 websites from the results of each engine were chosen. The websites were classified into institutional, commercial, charitable, supportive, or alternative medicine types according to the characteristics of each site. The websites were evaluated regarding content quality using the validated DISCERN instrument and the Journal of the American Medical Association benchmarks.
Results
The median score of all the websites according to the DISCERN instrument was 32 (interquartile range, 25 to 47) out of 80, indicating an insufficient overall quality of information. The alternative medicine sites scored the lowest, whereas the institutional sites scored the highest (p < 0.05). The quality of information was significantly different among the search engines (p = 0.028). The rank of appearance in the Google search result did not correlate with the quality level of the information.
Conclusions
The quality of information on the Internet regarding IBD varied according to the website type and search engine. Accreditation and quality assurance systems should be implemented for websites to ensure that the public and patients obtain accurate information on IBD.
Keywords: Inflammatory bowel diseases; Crohn disease; Colitis, ulcerative; Internet; Quality improvement
INTRODUCTION
Inflammatory bowel disease (IBD), comprising Crohn’s disease (CD) and ulcerative colitis (UC), is a disorder that causes chronic inflammation of the gastrointestinal tract. Although the clear etiology of IBD has not been elucidated, it is known that genetic, immunologic, and environmental factors are associated with the development of IBD [1]. A growing body of evidence indicates that the incidence of IBD is rapidly increasing in Asian countries [2-5].
At present, the numbers of patients and members of the public who search for health information on the Internet are soaring. Research has shown that 85% of physicians have had a patient bring health information from the Internet to a visit [6]. The number of individuals using the Internet is rapidly growing worldwide, and Internet users in Asia account for 51% of all global Internet users [7]. A survey has shown that 88.3% of South Koreans use the Internet [8] and that the highest Internet use according to age is observed in teenagers, followed by people in their 20s and 30s. Given that IBD is a chronic disease and often occurs at a young age, it can be assumed that many IBD patients are searching for health information on the Internet. Several studies regarding the quality of IBD information on the Internet have been reported in Western countries [9-15]. However, the quality of such information in Asia has not been evaluated.
Therefore, the aim of our study was to evaluate the quality of the content of Korean websites containing information on IBD. We also assessed the quality of the information on IBD according to website type and search result ranking.
METHODS
We chose three of the most commonly used search engines in South Korea: Naver (www.naver.com), Daum (www.daum.net), and Google Korea (www.google.co.kr). The search terms “Crohn’s disease” and “ulcerative colitis” were entered in the Korean language. Websites covering information on CD or UC for the public or patients that were written in the Korean language were included. We adopted the first 30 results from each search engine for both the CD and UC searches. Duplicate websites were excluded. The websites were classified into one of the following five types: (1) institutional websites managed by the government, a hospital, or a university (institutional sites); (2) commercially sponsored websites or private medical websites (commercial sites); (3) charitable websites managed by nonprofit organizations (charitable sites); (4) personal web pages for support or websites managed by patient support groups (supportive sites); and (5) alternative medicine websites about unorthodox therapies (alternative sites) [10]. The quality of the websites was evaluated using the DISCERN instrument [16] and the Journal of the American Medical Association (JAMA) benchmarks [17]. Two researchers independently performed the quality evaluation from October 1 to 30, 2016. One researcher had more than 5 years’ experience in the field of gastroenterology and the other had been taking care of IBD patients for 12 years. Disagreements between the two researchers were resolved by re-examining the websites after the robust discussion. Since this study was designed to evaluate the information of IBD on Korean websites, there was no potential ethical issue and the Institutional Review Board (IRB) approval was waived. Informed consent was also not required because data was obtained from the websites.
Quality ratings
The DISCERN instrument is a questionnaire that provides users with a valid and reliable way to assess the quality of information on treating a health problem [16]. The instrument consists of 15 key questions and an overall quality rating. The score for each question is determined based on a 5-point scale ranging from ‘no’ to ‘yes’ (Supplementary Table 1). A higher score indicates better quality. Questions 1 to 8 address the reliability of the information, while Questions 9 to 15 focus on specific details of the information regarding treatment choices, and Question 16 is the overall quality rating. The instrument has been used in several reports to evaluate the quality of websites regarding health information on various diseases, including IBD [10-12,18]. The JAMA benchmarks are also designed to assist users in assessing the quality of health information on websites [17]. The benchmarks consist of the following four concepts: (1) authors, contributors, affiliations, and credentials should be provided (authorship); (2) references, sources, and copyright for all information should be clearly provided (attribution); (3) the ownership of websites should be prominently and fully disclosed (disclosure); and (4) dates when the information was posted and updated should be provided (currency). As 1 point is allocated for each of the criteria met, scores range from 0 to 4; a higher score indicates better quality. Although these benchmarks have been applied to evaluate the quality of website-derived health information on various diseases [19-21], the quality of IBD-related information on websites has not yet been reported.
Statistical methods
The Kruskal-Wallis test with the post hoc Dunn’s test was used to compare the DISCERN scores between subgroups. Spearman’s rank correlation coefficient was used to evaluate the correlation between the DISCERN scores and the total number of JAMA benchmarks achieved. To evaluate the relationship between the order of appearance in the Google search engine results and the ranking according to the total DISCERN scores, Spearman’s rank correlation coefficient was used. The interobserver variability for the total DISCERN scores was calculated using an intraclass score. All analyses were performed using the SPSS version 18.0 (SPSS Inc., Chicago, IL, USA).
RESULTS
Among 180 websites, 42 websites were duplicates (Fig. 1); therefore, a total of 138 websites were ultimately selected for analysis, of which 70 websites were related to CD and 68 were related to UC. These websites were assigned to five groups according to website type. Supportive sites were the most common type (64 websites, 46.4%), followed by alternative sites (30 websites, 21.7%). The number of alternative sites in the Naver search engine results was significantly higher than that obtained using Google (39.6% [21/53] vs. 9.3% [4/43], p = 0.006) (Table 1). Two certified gastroenterologists independently evaluated the selected websites using the DISCERN instrument and the JAMA benchmarks. The intraclass score for the sum of the total DISCERN scores determined by the two researchers was 0.948 (95% confidence interval, 0.927 to 0.963), which indicates a nearly perfect agreement between the two researchers.
Figure 1.
Flow chart for website enrollment. CD, Crohn’s disease; UC, ulcerative colitis.
Table 1.
Enrolled website types from three search engines
| Naver (n = 53) | Daum (n = 42) | Google (n = 43) | |
|---|---|---|---|
| Institutional | 8 (15.1) | 3 (7.1) | 8 (18.6) |
| Commercial | 1 (1.9) | 6 (14.3) | 4 (9.3) |
| Charitable | 5 (9.4) | 5 (11.9) | 4 (9.3) |
| Support | 18 (34.0) | 23 (54.8) | 23 (53.5) |
| Alternativea | 21 (39.6) | 5 (11.9) | 4 (9.3) |
Values are presented as number (%).
Chi-square analysis between Naver and Google, p = 0.006.
The median score of the websites based on the DISCERN instrument was 32 (interquartile range [IQR], 25 to 47) out of 80 indicating a poor overall quality of information. Box plots for the total DISCERN scores according to website type are shown in Fig. 2A. There were significant differences in the total DISCERN scores among the groups. The institutional sites had the highest median score, whereas the alternative sites had the lowest median score. In addition, the institutional, charitable, and commercial sites each scored significantly higher than the score for the alternative sites (p < 0.001), while there was no difference between the supportive and alternative sites. We presented the median total DISCERNS score values among the websites types in Supplementary Table 2. The total DISCERN score per search engine is shown in Fig. 2B. Google had the highest median score, followed by Daum and Naver. The post hoc test revealed a significant difference in the DISCERN scores between Google and Naver (p = 0.028).
Figure 2.
Box plots of the total DISCERN score per (A) website type and (B) search engine. The box plots indicate that the alternative sites have the lowest DISCERN score among website types and Naver has the lowest DISCERN score among search engines. NS, non specific.
The proportion of websites meeting each JAMA benchmark is shown in Fig. 3. “Currency” was the most frequently achieved benchmark among all the websites (63%), followed by “authorship,” “disclosure,” and “attribution” (47.1%, 31.9%, and 11.6%, respectively). The average number of JAMA benchmarks achieved per website type was the highest for the institutional sites and the lowest for the alternative sites (Fig. 4). In addition, both the institutional and charitable sites had significantly higher average scores than the alternative sites (p < 0.001), while there was no difference between the supportive and alternative sites. A strong relationship was identified between the total DISCERN score and the total number of JAMA benchmarks achieved (r = 0.687, p < 0.001), as shown in Fig. 5. There was no significant difference between UC and CD websites according to the median total DISCERN score and JAMA benchmarks (data not shown).
Figure 3.
Benchmark achievement of each Journal of the American Medical Association benchmark for all websites and for website types for inflammatory bowel disease. “Currency” was the most frequently achieved benchmark among all the websites.
Figure 4.
The average number of Journal of the American Medical Association (JAMA) benchmarks achieved per website type. The institutional sites have the highest average number of achieved JAMA benchmarks and the alternative sites have the lowest. NS, non specific.
Figure 5.
Correlation of total DISCERN scores with the total number of achieved Journal of the American Medical Association (JAMA) benchmarks. A strong correlation is detected between total DISCERN scores and the total number of achieved JAMA benchmarks.
There was no significant correlation between the ranking for quality according to the total DISCERN score and the order of appearance in the Google search engine results (CD, r = –0.174, p = 0.359; UC, r = –0.263, p = 0.160) (Fig. 6). For CD, the website that had the highest DISCERN score was 9th in the list of search results (Fig. 6A). In the case of UC, the website that achieved the highest DISCERN score was 24th in the list of search results (Fig. 6B).
Figure 6.
Correlation between total DISCERN scores and the order of appearance in the search result list for (A) Crohn’s disease (CD) and (B) ulcerative colitis (UC) using the Google search engine. The dot plots indicate that total DISCERN scores and the order of appearance in the Google result list are not correlated.
DISCUSSION
This study showed that there was a wide variation in the quality of IBD-related health information obtained from Korean websites. Using the DISCERN and JAMA assessment tools, we found that the institutional sites had the highest quality and that the alternative sites had the lowest. Notably, the order of appearance of the website in the search engine result list had no correlation with the quality ranking, which hinders patients in accessing the accurate Internet information related to IBD. To our knowledge, this is the first study to evaluate the quality of IBD-related information on the Internet in an Asian country.
The median overall DISCERN score of the websites was 32 (IQR, 25 to 47) out of 80, which indicates a disappointing level of quality. Previous Western studies have revealed that the total DISCERN score of the websites regarding IBD was 42.2 to 48, which is higher than that of our study [10,12]. Because these studies evaluated websites written in the English language using English-language search engines such as Google, Yahoo, and Bing, a direct comparison with our result may not be possible. However, the composition of the website types in these studies was different from ours. In the study by van der Marel et al. [10], the number of institutional sites (34%) was much higher than the number of alternative sites (8%). That study also showed that the institutional sites scored significantly higher than the alternative sites, which is consistent with our study. In contrast, we found that only 13.8% of the sites consisted of institutional sites and 21.7% consisted of alternative sites. Therefore, we can assume that the relatively high proportion of alternative websites in our study may have contributed to the low overall quality score.
The results of the quality assessment using the JAMA benchmarks was similar to that using the DISCERN instrument. The alternative and supportive sites had a significantly lower total number of achieved JAMA benchmarks than did the institutional and charitable sites. The institutional sites had the highest total number of achieved JAMA benchmarks. The superior quality of institutional sites may have been because most of these sites (84.2%, 16 out of 19 sites) were academic websites such as those of university hospitals. There have been no prior studies on the quality assessment of IBD websites using the JAMA benchmarks. In our study, we found a strong relationship between the total DISCERN score and the total number of JAMA benchmarks achieved (r = 0.687, p < 0.001) (Fig. 5). Therefore, the simple JAMA benchmark assessment may also be used to evaluate the quality of IBD websites.
The quality of the IBD-related information on the Internet did not correlate with the order of appearance in the list of search results. Most Internet users click on a site appearing on the first page of search engine results, and very few Internet users are likely to click on any results that appear past the third page [22]. Therefore, the quality of the websites that are high on the search result list should be of better quality. The search algorithms of Google analyze hundreds of different factors in an attempt to uncover the best information the Internet can offer [23]. Unfortunately, these algorithms do not take into account the quality of health information. In our study, consistent with the results of previous studies [10,11], the order of appearance of websites on IBD in the Google search results did not correlate with the quality of the sites according to the DISCERN instrument. An accreditation system for health information on the Internet regarding IBD is required to enable Internet users to obtain high-quality information.
Notably, we found that the quality of information on IBD is different among search engines. The quality of information obtained using Naver, the most popular search engine in Korea, is significantly inferior to that obtained with Google when assessed using the DISCERN instrument. This difference might be explained by the different website compositions among the search engines. The Naver search engine displayed a significantly greater number of alternative sites, which had the lowest quality of information, than did Google (39.6% vs. 9.3%, p = 0.006) (Table 1). Furthermore, most of the alternative websites listed in the Naver results (80%, 17 out of 21, data not shown) focused on the advertisement and were not for the purpose of providing appropriate information on IBD. Internet users should be aware of this difference when using search engines to obtain information on IBD.
This study has several potential limitations. First, our study only included websites searched at a certain point in time. Therefore, our results may not represent the entire number of websites available over time. Future studies should assess the quality of information on the Internet regarding IBD over time. Second, only the top 30 websites were selected from the results of each search engine, but high-quality websites may exist well beyond the third page of results. Third, we did not evaluate the readability of the websites. Readability is essential for the general public to receive health information through the Internet and should not exceed the 6th grade readability level [24]. The readability level of websites in English can be calculated using the Flesch-Kincaid Grade Level readability score [25] or the Coleman-Liau Readability Index score [26]. However, there is no validated tool to calculate the readability level of Korean-language websites. As evaluating the quality of health information on the Internet becomes more important over time, it is necessary to develop an appropriate tool that can calculate the readability level of languages other than English. Finally, we did not evaluate the accuracy of IBD information on websites. Our study focused only on quality of IBD information on websites, and did not evaluate the accuracy of that. However, previous studies have shown that the websites with good quality of IBD information tend to have the good accuracy [9-11].
In conclusion, the overall quality of information regarding IBD on the Internet in Korea is not optimal and varies significantly depending not only on website type but also on the search engine used. Websites appearing higher among the search results list are not guaranteed to provide higher quality IBD-related information. An accreditation system for Internet search engines using validated quality assessment tools will allow the recommendation of high-quality websites to the general public and patients.
KEY MESSAGE
1. The quality of information on the Internet for Korean patients with inflammatory bowel disease was insufficient and varied according to the website type and search engine.
2. The alternative medicine sites had the lowest quality of information, whereas the institutional sites had the highest. The higher search ranking did not guarantee better quality.
3. Accreditation and quality assurance systems will allow the recommendation of high-quality websites to the general public and patients.
Footnotes
No potential conflict of interest relevant to this article was reported.
Supplementary Materials
The DISCERN instrument
The total DISCERN score per website type
REFERENCES
- 1.Stenke E, Hussey S. Ulcerative colitis: management in adults, children and young people (NICE Clinical Guideline CG166) Arch Dis Child Educ Pract Ed. 2014;99:194–197. doi: 10.1136/archdischild-2013-305512. [DOI] [PubMed] [Google Scholar]
- 2.Zhao J, Ng SC, Lei Y, et al. First prospective, populationbased inflammatory bowel disease incidence study in mainland of China: the emergence of "western" disease. Inflamm Bowel Dis. 2013;19:1839–1845. doi: 10.1097/MIB.0b013e31828a6551. [DOI] [PubMed] [Google Scholar]
- 3.Zeng Z, Zhu Z, Yang Y, et al. Incidence and clinical characteristics of inflammatory bowel disease in a developed region of Guangdong Province, China: a prospective population-based study. J Gastroenterol Hepatol. 2013;28:1148–1153. doi: 10.1111/jgh.12164. [DOI] [PubMed] [Google Scholar]
- 4.Morita N, Toki S, Hirohashi T, et al. Incidence and prevalence of inflammatory bowel disease in Japan: nationwide epidemiological survey during the year 1991. J Gastroenterol. 1995;30 Suppl 8:1–4. [PubMed] [Google Scholar]
- 5.Yang SK, Yun S, Kim JH, et al. Epidemiology of inflammatory bowel disease in the Songpa-Kangdong district, Seoul, Korea, 1986-2005: a KASID study. Inflamm Bowel Dis. 2008;14:542–549. doi: 10.1002/ibd.20310. [DOI] [PubMed] [Google Scholar]
- 6.Murray E, Lo B, Pollack L, et al. The impact of health information on the Internet on health care and the physician-patient relationship: national U.S. survey among 1.050 U.S. physicians. J Med Internet Res. 2003;5:e17. doi: 10.2196/jmir.5.3.e17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.International Telecommunications Union . Global and regional ICT data 2017 [Internet] Geneva (CH): International Telecommunications Union; c2017 [cited 2018 Aug 1]. Available from: http://www.itu.int. [Google Scholar]
- 8.Korea Internet & Security Agency . Survey on the Internet usage 2016 [Internet] Seoul (KR): Korea Internet & Security Agency; c2017 [cited 2018 Aug 1]. Available from: http://isis.kisa.or.kr. [Google Scholar]
- 9.Bernard A, Langille M, Hughes S, et al. A systematic review of patient inflammatory bowel disease information resources on the World Wide Web. Am J Gastroenterol. 2007;102:2070–2077. doi: 10.1111/j.1572-0241.2007.01325.x. [DOI] [PubMed] [Google Scholar]
- 10.van der Marel S, Duijvestein M, Hardwick JC, et al. Quality of web-based information on inflammatory bowel diseases. Inflamm Bowel Dis. 2009;15:1891–1896. doi: 10.1002/ibd.20976. [DOI] [PubMed] [Google Scholar]
- 11.Langille M, Bernard A, Rodgers C, Hughes S, Leddin D, van Zanten SV. Systematic review of the quality of patient information on the Internet regarding inflammatory bowel disease treatments. Clin Gastroenterol Hepatol. 2010;8:322–328. doi: 10.1016/j.cgh.2009.12.024. [DOI] [PubMed] [Google Scholar]
- 12.Azer SA, AlOlayan TI, AlGhamdi MA, AlSanea MA. Inflammatory bowel disease: an evaluation of health information on the Internet. World J Gastroenterol. 2017;23:1676–1696. doi: 10.3748/wjg.v23.i9.1676. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13.Chang MY, Kim JW, Rhee CS. The quality of health information on allergic rhinitis, rhinitis, and sinusitis available on the Internet. Allergy Asthma Immunol Res. 2015;7:141–147. doi: 10.4168/aair.2015.7.2.141. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Lee YJ, Kim HJ, Yu DS, Lee YB, Hahn HJ, Kim JW. Current status of atopic dermatitis-related information available on the Internet in South Korea. Ann Dermatol. 2016;28:1–5. doi: 10.5021/ad.2016.28.1.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Song H, Omori K, Kim J, et al. Trusting social media as a source of health information: online surveys comparing the United States, Korea, and Hong Kong. J Med Internet Res. 2016;18:e25. doi: 10.2196/jmir.4193. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Charnock D, Shepperd S, Needham G, Gann R. DISCERN: an instrument for judging the quality of written consumer health information on treatment choices. J Epidemiol Community Health. 1999;53:105–111. doi: 10.1136/jech.53.2.105. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the Internet: Caveant lector et viewor: let the reader and viewer beware. JAMA. 1997;277:1244–1245. [PubMed] [Google Scholar]
- 18.Guo L, Reich J, Groshek J, Farraye FA. Social media use in patients with inflammatory bowel disease. Inflamm Bowel Dis. 2016;22:1231–1238. doi: 10.1097/MIB.0000000000000713. [DOI] [PubMed] [Google Scholar]
- 19.Clancy AA, Hickling D, Didomizio L, et al. Patient-targeted websites on overactive bladder: what are our patients reading? Neurourol Urodyn. 2018;37:832–841. doi: 10.1002/nau.23359. [DOI] [PubMed] [Google Scholar]
- 20.Janssen S, Kasmann L, Fahlbusch FB, Rades D, Vorder-mark D. Side effects of radiotherapy in breast cancer patients: the Internet as an information source. Strahlenther Onkol. 2018;194:136–142. doi: 10.1007/s00066-017-1197-7. [DOI] [PubMed] [Google Scholar]
- 21.Joury A, Joraid A, Alqahtani F, Alghamdi A, Batwa A, Pines JM. The variation in quality and content of patient-focused health information on the Internet for otitis media. Child Care Health Dev. 2018;44:221–226. doi: 10.1111/cch.12524. [DOI] [PubMed] [Google Scholar]
- 22.iProspect . iProspect search engine user behavior study 2006 [Internet] London (UK): iProspect; c2017 [cited 2018 Aug 1]. Available from: http://www.iprospect.com. [Google Scholar]
- 23.Google Search . How search works [Internet] Mountain View (CA): Google; c2017 [cited 2018 Aug 1]. Available from: http://www.google.com/search/howsearchworks. [Google Scholar]
- 24.MedlinePlus . How to write easy-to-read health materials [Internet] Bethesda (MD): U.S. National Library of Medicine; c2017 [cited 2018 Aug 1]. Available from: http://www.nlm.nih.gov/medlineplus/etr.html. [Google Scholar]
- 25.Friedman DB, Hoffman-Goetz L. A systematic review of readability and comprehension instruments used for print and web-based cancer information. Health Educ Behav. 2006;33:352–373. doi: 10.1177/1090198105277329. [DOI] [PubMed] [Google Scholar]
- 26.Eltorai AE, Naqvi SS, Ghanian S, et al. Readability of invasive procedure consent forms. Clin Transl Sci. 2015;8:830–833. doi: 10.1111/cts.12364. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
The DISCERN instrument
The total DISCERN score per website type