Abstract
Objective
To review published criteria for specifically evaluating health related information on the world wide web, and to identify areas of consensus.
Design
Search of world wide web sites and peer reviewed medical journals for explicit criteria for evaluating health related information on the web, using Medline and Lexis-Nexis databases, and the following internet search engines: Yahoo!, Excite, Altavista, Webcrawler, HotBot, Infoseek, Magellan Internet Guide, and Lycos. Criteria were extracted and grouped into categories.
Results
29 published rating tools and journal articles were identified that had explicit criteria for assessing health related web sites. Of the 165 criteria extracted from these tools and articles, 132 (80%) were grouped under one of 12 specific categories and 33 (20%) were grouped as miscellaneous because they lacked specificity or were unique. The most frequently cited criteria were those dealing with content, design and aesthetics of site, disclosure of authors, sponsors, or developers, currency of information (includes frequency of update, freshness, maintenance of site), authority of source, ease of use, and accessibility and availability.
Conclusions
Results suggest that many authors agree on key criteria for evaluating health related web sites, and that efforts to develop consensus criteria may be helpful. The next step is to identify and assess a clear, simple set of consensus criteria that the general public can understand and use.
Key messages
Many organisations and individuals have published criteria to evaluate health related information on the world wide web
A literature and world wide web search found that the most frequently cited criteria were those dealing with content, design and aesthetics of site, disclosure of authors, sponsors, or developers, currency of information, authority of source, and ease of use
Criteria related to confidentiality and privacy were only cited by one author
Consensus regarding critical criteria for evaluation of web based health information seems to be emerging
Our results indicate that many authors agree on key criteria for evaluating health related web sites, and that efforts to develop a set of key criteria may be helpful
Introduction
The large volume of health information resources available on the internet has great potential to improve health,1–3 but it is increasingly difficult to discern which resources are accurate or appropriate for users.3–8 Because of the potential for harm from misleading and inaccurate health information,9–14 many organisations and individuals have published or implemented criteria for evaluating the appropriateness or quality of these resources.15,16 Two published reviews of evaluation criteria for health related web sites did not present information on the range of criteria proposed by various authors, and included rating tools that were not developed exclusively for health related sites.15,17 Our study reviews criteria currently proposed or employed specifically to evaluate health related web sites.
Methods
Databases and search engines
Between September 1997 and May 1998 we conducted a search of the web and peer reviewed medical journals for criteria for evaluating health related information on the web using Medline and Lexis-Nexis databases, and web search engines including Yahoo!, Excite, Altavista, Webcrawler, HotBot, Infoseek, Magellan Internet Guide, and Lycos. Medline searches (using PubMed) used variations of the following: “quality,” “Internet,” “World Wide Web,” “computer communication networks/standards,” “quality control,” and “medical Informatics/standards.” Searches with web search engines and Lexis-Nexis used “quality,” “health information,” “health,” and variations of “rating,” “ranking,” “evaluate,” “award,” and “assess.” Investigating references and hyperlinks from initial results gave additional resources. We ended the sampling period when searches produced similar results, and when previous search results became outdated.
Criteria
We included criteria when they were explicit, specifically used for evaluating health related web sites, and published in a peer reviewed journal or publicly accessible web site. We also considered peer reviewed journals not indexed by Medline. We included resources framed as “guidelines” because there was little difference between them and other criteria, and the intent of the authors was similar. When subcriteria provided details about main criteria, we included only the main criteria to prevent overrepresenting that author’s perspective. Criteria were extracted and sorted into similar groups according to their wording and description. When a criterion seemed to combine several concepts and could fit in multiple groups, we considered the first mentioned concept.
To examine the reproducibility of the criteria groupings, four independent, naive coders assigned 40 randomly selected criteria to the 13 criteria groups. Overall, the coders’ assignment of criteria agreed with us 76% of the time. The agreement coefficient, indicating “per cent agreement above chance” was 0.74 or 74%.18
Results
Twenty nine rating tools and articles—24 web sites and five journal articles—had explicit criteria for assessing health related web sites (table 1). Of the 165 criteria identified, 132 (80%) were grouped under 12 specific categories (table 2). Thirty three (20%) criteria that lacked specificity or were unique were categorised as “miscellaneous.” Frequently cited criteria included those dealing with content, design and aesthetics of site, and disclosure of authors, sponsors, or developers.
Table 1.
Rating tools and journal articles with explicit criteria for evaluation of health related web sites
| Source of rating tool or article | URL address* |
|---|---|
| AltiMed/PharmaLINKS | www.altimed.com/links/ratings.html |
| American Medical Association | www.ama-assn.org/med_link/med_link.htm |
| Biosites, Pacific Southwest Regional Medical Library | www.library.ucsf.edu/biosites/help/guidelines.html |
| British Healthcare Internet Association | www.bhia.org/public/reference/recommendations/medpubstandards.htm |
| Growth House | www.growthhouse.org/award.html |
| Health A to Z | www.healthatoz.com/aboutus.htm |
| Health Information Institute’s Aesculapius Awards | www.hii.org/judging.htm |
| Health On the Net Foundation | www.hon.ch/HONcode/Conduct.html |
| Health Summit I Mtg/Mitretek Systems | www.mitretek.org/hiti/showcase/documents/criteria.html |
| Health Web | healthweb.org/wg/content/papers/guidelines.html |
| Healthfinder | www.healthfinder.gov/aboutus/selectionpolicy.htm |
| Kotecki JE, Siegel DE. Electronic notes: use of a critical thinking/questioning approach to evaluate WWW information. Am J Health Behav 1998;22:75-6. | Not available online |
| Larkin M. Health information on-line. FDA Consumer 1996;30:21-5. | www.fda.gov/fdac/features/596_info.html |
| McGill University Health Sciences and Osler Libraries, Canada | www.health.library.mcgill.ca/resource/criteria.htm |
| Medical Matrix | www.medmatrix.org/info/sitesurvey.html |
| Medsite Navigator, Guide to Digital Science and Medicine | www.medsitenavigator.com/mail/submit.html |
| Mental Health Net | www.cmhc.com/help/ratings.htm |
| Mountain and Plains Partnership | www.uchsc.edu/csa/areahec/home/mapp/8aWWW.html#public |
| Nutrition Navigator | navigator.tufts.edu/ratings.html |
| Organising Medical Networked Information | omni.ac.uk/agec/evalguid.html |
| Reference 16 | Not available online |
| Physician’s Choice | www.mdchoice.com/instruc.htm |
| Psych Central: Best of the Web in Mental Health | www.grohol.com/rateguid.htm |
| Reference 6 | www.ama-assn.org/sci-pubs/journals/archive/jama/vol_277/no_15/ed7016x.htm |
| Sympatico’s HealthyWay Health Links | www1.sympatico.ca/healthyway/GENERAL/info_2.html |
| The Six Senses Review | www.sixsenses.com/FAQ.html#rating |
| The Virtual Hospital | indy.radiology.uiowa.edu/Beyond/PeerReviews/01Introduction.html |
| The Wilton Library | w3.nai.net/∼wla/eval.htm |
| Reference 8 | www.bmj.com/archive/7098ip2.htm |
Because of dynamic nature of web, some URLs may have changed. URLs prefixed with http://
Table 2.
Frequency of explicit criteria for evaluation of health related web sites by criteria groups*
| Criteria groups | Frequency (%) (n=165) |
|---|---|
| Content of site (includes quality, reliability, accuracy, scope, depth) | 30 (18) |
| Design and aesthetics (includes layout, interactivity, presentation, appeal, graphics, use of media) | 22 (13) |
| Disclosure of authors, sponsors, developers (includes identification of purpose, nature of organisation, sources of support, authorship, origin) | 20 (12) |
| Currency of information (includes frequency of update, freshness, maintenance of site) | 14 (8) |
| Authority of source (includes reputation of source, credibility, trustworthiness) | 11 (7) |
| Ease of use (includes usability, navigability, functionality) | 9 (5) |
| Accessibility and availability (includes ease of access, fee for access, stability) | 9 (5) |
| Links (includes quality of links, links to other sources) | 5 (3) |
| Attribution and documentation (includes presentation of clear references, balanced evidence) | 5 (3) |
| Intended audience (includes nature of intended users, appropriateness for intended users) | 3 (2) |
| Contact addresses or feedback mechanism (includes availability of contact information, contact address) | 2 (1) |
| User support (includes availability of support, documentation for users) | 2 (1) |
| Miscellaneous (includes criterion that lacked specificity or were unique) | 33 (20) |
Of five authors who assigned weights or priorities to their proposed criteria, four cited content and one cited peer review (categorised as miscellaneous) as most important criterion. Percentage total does not equal 100 owing to rounding-off.
Discussion
Not surprisingly, “content” of the site, which includes concepts of information quality and accuracy, was the most commonly cited criterion group. Design and aesthetics of the site and ease of use were the second and sixth most frequently cited groups respectively, indicating that authors highly value good quality application design and user interfaces. Disclosure of authors, sponsors, or developers had the third highest frequency, highlighting the need for users to be able to consider a site’s content in the context of who created or financed the site. It was somewhat surprising that disclosure was not more commonly cited given recent reports about misleading health information and fraud on the internet.9,11,12 Most rating tools discriminated between content and the fourth most common criterion group, currency of information (includes frequency of update, freshness, maintenance of site), suggesting that currency of information is nearly as important as the information itself.
Criteria related to confidentiality and privacy of information were only cited by one author despite widespread interest in this issue.19 Some health related web sites are already collecting personal health information to “tailor” content, and as sites begin to integrate healthcare services and information, confidentiality and privacy safeguards will become increasingly important.19–21
Study limitations
Study limitations include the subjective variables around the scope of the criteria categories used. Testing of the category groupings, however, showed that they were reproducible by others. It is also possible that some authors used the same criteria terms to describe different concepts. Because subcriteria were not included, some concepts may not have been represented. Inherent limitations of web search engines and the dynamic nature of the web also prevented us from locating all existing published criteria.22 Nevertheless, our review located more sources of criteria specifically for health related sites than did previous reviews.15,17
Conclusion
Given the evolving state of the internet, it may be difficult or even inappropriate to develop a static tool or system for assessing health related web sites. Our results suggest that many authors agree on key criteria, and that efforts to develop consensus criteria may be helpful.6,16,23–25 The next step is to identify and assess a clear, simple set of consensus criteria that the general public can understand and use. Tools that integrate them need to be developed and validated, and their ultimate impact and effectiveness in assisting the public with health related decisions should be monitored to ensure that they remain useful.
Supplementary Material
Acknowledgments
We thank Farrokh Alemi and Anne Restino for their assistance and advice on this study. The views expressed in this paper are solely those of the authors and do not necessarily reflect those of the US Department of Health and Human Services.
Footnotes
Funding: Internal funds of the US Department of Health and Human Services.
Competing interests: None declared.
References
- 1.Government Accounting Office. Consumer health informatics. Emerging issues. Publication Government Accounting Office/Accounting and Information Management Division-96-86, July 1996.
- 2.Robinson TN, Patrick K, Eng TR, Gustafson D.for the Science Panel on Interactive Communication and Health. An evidence-based approach to interactive health communication: a challenge to medicine in the Information Age JAMA 19982801264–1269. [DOI] [PubMed] [Google Scholar]
- 3.Eng TR, Maxfield A, Patrick K, Deering MJ, Ratzan S, Gustafson D. Access to health information and support: a public highway or a private road? JAMA. 1998;280:1371–1375. doi: 10.1001/jama.280.15.1371. [DOI] [PubMed] [Google Scholar]
- 4.Coiera E. The internet’s challenge to health care provision. BMJ. 1996;312:3–4. doi: 10.1136/bmj.312.7022.3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Anon The web of information inequality [editorial] Lancet. 1997;349:1781. [PubMed] [Google Scholar]
- 6.Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling, and assuring the quality of medical information on the internet. Caveant lector et viewor—let the reader and buyer beware. JAMA. 1997;277:1244–1245. [PubMed] [Google Scholar]
- 7.Sonnenberg FA. Health information on the internet. Opportunities and pitfalls [editorial] Arch Intern Med. 1997;157:151–152. [PubMed] [Google Scholar]
- 8.Wyatt JC. Commentary: measuring quality and impact of the world wide web. BMJ. 1997;314:1879–1881. doi: 10.1136/bmj.314.7098.1879. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Federal Trade Commission. North American Health Claim Surf Day targets Internet ads. Hundreds of e-mail messages sent. Press release, Nov. 5, 1997. Accessed May 12, 1998. http://www.ftc.gov/opa/9711/hlthsurf.htm
- 10.Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the world wide web: systematic survey of advice on managing fever in children at home. BMJ. 1997;314:1875–1879. doi: 10.1136/bmj.314.7098.1875. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Food and Drug Administration. FDA warns consumers on dangerous products promoted on the internet. FDA Talk Paper T97-26, June 17, 1997.
- 12.Bower H. Internet sees growth of unverified health claims. BMJ. 1996;313:497. doi: 10.1136/bmj.313.7054.381. [DOI] [PubMed] [Google Scholar]
- 13.Micke MM. The case of hallucinogenic plants and the internet. J Sch Health. 1996;66:277–280. doi: 10.1111/j.1746-1561.1996.tb03397.x. [DOI] [PubMed] [Google Scholar]
- 14.Weisbord SD, Soule JB, Kimmel PL. Poison on line—acute renal failure caused by oil of wormwood purchased through the internet. N Engl J Med. 1997;337:825–827. doi: 10.1056/NEJM199709183371205. [DOI] [PubMed] [Google Scholar]
- 15.Jadad AR, Gagliardi A. Rating health information on the internet. Navigating to knowledge or to Babel? JAMA. 1998;279:611–614. doi: 10.1001/jama.279.8.611. [DOI] [PubMed] [Google Scholar]
- 16.Pealer LN, Dorman SM. Evaluating health-related web sites. J Sch Health. 1997;67:232–235. doi: 10.1111/j.1746-1561.1997.tb06311.x. [DOI] [PubMed] [Google Scholar]
- 17.Murray PJ, Rizzolo MA. Web site reviews and evaluations. Nurs Stand Online 1997 Jul 30;11(45). Accessed January 29, 1998. http:\\www.nursing-standard.co.uk/vol11-45/ol-art.htm
- 18.Krippendorf K. Content analysis: an introduction to its methodology. Beverly Hills, CA: Sage; 1980. [Google Scholar]
- 19.Bowen JW, Klimczak JC, Ruiz M, Barnes M. Design of access control methods for protecting the confidentiality of patient information in networked systems. Proceedings of the American Medical Informatics Association annual fall symposium 1997:46-50. [PMC free article] [PubMed]
- 20.National Research Council, Computer Science and Telecommunications Board (US) For the record: protecting electronic health information. Washington: National Academy Press; 1997. [Google Scholar]
- 21.Patrick K, Robinson TN, Alemi F, Eng TR.for the Science Panel on Interactive Communication and Health. Policy issues relevant to the evaluation of interactive health communication applications Am J Prev Med 19991635–42. [PubMed] [Google Scholar]
- 22.Lawrence S, Giles CL. Searching the world wide web. Science. 1998;280:98–100. doi: 10.1126/science.280.5360.98. [DOI] [PubMed] [Google Scholar]
- 23.British Healthcare Internet Association. Quality standards for medical publishing on the web. Accessed May 26, 1998. http:\\www.bhia.org/public/reference/recommendations/medpubstandards.htm
- 24.Health On the Net Foundation. HON code of conduct for medical and health web sites. Accessed January 27, 1998. http:\\www.hon.ch/HONcode/Conduct.html
- 25.Health Information Technology Institute, Mitretek Systems. Criteria for assessing the quality of health information on the internet. Accessed January 27, 1998. http:\\www.mitretek.org/hiti/showcase/index.html
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.