Abstract
Objectives
To describe techniques for retrieval and appraisal used by consumers when they search for health information on the internet.
Design
Qualitative study using focus groups, naturalistic observation of consumers searching the world wide web in a usability laboratory, and in-depth interviews.
Participants
A total of 21 users of the internet participated in three focus group sessions. 17 participants were given a series of health questions and observed in a usability laboratory setting while retrieving health information from the web; this was followed by in-depth interviews.
Setting
Heidelberg, Germany.
Results
Although their search technique was often suboptimal, internet users successfully found health information to answer questions in an average of 5 minutes 42 seconds (median 4 minutes 18 seconds) per question. Participants in focus groups said that when assessing the credibility of a website they primarily looked for the source, a professional design, a scientific or official touch, language, and ease of use. However, in the observational study, no participants checked any “about us” sections of websites, disclaimers, or disclosure statements. In the post-search interviews, it emerged that very few participants had noticed and remembered which websites they had retrieved information from.
Conclusions
Further observational studies are needed to design and evaluate educational and technological innovations for guiding consumers to high quality health information on the web.
What is already known on this topic
Little is known about how consumers retrieve and assess the quality of health information on the internet
Qualitative data are needed to design educational and technological innovations to guide consumers to high quality health information
What this study adds
Users of the internet explore only the first few links on general search engines when seeking health information
Consumers say that when assessing the credibility of a site they primarily look for the source, a professional design, and a variety of other criteria
In practice, internet users do not check the “about us” sections of websites, try to find out who authors or owners of the site are, or read disclaimers or disclosure statements
Very few internet users later remember from which websites they retrieved information or who stood behind the sites
Introduction
Little is known about how consumers retrieve and assess health information on the world wide web. Some surveys have elicited data by using semistructured questionnaires or focus groups,1–3 but little (if any) unobtrusive observational research has been done to explore how consumers are actually surfing the web. Although several criteria for quality of health websites have been proposed—including disclosure of site owners, authors, and update cycle4,5—little or nothing is known about whether and to what degree such markers are recognised or even looked at by consumers or what other credibility markers consumers are looking for. We aimed to obtain qualitative and semiquantitative data to generate some hypotheses on how consumers might search for and appraise health information.
Methods
We used multiple methods of data collection that are commonly used in studies of human-computer interactions,6 combining focus groups,7 naturalistic observation of consumers searching the internet, and post-search in-depth interviews. Two researchers independently analysed transcripts by using N5 (NUD*IST 5.0; QSR International, Melbourne) with the grounded theory approach.8
Participants in the focus groups and the observational study were mostly healthy volunteers recruited through newspaper advertisements seeking people who had already searched for health information on the web. We selected them on a first come first served basis.
Focus groups
We held three focus group sessions with 6-8 participants each (21 participants in total: five men, 16 women; mean and median age 37, range 19-71 years) in March 2001. Self reported internet experience of the participants ranged from 17 to 84 months (mean 46 months; median 42 months). Each session was facilitated by a skilled moderator and lasted about two to three hours. The aim of the focus groups was to identify the needs, expectations, and problems of consumers with respect to health information on the internet, with emphasis on how consumers appraise the quality of such information. We excluded health professionals from the focus groups. We audiotaped, videotaped, and transcribed the focus group sessions.
Naturalistic observation of search and retrieval processes
We conducted the observational study in September 2001, with the main goal of establishing how consumers search the web, how successful they are, and what markers for quality they use. We gave eight or nine health related questions individually to six participants from the focus groups and 11 further participants and observed them in a usability laboratory setting while they searched the internet for answers. Six of the participants were men and 11 were women, including three nurses (mean age 38, median age 36, range 18-71 years). Their internet experience ranged from two months to 12 years (mean 33 months, median 24 months). We chose questions from the Heidelberg Database of Patient Questions (HEIDPAQ), an anonymised repository of health related questions asked by consumers on the internet to an “ask the doctor” service. Example questions were “If you want to travel to Australia, do you need malaria prophylaxis?” and “What is the definition of being overweight?” We gave participants a maximum of 20 minutes per question to find an answer that they felt confident about. We instructed participants to think aloud during the searches and to use the same techniques that they would use at home. No specific search engine or method was prescribed or recommended, and participants started with an empty browser screen.
The usability laboratory consists of a computer with special software (Surf Spy; Bysoft Data AB, Byske, Sweden) to log web addresses visited, a digital video camera to record the participant and the computer screen, and one or two observers sitting in the background and taking field notes.
In-depth interviews
We conducted semistructured interviews with all 17 participants in the observational study immediately after their searches to elicit decision making processes and criteria for selection of a particular site.
Results
Transcripts of focus groups and interviews
We analysed the transcripts of the focus groups and interviews for emerging themes concerning markers used by consumers to evaluate the quality of health information (box B1) and general statements about the use of the internet to obtain health information (box B2).
Box 1.
Quotes from focus groups and in-depth interviews—criteria for credibility of healthcare information on the internet
Box 2.
Quotes from focus groups and in-depth interviews—use of the internet for obtaining health information
Several criteria to assess credibility emerged from the interviews. Websites from official authorities, a professional layout, understandable and professional writing, and citation of scientific references were the most often mentioned criteria for trustworthiness. Some participants also mentioned accessibility and certain site features as quality criteria—in particular, many wished to have “a quick overview over what is on the site” (site map); search capabilities; a user friendly, uncluttered, speedy interface; the ability to send an email to the site owner; and outbound links to further recommended websites. Some participants expressed a desire to see a picture of the site owner or author—if the owner looked trustworthy, the site content was perceived to be credible as well. Other markers for quality mentioned were that the text should “sound plausible” or “sound scientific” and that treatment discussed “doesn't have any side effects” or is “natural, no chemistry.” Some participants mentioned that some form of controlling authority or an endorsement by a third party would be helpful.
A few participants noted that it is easier to assess the quality of information on the internet than elsewhere because they could verify and cross check the information on different sites. In general, consumers said that they enjoyed the new opportunities that the internet is opening up to verify what the physician says and to look for alternatives, but at the same time they stressed that they would always verify what they found on the web with their physician.
Observations during the retrieval experiment
We could do a detailed analysis of the search process for 16 of the 17 participants; one participant's log file was accidentally deleted. All 16 participants together visited 763 different web pages from 375 different sites if multiple visits of a given participant to the same page or site were counted only once, or a total of 646 unique web pages from 271 unique sites if all pages and sites were counted only once even if they were visited by more than one participant. In addition, participants used eight different search engines (Google, AltaVista, Yahoo, Web.de, dr-antonius.de, Abacho, Lycos, and Fireball.de) for a total of 280 search queries. On average, participants remained on each website for only 1 minute and 9 seconds (median 37 seconds); the three nurses spent slightly less time per page (mean 40 seconds, median 28 seconds). Although some of the websites carried a Health on the Net logo, none of the participants clicked on this logo.
Search strategy and effectiveness of searches
None of the participants used medical portals or the sites of medical societies or libraries as a starting point. Rather, they used search engines in an attempt to find relevant pages, except in two cases where for a given question participants tried to guess a web address—for example, with the pattern www.disease.de.
In most cases we felt that the search strategy used by participants was somewhat suboptimal. For all answers it would have been most effective to enter combinations of words (for example, “Australia malaria prophylaxis”), but only 99 (35%) of the 280 search queries consisted of more than one search term. Only one participant used explicit Boolean operators (in this case “AND”). Phrase searches (words enclosed in quotes to search for documents where they appear together) were used by only five participants in 10 (3.5%) of 280 searches.
As shown in the figure, participants usually chose one of the first results displayed by the search engine and then rephrased their search rather than turning to the second page and exploring further results. In 281 (97.2%) of 289 clicks on a link, the link ranked among the first 10 search results; in 206 (71.3%) of 289 cases the participants followed a link ranked among the first five results (figure). Only nine participants ever looked at the second or following pages of the search result pages, and only five of them ever clicked a link on the second or following pages.
One elderly participant thought that the 10 search results displayed on the first page were all that existed. Further suboptimal uses of the search engine included one participant who desperately tried to find German sites when entering “HIV test” but found only English pages, without realising that the search engine had an option to limit results to pages written in a particular language. Instead, he used the translation tool of Google to translate the resulting pages. None of the 13 German participants who self reported good command of English entered search terms in English to deliberately find non-German sources.
Despite the somewhat ineffective search strategies used, participants were very successful in retrieving information that enabled them to answer the questions. The 17 participants could find no web page that contained the answer to the question for only seven of the total of 136 questions given to them. (Five of these seven concerned a question about whether a 16 year old girl needs parental agreement for an abortion.) On average, it took only 5 minutes 42 seconds (median 4 minutes 18 seconds; range 38 seconds to 20 minutes) per question to find an answer. Nurses had very similar search times to non-nurses (mean 5 minutes 37 seconds v 5 minutes 45 seconds; median 4 minutes 48 seconds v 4 minutes 16 seconds). We found no correlation between internet experience and search time. In some cases, participants continued the search even after finding a page with an answer, generally because they did not understand the information encountered rather than because they did not trust the source. It was not within the scope of this study to evaluate the quality of answers retrieved by consumers, but in general we did not encounter any gross misinformation—wrong answers given by consumers were more likely to have occurred through misunderstanding on the part of the consumer.
Did consumers pay attention to the sources?
In most cases, participants used websites that they said they had not known before. Contrary to the statements made in the focus groups, in practice we observed that none of the participants actively searched for information on who stood behind the sites or how the information had been compiled; often they did not even visit the home page. In the course of the post-search interview we asked 12 of the participants, while going step by step through their answers, from which source they thought they had retrieved the particular information. Participants could correctly reproduce the name of the website or the company or organisation who stood behind the site for only 20.9% (18/86) of the answers; they could tell to which broad category the site owner belonged to (for example, government agency, public institution, university, commercial organisation, private person, self help group) for only 23.2% (20/86). In all cases, the reason for this failure was not that the site did not disclose such information but that users did not pay attention to the origin of the information.
In six cases, participants ended up on an Austrian or Swiss website when searching for legal information about abortion. Only half of them realised this fact, although these sites contained information that was not necessarily applicable in Germany.
Discussion
The results of this study indicate that consumers search for and appraise information in a different way than is implicitly assumed in many studies in which investigators assessed the quality of information on the web by entering a single search phrase and systematically evaluating the quality of all hits. This study is, to our knowledge, the first observational study carried out in a usability laboratory to investigate the retrieval strategies of people searching for health information on the web. Participants were very successful in finding answers to health questions by trying various search terms, exploring the first few hits by cursory examination of the content of the page, and iteratively refining their search strategy. However, few participants took notice and later remembered from which websites they retrieved information or who stood behind the sites.
The study has several important limitations. Firstly, we observed only a small number of participants in a somewhat artificial environment under experimental conditions. We cannot rule out the possibility that participants may have felt time pressure and thus put less emphasis on checking the quality and source. We tried to minimise this by explicitly instructing them to take their time and try to retrieve the correct answer rather than a quick answer. Secondly, as answering the questions did not have any direct impact on the participants, people in a real setting with a greater stake in the outcome of the search might care more about quality and therefore more actively look for markers of quality. Thirdly, we do not know to what degree the sample we tested was representative, but it is likely that we had a relatively experienced sample as people volunteered to participate in this study, we had three nurses in the sample, and some of our participants had taken part in the preceding focus groups. Although the nurses in our sample did not perform notably better than other participants, future studies with more participants and greater statistical power could explore whether health professionals use different strategies for search and appraisal than do consumers, are more successful than consumers, and if so why.
More observational studies are needed to design and evaluate educational and technological innovations for guiding consumers to high quality health information on the web. This is one of the important challenges in the age of “consumer health informatics.”9
Figure.
Number of times participants clicked on a search result by rank order of the link as it appeared in the list of search results
Acknowledgments
The Heidelberg Database of Patient Questions (HEIDPAQ) largely contains “ask the doctor” questions kindly supplied by Netdoktor.de and Netdoctor.co.uk
Footnotes
Funding: Partly supported by the EU Safer Internet Action Plan (http://europa.eu.int/iap/), conducted as part of the MedCERTAIN project (http://www.medcertain.org), a joint project of the Department of Clinical Social Medicine, University of Heidelberg, Germany; the Institute for Learning and Research Technology (ILRT), University of Bristol, UK; and the Finnish Office for Health Care Technology Assessment (FinOHTA) at the Finnish National Research and Development Centre for Welfare and Health (STAKES).
Competing interests: None declared.
References
- 1.Pennbridge J, Moya R, Rodrigues L. Questionnaire survey of California consumers' use and rating of sources of health care information including the internet. West J Med. 1999;171:302–305. [PMC free article] [PubMed] [Google Scholar]
- 2.Pew Internet and American Life. The online health care revolution: how the web helps Americans take better care of themselves (26 Nov 2000). www.pewinternet.org/reports/toc.asp?Report=26 (accessed 1 Nov 2001).
- 3.Quintana Y, Feightner JW, Wathen CN, Sangster LM, Marshall JN. Preventive health information on the internet. Qualitative study of consumers' perspectives. Can Fam Physician. 2001;47:1759–1765. [PMC free article] [PubMed] [Google Scholar]
- 4.Boyer C, Selby M, Scherrer JR, Appel RD. The Health on the Net code of conduct for medical and health websites. Comput Biol Med. 1998;28:603–610. doi: 10.1016/s0010-4825(98)00037-7. [DOI] [PubMed] [Google Scholar]
- 5.e-Health Ethics Initiative. e-Health code of ethics. J Med Internet Res 2000;2:e9. www.jmir.org/2000/2/e9/ (accessed 1 Nov 2001).
- 6.Nielsen J. Usability engineering. San Diego, CA: Morgan Kaufmann; 1993. [Google Scholar]
- 7.Kitzinger J. Qualitative research. Introducing focus groups. BMJ. 1995;311:299–302. doi: 10.1136/bmj.311.7000.299. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Strauss AL, Corbin JM. Basics of qualitative research: grounded theory procedures and techniques. Newbury Park, CA: Sage Publications; 1990. [Google Scholar]
- 9.Eysenbach G. Consumer health informatics. BMJ. 2000;320:1713–1716. doi: 10.1136/bmj.320.7251.1713. [DOI] [PMC free article] [PubMed] [Google Scholar]