Descriptions of Web Sites in Search Listings: A Potential Obstacle to Informed Choice of Health Information

The American public increasingly uses the Internet to obtain health information.1–5 Researchers have scrutinized the quality of health Web sites,6–10 but relatively few have studied the Web portals that deliver sites to users.11,12 Health consumers must rely on brief site listings generated by portal searches when deciding which Web site to access. These analyses examine the extent to which such site descriptions provide adequate cues about site sponsors and their intent to sell products on Web sites.

METHODS

A content analysis was conducted on Web sites generated in response to 24 search terms selected in consultation with health education experts concerning heart disease, cancer, and weight loss. Heart disease terms included heart disease, heart attacks, heart attack prevention, heart attack treatment, heart disease cure, blocked arteries, angina, and cardiac arrest. Cancer terms included cancer, cancer prevention, cancer treatment, cancer cure, skin cancer, breast cancer, prostate cancer, and lung cancer. Weight loss terms included weight loss, diets, dieting, weight control, cellulite, losing fat, burn fat, and overweight.

Each of the 24 terms was searched on the 5 most widely used portals at the time of the study in late fall 2001 (Yahoo!, Microsoft Network [MSN], America Online, Lycos, and Go),13 and the first 10 sites listed were coded, for a total of 1200 Web sites analyzed (minus 13 eliminated because of data entry errors). Coding was restricted to the first 10 listings because prior research indicates that lay users rarely go beyond the first screen or two of listings.12,14 This sample, therefore, represents the sites likely to be encountered in a lay search on these health topics at the time of study.

The first 12.5% of the sites were double-coded to establish reliability. Another 12.5% were double-coded midway through the analysis to test coder drift. Cohen κ values for intercoder reliability were 0.76 (first iteration) and 0.73 (second iteration) for whether or not text accompanying the site listing indicated that a Web site was selling products or services. Descriptive information included text with the site listing, text available when the cursor “rolled over” the site listing, and the uniform resource locator (URL) itself. It proved more difficult to reliably ascertain whether the organization sponsoring the Web site was adequately identified in the site listing description or URL, with a kappa of 0.52 for the first coding iteration and 0.63 for the second. This lower reliability reflected ambiguities in the site descriptions. Kappa is a highly conservative statistic with reliabilities of 0.5 deemed adequate for analysis and report.15,16

RESULTS

Overall, 30.2% of the site listings indicated that the corresponding Web site was selling a product. Analyses reported elsewhere determined that 39.1% of these Web sites were actually selling products.17 Therefore, 23% of the Web sites selling products (8.9% of all sites coded) were not identifiable as such based on the site listings. The difference between the number of sites selling products and those that were identifiable as such from the site listings was highly significant (t₁₁₃₇ = 8.17, P < .001).

Search listings normally began with one or more paid commercial listings, typically identified as “sponsored” sites. Most portals included a link to a disclaimer defining “sponsored” sites as paid advertisers. Only America Online included the disclaimer in the listing itself. MSN listed “featured” sites, including both recommended sites and advertisers.

Moreover, only 46.9% of the listings identified the organizational sponsor of the Web site, although the precision of this estimate must be qualified given the difficulty in unambiguously assessing whether the source could be adequately identified from the listing description.

There were no significant differences in the number of listings that clearly identified the source or in discrepancy between commercial sites located and those clearly listed, by search portal or by health topic. These were tested by using analysis of variance with search term as the unit of analysis, nested within health topic, and with search portal as a within-subjects factor.

DISCUSSION

Listings of search results provided by the most widely used Web portals often do not provide basic information a consumer would need to select an objective and reliable health information Web site. Search descriptions of health sites ideally should permit consumers, before actually accessing a Web site, to determine more consistently and accurately the source of the information and whether the site is selling products or providing information free of commercial intent. Commercial sites are of particular concern given evidence that the large majority of such sites promote unregulated supplements and unproven remedies and services17 and that pseudoscientific claims made by such promotional sites may be persuasive even to readers with college-level scientific training.18 Future research should examine how diverse members of the public use information on site source, commercial intent, and commercial sponsorship of the listing when selecting Web sites that they will use to obtain health information.