Health Information on the Internet: Accessibility, Quality, and Readability in English and Spanish

Abstract

Context

Despite the substantial amount of health-related information available on the Internet, little is known about the accessibility, quality, and reading grade level of that health information.

Objective

To evaluate health information on breast cancer, depression, obesity, and childhood asthma available through English- and Spanish-language search engines and Web sites.

Design and Setting

Three unique studies were performed from July 2000 through December 2000. Accessibility of 14 search engines was assessed using a structured search experiment. Quality of 25 health Web sites and content provided by 1 search engine was evaluated by 34 physicians using structured implicit review (interrater reliability >0.90). The reading grade level of text selected for structured implicit review was established using the Fry Readability Graph method.

Main Outcome Measures

For the accessibility study, proportion of links leading to relevant content; for quality, coverage and accuracy of key clinical elements; and grade level reading formulas.

Results

Less than one quarter of the search engine’s first pages of links led to relevant content (20% of English and 12% of Spanish). On average, 45% of the clinical elements on English- and 22% on Spanish-language Web sites were more than minimally covered and completely accurate and 24% of the clinical elements on English-and 53% on Spanish-language Web sites were not covered at all. All English and 86% of Spanish Web sites required high school level or greater reading ability.

Conclusion

Accessing health information using search engines and simple search terms is not efficient. Coverage of key information on English- and Spanish-language Web sites is poor and inconsistent, although the accuracy of the information provided is generally good. High reading levels are required to comprehend Web-based health information.

The Internet is an increasingly important source of health-related information for consumers. One recent survey estimated that more than 60 million US residents went online in search of health information in the past year.¹ The online population is becoming more representative of the larger US population in terms of race, age, income, and educational attainment.² Among those who use the Internet, more than 70% report the health information they find influences a decision about treatment.¹

The ability to obtain accurate medical information quickly, conveniently, and privately online presents to consumers an opportunity for better-informed decision making and greater participation in care.³ Little is known, however, about whether the available material is sufficiently complete and accurate to support consumer decision making. Several studies of single medical conditions have suggested deficiencies in the quality of Web-based health information.^4–11

Several organizations have developed criteria to guide and evaluate health-related Website content (eg, HON Code, American Medical Association, Internet HealthCare Coalition, Hi-Ethics, MedCertain),^12–18 but these criteria have not been systematically applied to a broad set of Web pages and conditions. Furthermore, because many of these systems rely on voluntary self-assessments by Web page developers, the reliability and validity of many of these evaluations is unknown.^19,20

Even if online materials are comprehensive and accurate, the ability of users to apply these assessment tools depends on their ability to locate and understand those materials. The Internet has the potential to eliminate barriers in access to information for patients, but only if online material can be read and understood by many different types of users.^21,22

Preliminary data from the 2000 US Census indicate that the population is becoming increasingly diverse. Since 1990, the US Hispanic population has grown from 22.3 million to 35.3 million, making Hispanics the largest minority group in the United States.²³ Among immigrant Hispanics, more than 98% report speaking primarily Spanish at home.²⁴ While accessible and high-quality health information on the Internet is important for English speakers, it could be even more useful for Spanish speakers, who face greater barriers to traditional sources of medical care and information.^25,26 We are unaware of any studies that have evaluated Spanish-language materials.

We conducted a large cross-sectional study to describe and evaluate health information on the Internet in English and Spanish. We evaluated information that we found using search engines and by visiting health-related Web sites on 4 medical conditions: breast cancer, childhood asthma, depression, and obesity. We asked 4 main questions: What are consumers likely to find when they search online about these conditions? How comprehensive is the information? How accurate is it? At what grade reading level is the material presented?

METHODS

We conducted 3 studies to assess the accessibility of relevant content; the quality of health information; and the reading grade level of text. Each study used different methods to assess the same 4 conditions. Conditions were selected by project staff based on prevalence, clinical significance, and diversity of the affected populations.^27–41 Each study was conducted independently in English and Spanish.

Study 1: Accessibility of Relevant Content

Selecting Search Engines

Search engines are designed to help people locate information on the Internet. To assess how well search engines perform this function, we selected 10 English-language and 4 Spanish-language search engines. Three of the English-language and 2 of the Spanish-language search engines were chosen based on popularity (defined as the number of unique visitors per month as reported by Media Metrix, Inc in June 2000⁴²). The remaining 9 search engines were selected because they featured unique methods of ranking Web sites.⁴³ Examples of ranking methods included ranking by location and frequency of key words within a site; ranking by the number of times a site is linked to by another site; ranking by payment from sites; and ranking by human editing.^43,44

Conducting Standardized Searches

Trained searchers entered the 4 search terms (“breast cancer,” “childhood asthma,” “depression,” and “obesity”) into each of the 14 search engines. All links on the first electronic page for each search engine were then counted and classified. Links were classified as relevant if the search term or 1 of 30 to 40 related key terms per condition (eg, tamoxifen, inhaler, gastric bypass surgery, St John’s wort) was present in the link itself or the surrounding text.

Searchers then followed a sample of relevant links to determine whether they led to relevant content. One sample was the first 5 relevant links on the search results page. All remaining links were enumerated and divided into 5 strata of equal size; 1 relevant link from each stratum was selected randomly. Searchers clicked on selected relevant links until they reached a Web page with content (defined as when 50% of the space occupied contained text that was not primarily an index of the site). If the first relevant link led to a content page, the page was saved for further analysis. If the first link led to more links, the searcher randomly selected a relevant link from the first 15 relevant links on that Web page. If searchers had not reached a content page after 10 cycles, the search was discontinued.

Characterizing Content

Using a standardized form, trained coders first classified Web page content by relevance. Web pages were coded as relevant if they contained any materials related to the 4 search terms or the 30 to 40 key terms related to each condition. Coders then assessed the relevant pages for the presence of promotional content (defined as material designed to encourage site visitors to purchase products or services or participate in research programs sponsored by the site). Explicit advertisements were classified separately from promotional material and had to be located in a banner or sidebar on the Web page.

Study 2: Quality of Health Information on the Internet

Selecting Web Sites

Eighteen unique English-language health Web sites (6 general health, 12 condition-specific) and 7 unique Spanish-language Web sites (3 general health, 4 condition-specific) were selected for this study (Table 1). We selected 6 English-language general health Web sites that were ranked highly in 2 widely used Internet industry reports, Cyber Dialogue and PC Data Online for September 2000.^45,46 Content provided by one of the most popular search engines was also included.⁴² Condition-specific English-language Web sites and all Spanish-language Web sites were selected by project staff to represent prominent examples of condition-specific Web sites from commercial, government, and nonprofit educational organizations. Project staff limited Web sites to those not requiring subscriptions or payments.

Table 1.

Selected English- and Spanish-Language Web Sites^*

	URL Address	Conditions Examined
English-Language Web Sites
Popular general health†
Allhealth.com	www.allhealth.com	Breast cancer, childhood asthma, depression, obesity
CBS Health Watch	www.cbshealthwatch.com	Breast cancer, childhood asthma, depression, obesity
DrKoop.com	www.drkoop.com	Breast cancer, childhood asthma, depression, obesity
Intelihealth	www.intelihealth.com	Breast cancer, childhood asthma, depression, obesity
Onhealth	www.onhealth.com	Breast cancer, childhood asthma, depression, obesity
WebMD	www.webmd.com	Breast cancer, childhood asthma, depression, obesity
Condition-specific‡
American Academy of Allergy, Asthma, and Immunology	www.aaaai.org	Childhood asthma
American Cancer Society	www.cancer.org	Breast cancer
American Obesity Association	www.obesity.org	Obesity
Athealth.com	www.athealth.com	Depression
Cancernet	www.cancernet.gov	Breast cancer
Depression.com§	www.depression.com	Depression
MyAsthma	www.myasthma.com	Childhood asthma
National Heart, Lung, and Blood Institute	www.nhlbi.nih.gov	Childhood asthma
National Institute of Mental Health	www.nimh.nih.gov	Depression
National Library of Medicine	www.nlm.nih.gov	Obesity
Obesity Online	www.obesity-online.com	Obesity
Oncolink	www.oncolink.com	Breast cancer
Search engine‡
Yahoo	www.yahoo.com	Breast cancer, depression, obesity
Spanish-Language Web Sites
General health‡
Graciasdoctor	www.graciasdoctor.com	Breast cancer, childhood asthma, depression, obesity
Salud	www.salud.com	Breast cancer, childhood asthma, depression, obesity
Salud Latina	www.saludlatina.com	Breast cancer, childhood asthma, depression, obesity
Condition-specific‡
Cancernet	www.cancernet.gov	Breast cancer
Centro Peso	www.centropeso.com	Obesity
National Institutes of Health	www.nih.gov	Childhood asthma
New York Online Access to Health	www.noah-health.org	Depression

^*URL indicates uniform resource locator.

^†Top 6 sites ranked by Cyber Dialogue and PC Data Online, September 2000.

^‡Selected by project staff.

^§This Web site was no longer accessible by JAMA and RAND staff as of May 7, 2001. Copies of the Web site content are available from the authors on request.

Developing Condition-Related Topics and Questions

Panels of 3 to 4 nationally recognized clinical experts and representatives from patient advocacy organizations identified 5 to 8 key clinical topic areas for each condition (26 in all). Panelists were recruited for their clinical or scientific experience, familiarity with national guidelines, current research, or national reputations in the medical conditions of interest. No panelist had consulted for, or had any financial involvement with, any e-health Web site. Panelists were asked to identify topics that were relevant to patients, their families, or laypersons seeking information on the study conditions. Panelists also considered whether it was reasonable to expect to find this information on the Web. The panels then wrote 36 consumer-oriented questions relating to the 26 topics.

For example, the topic “breast cancer screening” was characterized by the following questions: “No one in my family has had breast cancer. Do I still need breast exams and mammograms? When should I start having regular mammograms? Do I need one every year?” A complete list of all condition-related topics and questions is located in Online Table 1 (available at http://www.jama.com).

Online Table 1.

Condition-Related Topics, Consumer Questions, and Elements

Condition-Related Topics	Consumer Questions	Elements of Condition-Related Topics
Breast Cancer
Assessment of breast cancer risk and use of tamoxifen for risk reduction	Are there any medications I can take to reduce my risk of getting breast cancer?	Risk factors for breast cancer include a family or personal history of breast cancer, early menarche, pregnancy history, and a history of breast biopsies In the short run (meaning ≤ 5 years), tamoxifen reduces breast cancer risk in high-risk women
Screening for breast cancer	No one in my family has had breast cancer. Do I still need breast exams and mammograms? When should I start having regular mammograms? Do I need one every year?	Women >50 years should have mammograms every 1-2 years Early detection of breast cancer improves outcomes Most breast cancers occur in women without a family history of the disease There is a lack of consensus about the need for or appropriate interval of mammography in women from age 40-49 years
Evaluation of a palpable breast mass	I have a lump in my breast. What should be done to check this?	New breast lumps should be brought to the attention of a physician Mammography and ultrasonography are useful in evaluating lumps A negative mammogram result does not eliminate the need for further evaluation A persistent, noncystic (non-fluid-filled) breast mass palpated by a physician should be biopsied
Treatment, including primary treatment and availability of clinical trials for treatment of advanced cancers	If I have stage I or II breast cancer, which is better treatment, mastectomy or lumpectomy plus radiation? Where can I get information about breast cancer clinical trials?	Mastectomy and lumpectomy plus radiation are equivalent treatments for early-stage breast cancer Patient preferences should be considered in treatment decisions of mastectomy vs lumpectomy plus radiation Breast reconstruction is available for women who have mastectomy Clinical trials are available for women with advanced cancer; some information about finding clinical trials is given
Alternatives to standard medical and surgical treatments for breast cancer	What alternative therapies (such as acupuncture, herbs, or homeopathy) can help me fight breast cancer?	Alternative therapies to treat breast cancer have generally not been subjected to rigorous scientific studies Alternative therapies should not be used as a substitute for proven effective treatments Your physician should be informed of any alternative treatments you are using, including herbs, supplements, and over-the-counter products
Childhood Asthma
Symptoms of childhood asthma	What are the common symptoms of asthma in children?	A child with asthma can experience the following symptoms: cough, wheezing, chest tightness, shortness of breath or difficulty breathing, or an “asthma attack” (pronounced or prolonged presence of these symptoms) These symptoms can be worse at night, triggered by exercise, environmental irritants, changes in weather, viral illness, or can occur spontaneously at rest Children with asthma can have intermittent symptoms (twice a week or less) or persistent symptoms (more than twice a week) Children with intermittent symptoms may have a severe exacerbation
Symptoms suggestive of uncontrolled childhood asthma	I have been told by a physician that my child has asthma. He/she has difficulty breathing at night and uses the inhaler every day. Does this mean that my child’s asthma is not well controlled?	Children with intermittent symptoms (day symptoms twice a week or less or night symptoms twice a month or less) are considered “controlled” Children with persistent symptoms (day symptoms or need to use a rescue medication more than twice a week or waking up with symptoms during the night more than twice a month) are “not controlled”
Therapeutic modalities and associated adverse effects	What should I do about my child’s asthma, especially if it is not well controlled? Are there any medicines or special equipment that a physician can prescribe? Do they have any adverse effects?	Bronchodilator medications (eg, albuterol) open the airways (breathing passages); they are used as “quick relief” or “rescue” medications for patients whose symptoms are intermittent Inhaled corticosteroids (eg beclomethasone, flunisolide, triamcinolone) and cromolyn are 2 kinds of inhaled medications that reduce inflammation in the airways; they are used as long-term treatments for patients whose symptoms are persistent or uncontrolled A spacer device will improve delivery of inhaled medications to the lungs; such devices are required for young children and are strongly recommended for older children and adolescents Peak flow monitoring is a useful way for patients to recognize early signs of worsening asthma Oral steroids are effective for short-term exacerbations but have significant adverse effects over the long term Inhaled steroids, taken in usual doses, do not affect children’s growth; uncontrolled asthma can retard a child’s growth Alternative therapies for asthma (eg, herbal remedies and chiropractic manipulation) have not been shown to be effective Antileukotrienes are a new drug class that might be useful as an add-on to inhaled steroids or to prevent exercise symptoms in children older than 6 years; the safety and efficacy of these drugs in children younger than 6 years has not been demonstrated
Initial management of severe pediatric asthma	How do I know if my child is having life-threatening symptoms? What should I do?	Some children can die of asthma, especially if the early warning signs of a severe asthma attack are missed Signs of a life-threatening asthma episode include very difficult breathing, shortness of breath at rest, uncontrolled coughing, severe chest tightness, blueness around the lips or nails, difficulty talking, extreme tiredness or fatigue, or unresponsiveness Immediate home care for a severe asthma attack includes prompt administration of the child’s quick-relief or rescue medication If symptoms of a severe asthma attack are not relieved within 10 minutes or if the child’s symptoms worsen, the caretaker should call for emergency assistance
Childhood asthma risk factors	Could certain exposures in the indoor or outdoor environment have caused or made my child’s asthma worse? What can be done to identify, eliminate, and diminish factors in the environment that can worsen my child’s asthma symptoms?	Certain indoor allergens and irritants (eg, tobacco smoke, dust mites, cockroach allergens, cat hair) have been shown to cause worsening of acute asthma in children who are sensitive to these factors, but not to cause asthma per se Although pollution is not a proven cause of asthma, persons with asthma can experience more asthma exacerbations on high pollution days Other indoor allergens or irritants such as mold, animal dander other than cat, pollen, and strong odors have been reported to be associated with worsening asthma symptoms; however, there is scientific uncertainty about the role of these factors (for full credit, site must mention both the potential role of these allergens/irritants and uncertainty about their importance) Allergens or irritants that trigger a child’s asthma can usually be identified through a careful medical history taking; blood and skin tests conducted by an allergy specialist also can be helpful Most children being considered for immunotherapy should be evaluated and followed up by an allergy specialist Allergy immunotherapy for children with asthma should only be considered when there is clear evidence of a relationship between symptoms and exposure to an allergen to which the child is sensitive, symptoms occur all year or during a major portion of the year, and the symptoms are not controlled with medications Families of children who are sensitive to tobacco smoke, dust mites, cockroach antigens, or cats should undertake vigorous exposure reduction strategies
Etiology and risk factors	What causes asthma? Is it curable?	The cause of asthma is not known Most experts speculate that asthma may be caused by a combination of genetic (hereditary) and environmental factors (exposures) Asthma is not contagious and is not caused by psychological or psychiatric disturbances Although asthma medications can control symptoms, asthma is not curable given current science
Expectations from therapy	What should I expect from my child’s asthma treatment?	In 80%-85% of cases, children with asthma can be symptom free if they follow a preventive medication regimen and avoid allergens or irritants to which they are sensitive Even if total freedom from symptoms is not possible, the disease can be controlled so that the child experiences minimal symptoms during the day and night Children with asthma should be able to participate in normal activities (school, play, etc) and parents should not have to lose work or sleep because of children’s asthma symptoms
Depression
Symptoms of depression	I’ve been feeling a little sad lately. How do I know if I’m depressed?	The primary symptoms of depression are persistent low mood, loss of interest and enjoyment, and reduced energy lasting at least 2 weeks Other symptoms of depression include significant weight, sleep, and appetite changes, anxiety, feelings of worthlessness or inappropriate guilt, diminished ability to think or concentrate or indecisiveness, recurrent thoughts of death or suicidal ideation, apathy, or irritability; a person may have a depressive disorder without having all of these symptoms In older patients (defined as ≥65 years), depression may not always present with low mood as seen in younger patients. Instead, patients may seem apathetic and uninterested in normal activities; anxiety and memory impairment may also be the principal presenting symptoms Depression should not be regarded as a normal part of aging
Treatments for depression	What are the most effective treatments for depression?	Effective treatments for depression include prescription antidepressant drugs, specific psychological treatments (cognitive therapy, cognitive behavioral therapy, and interpersonal therapy), combination therapy, and electroconvulsive therapy No antidepressant is superior to another in efficacy or time to response; the choice of medication is based on adverse effect profile or prior response St John’s wort (Hypericum perforatum) may be an effective treatment for mild depression; however, because of reported drug interactions, patients who are taking other prescription medicines should consult with a physician before starting this preparation
Antidepressant medications	If my physician recommends an antidepressant medication for the treatment of my depression, how long should I take it for? What should I expect and when will I start to feel better?	Antidepressant medications typically begin to work within several weeks. However, many patients do not experience substantial benefits for 4-6 weeks, and it may take 3-4 months before patients taking antidepressants feel completely better Patients with a single episode of acute depression who experience initial improvement should continue to take the medication, usually for 6-12 months after they feel completely better, to keep feeling well With antidepressant medicines, many people have some adverse effects early in treatment (in the first 4-6 weeks); most adverse effects get better in the first month; for some people, the adverse effects can be bad enough to stop the medicine. Common adverse effects include anxiety, sexual dysfunction, sleepiness, trouble sleeping, weight gain/loss, restlessness, and nausea Continued antidepressant treatment can prevent recurrences of depression; the likelihood of experiencing a new episode of depression is greater with each previous episode of depression. Prolonged treatment with antidepressants (>12 months) is especially warranted when depressive episodes have been recurrent
Role of counseling	When should I consider psychological counseling instead or in addition to medication?	For mild to moderate depression, prescriptive antidepressant drugs and specific psychological therapies are equally effective For moderate to severe depression, prescription antidepressant drugs are more effective than psychological therapies For severe depression, the combination of drug therapy with psychological treatment is probably more effective than psychological therapy alone
Suicidal ideation	I feel so depressed I’ve thought about suicide. What should I do?	People who have suicidal thoughts but are confident they will not carry out suicide should obtain a medical or psychiatric evaluation promptly People with suicidal thoughts who think there is any chance they might attempt suicide should seek emergency evaluation and help from their physician or at an emergency department
Professional evaluation of depression	Who should I see for evaluation and treatment of my depression? A primary care physician, a psychiatrist, or a psychologist/therapist?	The best health professional to see for the evaluation and treatment of depression is uncertain; to date, no definitive scientific studies have proven which is best For mild depression, both the initial evaluation and subsequent treatment can be provided by a primary care physician, psychiatrist, or psychologist/therapist For moderate depression, an individual should either see a primary care physician or a psychiatrist for an initial evaluation; subsequent treatment may be provided by either the evaluating physician or a psychologist/therapist For severe depression, an individual should probably see a psychiatrist or a primary care physician for an initial evaluation; treatment should be by a physician and possibly an adjunctive therapist
Etiology	What causes depression?	The causes of depression are uncertain but it probably results from a combination of genetic predisposition and childhood and current psychosocial adversity
Obesity
Indications for weight loss, definitions of overweight and obesity	How do I know if I need to lose weight?	Body mass index (BMI, measured as kg/m2) is a useful way to determine whether someone is overweight or obese There is a distinction between overweight and obesity; overweight is currently defined as BMI between 25-29.9; obesity is defined as ≥30 Growing evidence suggests that these thresholds may be too high for certain nonwhite populations (eg, Chinese, Japanese, Hispanics) Waist circumference is by itself predictive of future morbidity; high-risk cutoffs are 89 cm (35 in) for women and 102 cm (40 in) for men The health risks of obesity also depend on disease conditions (eg, coronary artery disease, diabetes), cardiovascular risk factors (eg, family history, low-density lipoprotein cholesterol, hypertension), and other obesity-associated diseases and risk factors (eg, gallstones, degenerative joint disease) Treatment is indicated when the patient meets criteria for obesity, or when the patient meets criteria for overweight and the patient has (1) established cardiovascular disease or diabetes; (2) ≥2 other risk factors including hypertension, dyslipidemia, smoking, family history of heart disease, age ≥45 years for men or ≥55 years for women; or (3) a high waist circumference (see above)
Health risks of being overweight and obese	What are the health risks of being overweight/obese?	There is an increase in mortality as BMI exceeds 25; the risk increases rapidly above a BMI of 30 Important morbidities associated with obesity include diabetes mellitus, hypertension, abnormal blood lipids, coronary artery disease, and sleep apneas Other morbidities include gastroesophageal reflux disease, gallstones, urinary stress incontinence, and osteoarthritis In addition to medical morbidities, overweight/obesity can produce limitations in mobility, reduced functional status, and lower overall quality of life
Risks and benefits of low-carbohydrate, high-protein diets	What should I consider before starting a low-carbohydrate, high-protein, high-fat diet like the Atkins plan?	The benefits of Atkins-type diets include: good short-term weight loss; less hunger than a standard low-fat diet; and better short-term control of insulin-resistance states, including type 2 diabetes and hypertension Initial rapid weight loss is mostly water loss The long-term safety (>6 months) of these diets has not been established
Value of physical activity for weight loss, maintenance, and general health	What is the value of physical activity for promoting weight loss, maintaining weight at current levels, and for general health?	Regular physical activity results in modest weight loss, especially when combined with a low-calorie diet Physical activity is more effective at maintaining current weight than at reducing weight Physical activity benefits general health and fitness independent of weight loss Physical activity benefits some obesity-related problems (eg, diabetes and hypertension), independent of weight loss
Availability of drugs approved for weight loss	Should I consider weight-loss drugs, and if so, what prescription and nonprescription drugs are currently available?	Weight-loss drugs are a Food and Drug Administration (FDA)-approved option for patients with a BMI ≥27 (with concomitant risk factors) or ≥30 (without risk factors) FDA-approved prescription drugs for weight loss include sibutramine, orlistat, and phentermine Phenylpropanolamine (Dexatrim, Acutrim) is an over-the-counter weight-loss agent approved for short-term use (≤3 months) Phenylpropanolamine has been associated with strokes (although the magnitude of the stroke risk is not established)
Indications, risks, and benefits of weight-loss surgery	Who should consider weight-loss surgery, what are the risks, and how well does it work?	Weight-loss surgery should be considered when the BMI is ≥40, or when it is 35-39.9 in the presence of medical comorbidities (eg, diabetes, known cardiovascular disease, severe degenerative joint disease, hypertension, gastroesophageal reflux, sleep apnea) Gastric restrictive procedures (eg, vertical banded gastroplasty), gastric bypass, and malabsorptive procedures (eg, biliopancreatic diversion) have been shown to be effective Patients can achieve substantial weight loss, often >45 kg (>100 lb) Gastric bypass is somewhat more effective (in terms of weight loss) than vertical banded gastroplasty Death and major complication rates following surgery are approximately equal for gastric bypass and vertical banded gastroplasty; operative mortality is <0.5%, morbidity is approximately 5%, incisional hernia rate is approximately 5%, and small bowel obstruction occurs in 2% of cases Classic biliopancreatic diversion often results in diarrhea, flatulence, and a significant incidence of protein malnutrition
Safety and effectiveness of dietary supplements containing ephedra plus caffeine	Can herbal supplements containing ephedrine plus caffeine help me to safely lose weight?	Ephedrine (ephedra) plus caffeine has been shown to be effective as a weight-loss supplement Several safety concerns remain, especially for patients with comorbid conditions that might be worsened by sympathomimetic effects Patients who have heart disease or hypertension should consult a physician before taking this combination

Development of Clinical Elements

To enhance the consistency of the structured implicit review, the 4 clinical panels each developed a series of 1 to 8 clinical elements for each of the questions based on evidence-based guidelines and materials from selected literature reviews.^27–41 For example, for the topic of breast cancer screening, 4 clinical elements were developed. These included the following: women older than 50 years should have mammograms every 1 to 2 years; early detection of breast cancer improves outcomes; most breast cancers occur in women without a family history of the disease; and a lack of consensus exists about the need for or appropriate interval of mammography in women from age 40 to 49 years. A total of 100 clinical elements were developed (Online Table 1).

Retrieving Health Information

Four abstractors (2 monolingual in English, 2 bilingual in English and Spanish) independently reviewed each Web site (spending a maximum of 90 minutes per site using high-speed Internet connections) on October 18–30, 2000, and November 6–13, 2000, to retrieve content related to the questions (Online Table 1). Abstractors did not receive any of the condition-related clinical elements prior to conducting each search. On average, 65% of retrieved Web pages were common between abstractors. Search results were saved using a software application (Catch-TheWeb, Math Strategies, Greensboro, NC) that enabled project researchers to accurately save, abstract, and manage Web pages for later use.

Retrieved materials were stripped of identifying information, printed, and assembled into notebooks. Each notebook contained the materials retrieved from a single search on a Web site (eg, 1 condition per site). The 78 unique English-language notebooks averaged 250 printed pages (range, 21–547 printed pages). The 32 unique Spanish-language notebooks averaged 68 printed pages (range, 8–366 printed pages). A total of 21711 printed pages (2660 Web pages, defined by the programmer’s end-of-page mark) were abstracted across 4 conditions: 19529 printed pages (2262 Web pages) from English-language and 2182 printed pages (398 Web pages) from Spanish-language Web sites.

Evaluating the Web Sites

Thirty-four physicians (30 monolingual in English, 4 bilingual in English and Spanish) from around the United States were recruited to evaluate the abstractor-retrieved material. All reviewers were board eligible or board certified in family medicine, general surgery, internal medicine (including allergy and immunology, hematology and oncology, infectious diseases, pulmonary and critical care), or pediatrics. No reviewer rated more than 5 notebooks for any condition or evaluated materials from the same Web site twice. Forty English-language (51%) and 14 Spanish-language (44%) randomly selected notebooks were evaluated by 2 reviewers. Each Web site underwent 2 to 4 reviews per condition.

Four standardized rating forms were developed that listed the condition-related topics, questions, and clinical elements (eg, 1 condition per form). Reviewers were asked to rate the level of coverage for each clinical element as not addressed, minimally addressed, or more than minimally addressed. Not addressed meant there was no reference to the issue on any page of the notebook. Minimally addressed meant the clinical element was mentioned at least briefly. For example, for breast cancer screening, if mammography was mentioned as a way to identify early breast cancer, but no mention was made of who should have mammograms, how often they should be done, or their utility in reducing breast cancer mortality, this was considered minimal coverage. More than minimally addressed meant that most of the clinical elements were mentioned and the level of explanation was more than cursory. For example, coverage was considered more than minimal if a Web site mentioned that screening mammography was the best way for breast cancer to be detected early in women older than 50 years, or that breast cancer may be detected earlier by mammography than physical examination, or if a detailed discussion of the pros and cons of mammography and the appropriate ages for screening were provided.

Reviewers also rated the accuracy of content for each clinical element that was at least minimally addressed: mostly incorrect, mostly correct, and completely correct.

After rating Web site materials on coverage and accuracy, reviewers were asked to list instances of conflicting information found during their review. These conflicts were not limited to the set of clinical elements for which coverage and accuracy were evaluated. Six categories of conflicting information were identified: (1) treatments; (2) diagnosis; (3) definitions; (4) adverse effects; (5) etiology and risk factors; and (6) incidence and prevalence. Two project physicians (R.L.K. and J.I.A.) independently rated whether the examples of conflicting information were minor, significant, or potentially dangerous. Examples that were identified as significant or potentially dangerous by both physicians were included in the final analysis.

Analytic Methods

We used Stata statistical software (version 6.0; Stata Corporation, College Station, Tex). The unit of analysis was the link (specific URL [uniform resource locator]) for the study of search engine efficiency, the standardized rating form for the study of quality, and the Web site for the study of grade reading level.

Rating forms contained multiple ratings (corresponding to clinical elements) of coverage and accuracy using the 3-point ordinal scales mentioned previously. For purposes of analysis, summary measures were computed by averaging across elements within a given rating form.

All analyses were conducted separately for English- and Spanish-language search engines and Web sites. All statistical tests were 2-sided and were assessed for significance at the .05 level. Measures were tested for variation by condition, search engine, and site, as applicable. A 2-stage test procedure was used to examine variation in each outcome by these independent categorical variables. First, an omnibus or overall test of the association was performed. If the omnibus test established that variation in the outcome of interest was statistically significant for the categorical variable (condition, search engine, site), a series of 2-sample follow-up tests were performed comparing the outcome at each level of the categorical variable with the overall distribution of the outcome.

The omnibus tests used were 1-way analysis of variance, the Kruskal-Wallis rank-sum test, and the χ² test of homogeneity for measures that were normally, ordinally, and nominally distributed, respectively. Two-sample t tests, Wilcoxon rank-sum tests, and χ² tests of homogeneity were the corresponding follow-up tests.

Interrater Reliability

In the search engine study, interrater reliability of the judgments by searchers and coders was high for both classification of links and content (κ≥0.80).⁴⁷

In the Web site study, 2 measures of interrater reliability of Web site reviewers were computed. A standard measure of reliability, computed as the correlation in ratings between reviewers examining identical notebooks of material retrieved from the same Web site, was calculated. To assess the sensitivity of reviewer ratings to variation in the retrieved material (eg, the material retrieved by abstractor 1 vs abstractor 2 on the same Web site for the same condition), a second, more stringent measure of reliability was computed as the correlation in ratings between reviewers examining different notebooks of material from the same Web site and condition. We computed 16 interrater reliabilities by the standard rule and 16 by the stringent rule for each language: 1 for every combination of the 4 conditions and the 4 assessments (any coverage, more than minimal coverage, completely correct, and the combination of more than minimal coverage and complete correctness). Thirty reviews were included in each calculation of interrater reliability on English-language Web sites and 12 reviews were included in each calculation of interrater reliability on Spanish-language Web sites. The standard interrater reliability was 0.90 or greater for all conditions and measures, averaging 0.96 for both English- and Spanish-language sites. The second measure of interrater reliability averaged 0.77 for English- and 0.60 for Spanish-language Web sites.

Study 3: Reading Grade Levels of Web Sites

To determine reading grade level, we used the Fry Readability Graph (FRG) method, which has been validated in both English and Spanish.^48,49 Three sample passages of text exactly 100 words in length from the beginning, middle, and end of the material abstracted from each Web site were selected. For each 100-word sample, the number of sentences and syllables were counted. The FRG calculates an estimated grade level as a function of the average number of sentences and average number of syllables for each source document. The FRG accounts for the fact that Spanish documents tend to have more syllables per word than English documents of the same reading level.⁵⁰

RESULTS

Efficiency of Searches and Type of Information

Figure 1 and Table 2 summarize the experience that someone seeking information would have when using a search engine. The first page of search results from all English-language search engines listed 3735 links, 1265 (34%) of which were relevant. The proportion of these links that were relevant varied significantly by search engine (P<.001). Among 389 sampled relevant links, 288 (74%) selected led to a content page within 10 clicks, and 230 (79%) of those pages contained content relevant to the search topic. Thus, when following apparently relevant links, relevant content was identified 59% of the time (Figure 1 and Table 2). There was significant variation in the likelihood of reaching relevant content from potentially relevant links by search engine (range, 35%–88%, P<.001, Table 2). One in 5 (20%) links on the first page of search results led to relevant content (Table 2). There was no significant variation among search engines in the probability that first-page links would lead to relevant content.

Figure 1.

Flow of Search Engine Study

Table 2.

Search Engine Characteristics and Efficiency in Reaching Relevant Content

Search Engine	No. of Links on First Results Page (4 Medical Conditions)	Relevant Links From First Results Page, No. (%)	Selected Relevant Links Leading to Relevant Content, %	Proportion of All Links Leading to Relevant Content, %*
English-Language Engines
Overall	3735	1265 (34)	59	20
Altavista	282	142 (50)	35†	18
Ask Jeeves	1198	299 (25)†	68	17
Direct Hit	302	129 (43)‡	38†	16
Excite	232	61 (26)†	60	16
Google	163	51 (31)	74‡	23
Goto	344	155 (45)‡	43†	19
Lycos	353	132 (37)	50	19
Metacrawler	384	153 (40)‡	70	28
Northern Light	241	55 (23)†	88‡	20
Yahoo	236	88 (37)	67	25
Spanish-Language Engines
Overall	1685	296 (18)	63	12
Quepasa	283	52 (18)	59	11
Te Respondo	853	105 (12)†	78‡	9
Yahoo Espanol	181	63 (35)‡	66	23
Yupi	368	76 (21)	49†	10

^*Assuming randomly selected relevant links are representative of all relevant links.

^†Statistically worse performance than the mean of other search engines of this language (P≤.05).

^‡Statistically better performance than the mean of other search engines of this language (P≤.05).

Results for Spanish-language search engines were similar. The first page of results returned 1685 links, 296 (18%) of which were relevant. Among the 151 selected relevant links, 101 (67%) led to content and 95 (94%) of those pages contained relevant content. Overall, 63% of relevant links led to relevant content (Figure 1 and Table 2). There was significant variation in the likelihood of reaching relevant content by search engine (range, 49%–78%, P<.001, Table 2). Twelve percent of all links on the first page of search results led to relevant content, with no significant variation by search engine (Table 2).

Fifty-six percent (n=129) of the relevant English-language content pages contained explicit advertisements and 44% (n=101) contained other promotional material. The presence of advertisements and promotional materials on relevant Spanish-language content pages was 36% (n = 34) and 21% (n=20), respectively.

Quality of Health Information

Coverage of Topics

Coverage is reported as the mean proportion of clinical elements across sites with no coverage; minimal coverage; and more than minimal coverage of the clinical elements for each condition. Among English-language sites, the mean percentage of clinical elements that were not covered varied significantly across conditions: 16% for breast cancer, 27% for childhood asthma, 20% for depression, and 35% for obesity (Table 3). Topics that were not covered most often included alternatives to standard medical and surgical treatments for breast cancer (28%), symptoms suggestive of poorly controlled asthma (48%), evaluation of depression (33%), and safety and effectiveness of dietary supplements used for obesity (61%).

Table 3.

Comparison of Coverage and Accuracy of Selected Condition-Related Topics for 4 Conditions

	Average for 10 English-Language Web Sites per Condition*							Average for 4 Spanish-Language Web Sites per Condition
	Coverage, %			Correctness, %				Coverage, %			Correctness, %
Condition-Related Topics	None	Minimal	More Than Minimal	Mostly Not	Mostly	Completely	More Than Minimal Coverage and Completely Correct, %	None	Minimal	More Than Minimal	Mostly Not	Mostly	Completely	More Than Minimal Coverage and Completely Correct, %
Breast Cancer
Overall§	16†	17	67†	0	9	91	63†	49	12	39†	0	4	96	39†
Risk assessment and use of tamoxifen for risk reduction	10	12	78	0	11	89	73	45	14	41	0	17	83	36
Screening	10	11	79	1	13	86	69	27	12	61	0	0	100	61
Evaluation of a palpable breast mass	18	25	57	0	7	93	57	30	20	50	0	3	97	50
Treatment	13	14	73	2	6	92	70	61	12	27	0	12	88	27
Alternatives to standard surgical and medical treatments	28	23	49	0	4	96	49	90	0	10	0	0	100	10
Childhood Asthma
Overall§	27	30	43‡	3	13	84	36‡	33†	40	27	4	43	53	23
Symptoms	33	26	41	2	9	89	36	25	31	44	3	36	61	40
Poorly controlled asthma	48	29	23	19	9	72	18	42	33	25	14	29	57	25
Therapies and adverse effects	13	22	65	1	23	76	48	27	44	29	6	38	56	27
Initial management of severe asthma	33	46	21	1	15	84	19	40	50	10	3	59	38	10
Risk factors	29	32	39	0	16	84	32	32	42	26	0	53	47	21
Etiology	32	22	46	0	2	98	46	31	36	33	4	44	52	23
Expectations from therapy	23	41	36	0	10	90	36	44	39	17	0	44	56	14
Depression
Overall§	20†	27	53	3	22	75	44	61	24	15‡	18	19	63	12‡
Symptoms	13	15	72	1	17	82	61	45	37	18	5	31	64	14
Treatments	17	11	72	2	30	68	56	42	22	36	9	36	55	24
Antidepressant medications	16	17	67	0	22	78	55	77	14	9	0	6	94	9
Role of counseling	31	29	40	1	26	73	33	64	27	9	17	30	53	9
Suicidal ideation	12	46	42	5	11	84	37	55	40	5	14	36	50	5
Evaluation	33	54	13	6	23	71	8	84	16	0	67	0	33	0
Etiology	0	3	97	3	7	90	87	27	9	64	13	12	75	45
Obesity||
Overall§	35‡	25	40‡	3	11	86	37‡	69‡	15	16	0	32	68	15
Definitions and indications for weight loss	36	16	48	4	18	78	42	50	19	31	0	19	81	31
Health risks	12	29	59	1	9	90	56	23	50	27	0	48	52	21
Risk and benefits of popular diets	49	34	17	5	8	87	17	100	0	0	. . .	. . .	. . .	0
Physical activity/prevention	20	34	46	0	6	94	43	54	17	29	0	37	63	29
Medications endorsed for weight loss	47	25	28	0	17	83	25	79	11	10	0	50	50	8
Surgery	32	22	46	8	7	85	44	90	6	4	0	29	71	4
Safety and effectiveness of dietary supplements	61	19	20	0	4	96	19	100	0	0	. . .	. . .	. . .	0
Mean of all 4 conditions	24	25	51	2	13	84	45	53	23	24	6	24	70	22

^*Averages taken from 10 English-language Web sites with the exception of childhood asthma, for which 9 were used.

^†Statistically better performance than condition average within the same language (P≤.05).

^‡Statistically lower performance than condition average within the same language (P≤.05).

^§Weighting all clinical elements equally, rather than weighting all topics equally. See Online Table 2.

^||Ellipses indicate that an element was not addressed and could not be assessed as correct.

On Spanish-language Web sites, the mean percentage of clinical elements receiving no coverage also varied significantly across conditions: 49% for breast cancer, 33% for childhood asthma, 61% for depression, and 69% for obesity (Table 3). Topics that were not covered most often included alternatives to standard medical and surgical treatments for breast cancer (90%), expected benefits and possible adverse effects of asthma therapies (44%), evaluation of depression (84%), safety and effectiveness of dietary supplements (100%), and types of popular diets for obesity (100%).

Accuracy of Information

On English-language Web sites, the mean percentage of covered clinical elements for which the text was completely correct was 91% for breast cancer, 84% for childhood asthma, 75% for depression, and 86% for obesity. In Spanish, the mean percentages were 96% for breast cancer, 53% for childhood asthma, 63% for depression, and 68% for obesity.

On English-language sites, the mean percentages of covered clinical elements rated as mostly incorrect were 0% for breast cancer, 3% for childhood asthma, 3% for depression, and 3% for obesity. In Spanish, the mean proportions were 0% for breast cancer, 4% for childhood asthma, 18% for depression, and 0% for obesity. As an example, one depression site stated that omega-3 fatty acid deficiencies cause major depressive disorders. One childhood asthma site describes cockroaches as the leading cause of asthma among children.

Combined Measure of Coverage and Accuracy

In English, the mean percentage of clinical elements receiving more than minimal coverage that were completely accurate was 63% for breast cancer, 36% for childhood asthma, 44% for depression, and 37% for obesity. For breast cancer, depression, and obesity, there was significant variation among English-language Web sites (Table 3). Two sites performed statistically better than average: www.Oncolink.com for breast cancer and www.nimh.gov for depression (for both, P=.02). No Web site was statistically better than the condition average for childhood asthma and obesity.

On Spanish-language Web sites, the corresponding proportions receiving more than minimal coverage that were completely accurate were 39% for breast cancer, 23% for childhood asthma, 12% for depression, and 15% for obesity. There was significant variation among Web sites for breast cancer and depression (P<.05), but no Web site was statistically better than the condition average.

For a comprehensive summary of coverage and accuracy of elements of condition-related topics for the 4 conditions, see Online Table 2.

Online Table 2.

Comparison of Coverage and Accuracy of Elements of Condition-Related Topics for 4 Conditions

	Average for 10 English-Language Web Sites per Condition*							Average for 4 Spanish-Language Web Sites per Condition
Condition-Related Topics	Coverage, %			Correctness, %				Coverage, %			Correctness, %
Topic Areas and Elements of Condition-Related Topics	None	Minimal	More Than Minimal	Mostly Not	Mostly	Completely	More Than Minimal Coverage and Completely Correct, %	None	Minimal	More Than Minimal	Mostly Not	Mostly	Completely	More Than Minimal Coverage and Completely Correct, %
Breast Cancer
Risk assessment and use of tamoxifen for risk reduction
Risk factors for breast cancer include a family or personal history of breast cancer, early menarche, pregnancy history, and a history of breast biopsies	10	10	80	0	11	89	73	45	10	45	0	33	67	36
In the short run (meaning up to 5 years), tamoxifen reduces breast cancer risk in high-risk women	10	13	77	0	11	89	73	45	19	36	0	0	100	36
Screening for breast cancer
Women >50 years should have mammograms every 1–2 years	3	0	97	0	17	83	80	0	36	64	0	0	100	64
Early detection of breast cancer improves outcomes	0	7	93	0	3	97	90	0	9	91	0	0	100	91
Most breast cancers occur in women without a family history of the disease	17	16	67	0	4	96	67	55	0	45	0	0	100	46
There is a lack of consensus about the need for or appropriate interval of mammography in women from age 40–49 years	20	20	60	4	33	63	40	55	0	45	0	0	100	46
Evaluation of a palpable breast mass
New breast lumps should be brought to the attention of a physician	3	14	83	0	3	97	83	9	0	91	0	0	100	91
Mammography and ultrasonography are useful in evaluating lumps	10	13	77	0	4	96	77	0	55	45	0	0	100	46
A negative mammogram result does not eliminate the need for further evaluation	40	33	27	0	11	89	27	73	0	27	0	0	100	27
A persistent, noncystic (non-fluid-filled) breast mass palpated by a physician should be biopsied	20	40	40	0	12	88	40	36	28	36	0	14	86	36
Treatment, including primary treatment and availability of clinical trials for treatment of advanced cancers
Mastectomy and lumpectomy plus radiation are equivalent treatments for early-stage breast cancer	7	16	77	4	7	89	70	45	19	36	0	0	100	36
Patient preferences should be considered in treatment decisions of mastectomy vs lumpectomy plus radiation	13	20	67	0	8	92	67	91	0	9	0	0	100	9
Breast reconstruction is available for women who have mastectomy	13	10	77	0	0	100	77	73	0	27	0	0	100	27
Clinical trials are available for women with advanced cancer; some information about finding clinical trials is given	17	10	73	0	12	88	67	36	28	36	0	14	86	36
Alternatives to standard medical and surgical treatments for breast cancer
Alternative therapies to treat breast cancer have generally not been subjected to rigorous scientific studies	23	24	53	0	4	96	53	91	0	9	0	0	100	9
Alternative therapies should not be used as a substitute for proven effective treatments	37	23	40	0	0	100	40	91	0	9	0	0	100	9
Your physician should be informed of any alternative treatments you are using, including herbs, supplements, and over-the-counter preparations	23	24	53	0	4	96	53	91	0	9	0	0	100	9
Childhood Asthma
Symptoms
A child with asthma can experience the following symptoms: coughing, wheezing, chest tightness, shortness of breath or difficulty breathing or an “asthma attack” (pronounced or prolonged presence of these symptoms)	4	21	75	0	0	100	75	0	50	50	0	58	42	33
These symptoms can be worse at night, triggered by exercise, environmental irritants, changes in weather, viral illness, or can occur spontaneously at rest	0	54	46	0	11	89	36	0	25	75	0	25	75	75
Children with asthma can have intermittent symptoms (twice a week or less) or persistent symptoms (more than twice a week)	46	22	32	13	27	60	21	33	34	33	12	25	63	33
Children with intermittent symptoms may have a severe exacerbation	82	7	11	0	20	80	11	67	16	17	0	25	75	17
Symptoms suggestive of uncontrolled childhood asthma
Children with intermittent symptoms (day symptoms twice a week or less or night symptoms twice a month or less) are considered “controlled”	54	25	21	23	8	69	18	42	33	25	14	29	57	25
Children with persistent symptoms (day symptoms or need to use a rescue medication more than twice a week or waking up with symptoms during the night more than twice a month) are “not controlled”	43	32	25	19	12	69	18	42	33	25	14	29	57	25
Therapeutic modalities and associated adverse effects
Bronchodilator medications (eg, albuterol, Proventil,Ventolin) open the airways (breathing passages); they are used as “quick relief”’ or “rescue” medications for patients whose symptoms are intermittent	0	21	79	7	14	79	61	0	50	50	0	25	75	50
Inhaled corticosteroids (eg, beclomethasone, flunisolide, triamcinolone, and cromolyn are 2 kinds of inhaled medications that reduce inflammation in the airways; they are used as long-term treatments for patients whose symptoms are persistent or uncontrolled	0	18	82	0	36	64	50	0	42	58	0	25	75	58
A spacer device will improve delivery of inhaled medications to the lungs; such devices are required for young children and are strongly recommended for older children and adolescents	4	42	54	0	15	85	43	50	25	25	17	33	50	17
Peak flow monitoring is a useful way for patients to recognize early signs of worsening asthma	7	4	89	0	15	85	75	33	0	67	0	0	100	67
Oral steroids are effective for short-term exacerbations but have significant adverse effects over the long term	0	4	96	0	11	89	86	8	59	33	0	55	45	25
Inhaled steroids, taken in usual doses, do not affect children’s growth; uncontrolled asthma can retard a child’s growth	18	32	50	4	44	52	25	42	58	0	29	42	29	0
Alternative therapies for asthma (eg, herbal remedies and chiropractic manipulation) have not been shown to be effective	61	3	36	0	36	64	25	75	25	0	0	67	33	0
Antileukotrienes are a new drug class that might be useful as an add-on to inhaled steroids or to prevent exercise symptoms in children older than 6 years; the safety and efficacy of these drugs in children younger than 6 years has not been demonstrated	14	50	36	0	29	71	21	8	92	0	9	64	27	0
Initial management of severe childhood asthma
Some children can die of asthma, especially if the early warning signs of a severe asthma attack are missed	36	32	32	0	11	89	32	42	41	17	0	71	29	17
Signs of a life-threatening asthma episode include very difficult breathing, shortness of breath at rest, uncontrolled coughing, severe chest tightness, blueness around the lips or nails, difficulty talking, extreme tiredness or fatigue, or unresponsiveness	25	68	7	0	10	90	7	25	75	0	0	67	33	0
Immediate home care for a severe asthma attack includes prompt administration of the child’s quick-relief or rescue medication	25	36	39	5	14	81	32	25	50	25	11	45	44	25
If symptoms of a severe asthma attack are not relieved within 10 minutes or if the child’s symptoms worsen, the caretaker should call emergency services	46	47	7	0	20	80	4	67	33	0	0	75	25	0
Risk factors
Certain indoor allergens and irritants (eg, tobacco smoke, dust mites, cockroach allergens, cat hair) have been shown to cause worsening of acute asthma in children who are sensitive to these factors, but not to cause asthma per se	0	25	75	0	32	68	43	8	42	50	0	64	36	33
Although pollution is not a proven cause of asthma, persons with asthma can experience more asthma exacerbations on high pollution days	29	32	39	0	10	90	36	42	41	17	0	43	57	17
Other indoor allergens or irritants such as mold, animal dander other than cat, pollen, and strong odors have been reported to be associated with worsening asthma symptoms; however, there is scientific uncertainty about the role of these factors (for full credit, must mention both the potential role of these allergens/irritants and uncertainty about their importance)	4	82	14	0	33	67	4	8	92	0	0	82	18	0
Allergens or irritants that trigger a child’s asthma can usually be identified through a careful medical history taking; blood and skin tests conducted by an allergy specialist also can be helpful	29	35	36	0	5	95	36	33	42	25	0	25	75	25
Most children being considered for immunotherapy should be evaluated and followed up by an allergy specialist	75	7	18	0	0	100	18	58	9	33	0	20	80	25
Allergy immunotherapy for children with asthma should only be considered when there is clear evidence of a relationship between symptoms and exposure to an allergen to which the child is sensitive, symptoms occur all year or during a major portion of the year, the symptoms are not controlled with medications	68	28	4	0	11	89	4	67	33	0	0	75	25	0
Families of children who are sensitive to tobacco smoke, dust mites, cockroach antigens, or cats should undertake vigorous exposure reduction strategies	0	11	89	0	0	100	89	8	34	58	0	36	64	50
Etiology and risk factors
The cause of asthma is not known	25	18	57	0	10	90	57	33	42	25	0	37	63	17
Most experts speculate that asthma may be caused by a combination of genetic (hereditary) and environmental factors (exposures)	7	29	64	0	0	100	64	17	33	50	0	40	60	25
Asthma is not contagious and is not caused by psychological or psychiatric disturbances	79	21	0	0	0	100	0	75	25	0	33	67	0	0
Although asthma medications can control symptoms, asthma is not curable given current science	18	21	61	0	0	100	61	0	42	58	8	42	50	50
Expectations from therapy
In 80%–85% of cases, children with asthma can be symptom free if they follow a preventive medication regimen and avoid allergens or irritants to which they are sensitive	30	45	25	0	10	90	25	33	42	25	0	25	75	25
Even if 100% freedom from symptoms is not possible, the disease can be controlled so that the child experiences minimal symptoms during the day and night	19	49	32	0	9	91	32	33	50	17	0	37	63	17
Children with asthma should be able to participate in normal activities (school, play, etc) and parents should not have to lose work or sleep because of children’s asthma symptoms	22	28	50	0	9	91	50	67	25	8	0	100	0	0
Depression
Symptoms
The primary symptoms of depression are persistent low mood, loss of interest and enjoyment, and reduced energy lasting at least 2 weeks	0	0	100	0	20	80	80	0	82	18	9	36	55	18
Other symptoms of depression include significant weight, sleep, and appetite changes, anxiety, feelings of worthlessness or inappropriate guilt, diminished ability to think or concentrate or indecisiveness, recurrent thoughts of death or suicidal ideation, apathy or irritability; a person may have a depressive disorder without having all of these symptoms	0	7	93	0	20	80	77	0	45	55	0	27	73	36
In older patients (defined as ≥65 years), depression may not always present with low mood as seen in younger patients; instead, patients may seem apathetic and uninterested in normal activities; anxiety and memory impairment may also be the principal presenting symptoms	13	47	40	4	11	85	33	82	18	0	0	0	100	0
Depression should not be regarded as a normal part of aging	40	7	53	0	11	89	53	100	0	0	. . .	. . .	. . .	0
Treatments
Effective treatments for depression include prescription antidepressant drugs, specific psychological treatments (cognitive therapy, cognitive behavioral therapy, and interpersonal therapy), combination therapy, and electroconvulsive therapy	0	3	97	0	33	67	67	0	27	73	9	27	64	55
No antidepressant is superior to another in efficacy or time to response; the choice of medication is based on adverse effect profile or prior response	23	10	67	4	26	70	53	45	28	27	0	33	67	9
St John’s wort (Hypericum perforatum) may be an effective treatment for mild depression; however, because of reported drug interactions, patients who are taking other prescription medicines should consult with a physician before starting this preparation	27	20	53	0	18	82	47	82	9	9	0	50	50	9
Antidepressant medications
Antidepressant medications typically begin to work within several weeks; however, many patients do not experience substantial benefits for 4–6 weeks, and it may take 3–4 months before people taking antidepressants feel completely better	13	20	67	0	46	54	40	64	27	9	0	25	75	9
Patients with a single episode of acute depression who experience initial improvement should continue to take the medication, usually for 6–12 months after they feel completely better to keep feeling well	23	7	70	0	22	78	57	82	9	9	0	0	100	9
With antidepressant medicines, many people have some adverse effects early in treatment (in the first 4–6 weeks); most adverse effects get better in the first month; for some people, the adverse effects can be bad enough to stop the medicine. Common adverse effects include anxiety, sexual dysfunction, sleepiness, trouble sleeping, weight gain/loss, restlessness, and nausea	7	20	73	0	11	89	70	82	9	9	0	0	100	9
Continued antidepressant treatment can prevent recurrences of depression; the likelihood of experiencing a new episode of depression is greater with each previous episode of depression. Prolonged treatment with antidepressants (ie, >12 months) is especially warranted when depressive episodes have been recurrent	20	23	57	0	8	92	53	82	9	9	0	0	100	9
Role of counseling
For mild to moderate depression, prescriptive antidepressant drugs and specific psychological therapies are equally effective	27	26	47	0	27	73	37	55	36	9	20	20	60	9
For moderate to severe depression, prescription antidepressant drugs are more effective than psychological therapies	33	34	33	5	10	85	30	73	27	0	33	34	33	0
For severe depression, the combination of drug therapy with psychological treatment is probably more effective than psychological therapy alone	33	27	40	0	25	75	33	64	18	18	0	25	75	18
Suicidal ideation
People who have suicidal thoughts but are confident they will not carry out suicide should obtain a medical or psychiatric evaluation promptly	17	56	27	4	16	80	27	36	55	9	14	29	57	9
People with suicidal thoughts who think there is any chance they might attempt suicide should seek emergency evaluation and help from their physician or at an emergency department	7	36	57	7	7	86	47	73	27	0	0	67	33	0
Professional evaluation of depression
The best health professional to see for the evaluation of and treatment for depression is uncertain; to date, no definitive scientific studies have proven who is best	60	37	3	17	0	83	3	100	0	0	. . .	. . .	. . .	0
For mild depression, both the initial evaluation and subsequent treatment can be provided by a primary care physician, psychiatrist, or psychologist/therapist	17	66	17	0	36	64	7	73	27	0	67	0	33	0
For moderate depression, an individual should see either a primary care physician or a psychiatrist for an initial evaluation; subsequent treatment may be provided by either the evaluating physician or a psychologist/therapist	27	63	10	9	23	68	7	82	18	0	50	0	50	0
For severe depression, an individual should probably see a psychiatrist or a primary care physician for an initial evaluation; treatment should be by a physician and possibly an adjunctive therapist	27	50	23	9	23	68	17	82	18	0	50	0	50	0
Etiology
The causes of depression are uncertain but it probably results from a combination of genetic predisposition and childhood and current psychosocial adversity	0	3	97	3	7	90	87	27	9	64	12	13	75	45
Obesity
Indications for weight loss, definitions of overweight and obesity
Body mass index (BMI, kg/m2) is a useful way to determine whether someone is overweight or obese	13	14	73	0	19	81	63	8	9	83	0	0	100	83
There is a distinction between overweight and obesity; overweight is currently defined as BMI between 25–29.9; obesity is defined as BMI ≥30	13	10	77	8	23	69	60	25	8	67	0	11	89	67
Growing evidence suggests that these thresholds may be too high for certain nonwhite populations (eg, Chinese, Japanese, Hispanics)	97	3	0	0	100	0	0	92	8	0	0	100	0	0
Waist circumference is by itself predictive of future morbidity; high-risk cutoffs are 89 cm (35 in) for women and 102 cm (40 in) for men	33	10	57	0	5	95	57	75	17	8	0	0	100	8
The health risks of obesity also depend on disease conditions (eg, coronary artery disease, diabetes), cardiovascular risk factors (eg, family history, LDL cholesterol, hypertension), and other obesity-associated diseases and risk factors (eg, gallstones, degenerative joint disease)	23	17	60	0	17	83	57	50	42	8	0	33	67	8
Treatment is indicated when the patient meets criteria for obesity, or when the patient meets criteria for overweight and the patient has (1) established cardiovascular disease or diabetes; (2) ≥2 other risk factors including hypertension, dyslipidemia, smoking, family history of heart disease, age ≥45 years for men or ≥55 years for women; or (3) a high waist circumference	33	44	23	5	20	75	17	50	33	17	0	50	50	17
Health risks of being overweight and obese
There is an increase in mortality as BMI exceeds 25; the risk increases rapidly above a BMI of 30	23	44	33	4	26	70	27	33	67	0	0	37	63	0
Important morbidities associated with obesity include diabetes, hypertension, abnormal blood lipids, coronary artery disease, and sleep apnea	0	10	90	0	3	97	87	0	50	50	0	42	58	42
Other morbidities include gastroesophageal reflux disease, gallstones, urinary stress incontinence, and osteoarthritis	13	37	50	0	0	100	50	25	50	25	0	56	44	17
In addition to medical morbidities, overweight/obesity can produce limitations in mobility, reduced functional status, and lower overall quality of life	10	27	63	0	7	93	60	33	34	33	0	37	63	25
Risks and benefits of low-carbohydrate, high-protein diets
The benefits of Atkins-type diets include good short-term weight loss; less hunger than a standard low-fat diet; and better short-term control of insulin-resistance states, including type 2 diabetes and hypertension	53	47	0	0	14	86	0	100	0	0	. . .	. . .	. . .	0
Initial rapid weight loss is mostly water loss	37	36	27	5	6	89	27	100	0	0	. . .	. . .	. . .	0
The long-term safety (beyond 6 months) of these diets has not been established	57	20	23	0	0	100	23	100	0	0	. . .	. . .	. . .	0
Value of physical activity for weight loss, maintenance, and general health
Regular physical activity results in modest weight loss, especially when combined with a low-calorie diet	10	27	63	0	0	100	63	17	41	42	0	30	70	42
Physical activity is more effective at maintaining current weight than at reducing weight	37	46	17	0	16	84	13	100	0	0	. . .	. . .	. . .	0
Physical activity benefits general health and fitness independent of weight loss	13	30	57	0	8	92	50	50	8	42	0	17	83	42
Physical activity benefits some obesity-related problems (eg, diabetes and hypertension), independent of weight loss	20	33	47	0	4	96	47	50	17	33	0	33	67	33
Availability of drugs approved for weight loss
Weight-loss drugs are an FDA-approved option for patients with a BMI ≥27 (with concomitant risk factors) or≥30 (without risk factors)	33	27	40	0	10	90	40	50	8	42	0	33	67	33
FDA-approved prescription drugs for weight loss include sibutramine, orlistat, and phentermine	20	33	47	0	25	75	40	67	33	0	0	75	25	0
Phenylpropanolamine is an over-the-counter weight-loss agent approved for short-term use (≤3 months)	57	33	10	0	23	77	3	100	0	0	. . .	. . .	. . .	0
Phenylpropanolamine has been associated with strokes (although the magnitude of the stroke risk is not established)	77	6	17	0	0	100	17	100	0	0	. . .	. . .	. . .	0
Indications, risks, and benefits of weight-loss surgery
Weight-loss surgery should be considered when the BMI is ≥40 or when it is 35–39.9 in the presence of medical comorbidities (eg, diabetes, known cardiovascular disease, severe degenerative joint disease, hypertension, gastroesophageal reflux, sleep apnea)	10	27	63	7	8	85	63	83	9	8	0	50	50	8
Gastric restrictive procedures (eg, vertical banded gastroplasty), gastric bypass, and malabsorptive procedures (eg, biliopancreatic diversion) have been shown to be effective	20	13	67	0	8	92	63	83	9	8	0	0	100	8
Patients can achieve substantial weight loss, often ≥45 kg (≥100 lb)	20	20	60	0	4	96	60	83	9	8	0	50	50	8
Gastric bypass is somewhat more effective(in terms of weight loss) than vertical banded gastroplasty	53	7	40	0	7	93	37	100	0	0	. . .	. . .	. . .	0
Death and major complication rates following surgery are approximately equal for gastric bypass and vertical banded gastroplasty; operative mortality is <0.5%, morbidity is approximately 5%, incisional hernia rate is approximately 5%, and small bowel obstruction occurs in 2% of cases	27	53	20	0	14	86	20	92	8	0	0	0	100	0
Classic biliopancreatic diversion often results in diarrhea, flatulence, and a significant incidence of protein malnutrition	63	14	23	8	1	91	20	100	0	0	. . .	. . .	. . .	0
Safety and effectiveness of dietary supplements containing ephedra plus caffeine
Ephedrine (ephedra) plus caffeine has been shown to be effective as a weight-loss supplement	77	16	7	0	14	86	7	100	0	0	. . .	. . .	. . .	0
Several safety concerns remain, especially for patients with comorbid conditions that might be worsened by sympathomimetic effects	40	20	40	0	0	100	40	100	0	0	. . .	. . .	. . .	0
Patients who have heart disease or hypertension should consult a physician before taking this combination	67	20	13	0	10	90	10	100	0	0	. . .	. . .	. . .	0

^*Averages taken from 10 English-language Web sites with the exception of childhood asthma, for which 9 were used. Ellipses indicate not applicable because the topic was not covered.

Conflicting Information

Overall, just over half of English-language Web site reviews revealed 1 or more conflicts in the information provided (Table 4; Spanish reviewers noted no conflicts). Conflicts involved treatment (present in 35% of reviews), diagnosis (13%), definitions (7%), adverse effects (5%), etiology and risk factors (5%), and incidence and prevalence (4%). As an example, a childhood asthma site stated at one point that inhaled steroids do not stunt growth and later stated that inhaled steroids do stunt growth. Materials on depression were the most likely to have conflicts on treatment, whereas breast cancer materials were the most likely to contain conflicts on diagnosis (Table 4, P<.001).

Table 4.

Presence of Conflicting Information on English-Language Web Sites

		% of Reviews With Any Conflicting Information by Category
	% of Reviews Noting Any Conflict	Treatment	Diagnosis	Definition	Adverse Effects	Etiology and Risk Factors	Incidence and Prevalence
All medical conditions	53	35	13	7	5	5	4
Breast cancer	43	20*	27†	0	0	3	3
Childhood asthma	52	45	0*	3	10	7	3
Depression	73	50†	17	7	7	10	10
Obesity	43	27	7	17	3	0	0

^*Significantly fewer reviews for this condition noted conflicts of this type than for the mean condition (P<.05).

^†Significantly more reviews for this condition noted conflicts of this type than for the mean condition (P<.05).

Reading Grade Level

For English-language Web sites, the average reading level was collegiate (mean [SD] grade, 13.2 [2.1]) and ranged from 10th grade to graduate school level (Figure 2). For the Spanish-language Web sites, the average reading level was at 10th grade (mean [SD] grade, 9.9 [2.5]) and ranged from grades 7 to 13 (Figure 2). The mean grade reading level for the English-language Web sites was significantly higher than for Spanish-language Web sites (P<.003).

Figure 2. Reading Grade Levels for Selected Web Sites by Language.

Reading grade levels were determined using the Fry Readability Graph method.⁴⁸–⁵⁰

COMMENT

To our knowledge, this is the first study to examine English- and Spanish-language health information on the Internet across multiple conditions. We found that search engines are only moderately efficient in locating information on a particular health topic. More than half of consumers who use the Internet report that they spend about a half hour looking for health information, so efficiency is an important aspect of performance (Carolyn Gratzer, Cyber Dialogue, oral communication, October 13, 2000). Overall, 1 in 5 links identified by 10 English-language and 1 in 8 links from 4 Spanish-language search engines led to a Web page with relevant content.

We examined 2 dimensions of Web site quality: whether key consumer questions were covered and whether the information was accurate. Although we found thousands of pages of material related to key questions, there were substantial gaps in the availability of key information. Only half of the topics that the expert panels thought were important for consumers were covered more than minimally. This deficiency was particularly striking across Spanish-language sites, where more than half of the condition-related topics were not addressed.

Our results suggest that consumers using the Internet may have a difficult time finding complete and accurate information on a health problem. If people are relying on the Internet to make treatment decisions, including whether to seek care, deficiencies in information could negatively influence consumer decisions. For example, less than half of the Spanish-language materials explained that mastectomy and lumpectomy plus radiation are equivalent treatments for early-stage breast cancer.

The reading level of most Web-based material is quite high. All of the English-language sites had material that required at least a 10th-grade reading level, and more than half of the sites presented material at the college level. Although 1 Spanish-language site presented material at the elementary school level, all others required at least a ninth-grade reading level. According to the 1992 National Adult Literacy Survey, 92 million adults in the United States—almost 48% of the population—and more than 75% of current welfare recipients have low or very low reading skills.⁵¹ Thus, even if wider access to computer technologies narrowed the digital divide, the online health information currently available would be difficult for many people to understand.

This study has some important limitations. First, the Internet is a moving target, and we were able to take only a snapshot of its performance. Changes in content over time are not represented. However, without dedicated attention, it seems unlikely that the variability in performance, gaps in availability of information, and high reading levels will change dramatically. Second, we looked at a small set of search engines, Web sites, and conditions, and hence cannot draw more general conclusions. However, because we included the most popular search engines and Web sites, the results are likely to reflect common experiences. Their variability in performance suggests that the likelihood of finding the information one needs, on the topic of one’s choice, will depend on where one starts. Third, we studied the performance of search engines using very simple search terms. Had more sophisticated search strategies been used, our findings might have been different. Fourth, our research was not a natural experiment (eg, using actual consumers to search for information and testing their knowledge after such a search), so we cannot draw conclusions about what people actually encounter when they search for information, or about how well they are able to interpret the information they find. Fifth, the necessary inclusion of medical terms in analyzed text may be partially responsible for high estimated grade reading levels, although we assessed Flesch-Kincaid scores⁵² in the same passages with and without the medical terminology included, and when medical terminology was removed, the grade level declined by only 0.3 grade levels on average (range, 0.1–0.8). Sixth, because the Internet and many Web sites make available a large volume of material, it is possible that our searchers missed information that was available on a site. For that reason, we had 2 searchers look for information on each site, and they found different material. But the conclusions that reviewers reached about sites were quite consistent, even when the retrieved material they evaluated was different. Furthermore, our searchers were skilled, trained for the task, and devoted more time to finding information than people report spending on average. Thus, if our searchers could not find the information, probably most consumers also would have difficulty doing so.

Our results suggest several ways to make Web-based information more useful. First, variation among search engines suggests that it is possible to improve search efficiency, perhaps by improving the methods for indexing Web pages. Second, the lack of critical information for each of the 4 conditions suggests that Web site developers should focus on providing more complete information. Third, Web site developers need to ensure that the information is accurate and free from conflict. Although accuracy levels were generally high, the presence of conflicting information makes it possible that people will be more confused than enlightened. Fourth, some mechanism for routinely rating Web sites for coverage and accuracy may be useful. Comprehensive assessments of the type conducted for this study are highly labor intensive, but simpler methods also may be effective. Fifth, information on the Web needs to be made more readable if the Internet is to serve as a “leveler” across different socioeconomic backgrounds.

The Internet has the potential to be a powerful resource for meeting some of the public’s health information needs. Ideally, consumers would be able to learn much of what they need to know from high-quality Web sites, so that the limited time they have with their physicians could be used more efficiently. However, this requires that Web sites present well-organized and accurate information in a way that is understandable. Research is needed on how the public’s use of the Internet facilitates, complements, or complicates patient-physician communication and on how patients and health professionals can make better use of this resource.