Kathleen M. Griffiths, PhD (visiting fellow) & Helen Christensen, PhD (senior fellow);Centre for Mental Health Research, The Australian National University

Abstract: 250 words

Text (excl. refs, tables, insert table prompts, Key message): 2478 words

Correspondence: Dr Kathleen Griffiths
                           Centre for Mental Health Research
                           The Australian National University
                           Canberra 0200
                           Australia

e-mail: kathyg{at}apex.net.au
ph: +61 2 6279 9723

ABSTRACT

Objective: To evaluate the quality of web site information on the treatment of depression, identify potential indicators of content quality, and establish if accountability criteria are indicators of content quality.

Design: Audit of 21 frequently accessed depression web sites.

Main outcome measures: (i) Quality – concordance with evidence-based depression guidelines (guideline score), appropriateness of other relevant site information (issues score), subjective rating of site quality (global score); (ii) Accountability – conformity with core disclosure standards (Silberg score), quality of evidence cited in support of conclusions (level of evidence score); and (iii) Site characteristics.

Results: Although the sites contained useful information, their overall quality was poor, the mean guideline, issues and global scores being only 4.7 out of 43 (confidence interval (CI) 3.0 to 6.4), 9.8 out of 17 (CI 8.7 to 10.8 ) and 3 out of 10 (CI 2.2 to 3.7) respectively. Sites typically did not cite scientific evidence in support of their conclusions. The guideline score correlated with the two other quality of content measures. However, none of the content measures correlated with the Silberg accountability score. Content quality was superior for sites owned by organizations and sites with an editorial board.

Conclusion: There is a need for better evidence-based depression information on the web, to reconsider the role of core accountability standards as indicators of site quality, and to further evaluate and develop simple valid indicators of the web content quality. Ownership by an organization and the involvement of a professional editorial board are potentially useful indicators warranting further investigation.

The web represents an unprecedented opportunity to provide high quality, accessible health care information to consumers and health providers. However, in the absence of editorial controls, the information may be of low quality and potentially harmful¹.

In an influential paper, Silberg et al (1997)² proposed that accountability standards (disclosure of authorship, ownership and currency of information) may be useful indicators of the quality of web health information. These accountability criteria have been widely assumed to reflect web site quality.^3-5 However, to date the validity of these standards as indicators of the quality of web site content have not been investigated. Moreover, there have been few systematic studies of the actual quality of the content of health information on the web,^6-9 and these studies have typically used textbook summaries⁵or author opinion⁸ as the gold standard for assessing content quality rather than meta-analyses of the available evidence.¹⁰ Finally, no published studies have systematically evaluated the quality of mental health web sites although mental disorders are a common cause of disability and the WHO has predicted that depression will be the second largest cause of disability within 20 years.¹¹ Since only a minority of people with depressive disorders receive treatment¹², web sites provide a potentially useful tool for improving help seeking among depressed people.

The current study involved a survey of frequently accessed depression sites on the internet and was designed to evaluate: (i) the quality of web site information on the treatment of depression (including comparison with evidence-based guidelines and meta-analyses); and (ii) the relationship between content quality and accountability indicators and site characteristics.

METHODS

Selection of sites

We identified 21 popular depression sites using the search engines DirectHit, which identifies frequently accessed sites; and MetaCrawler, which integrates the results from nine other well known search engines. Searches were conducted in March 1999 using the key word ‘depression’. Site material was identified and printed out by systematically following all internal links. External links, news sections (typically internally contradictory), sections relating to bipolar disorder and schizophrenia and book reviews were excluded.

Site assessment

Site characteristics, content and accountability were independently evaluated by two rates. Disagreements were resolved soon after rating by means of discussion and reference to site material.

Characteristics of the site

Each site was rated as to its purpose, scope, ownership, country of origin, and involvement of a drug company, professional editorial board, and health professional. Sites were also rated according to whether they promoted products/services and whether they contained a disclaimer or qualifier regarding information provided.

Quality of content

Guideline score. Concordance between site information and best practice was assessed using a 43-item rating scale based on the evidence-based AHCPR clinical practice guidelines for treating depression.¹³ The scale covered the use of medications, psychotherapy, combined medication and psychotherapy and electroconvulsive therapy (ECT). Topics included effectiveness, indications, selection within a treatment type, failure to respond, and frequency of visits. A guideline score was computed for each site by cumulating the number of items on the scale for which site information was concordant with the guidelines. In addition, a core guideline score (out of 5) was calculated from a subset of key items relating to indications for and effectiveness of the four major treatment types. The core guidelines are summarized in table 1.

1. Antidepressant medication is an effective treatment for major depressive disorder.

2. Antidepressant medication is the first line of treatment in the following circumstances: moderate to severe depression; psychotic/melancholic atypical symptoms (overeating, oversleeping, weight gain); patient request; psychotherapy unavailable; previous response to medication). [Site must have identified at least one of the above qualifiers to be rated as in agreement with guideline].

3. Psychotherapy can be an effective first line treatment for mild to moderate depression.

4. Combined initial medication and psychotherapy is reasonable in only some circumstances (eg, chronic prior course of illness/poor inter-episode recovery, psychotherapy alone or medication alone only partly effective; history of psychosocial problems both during and outside of depressive episodes; history of poor treatment adherence).

ECT may be effective in certain cases of severe depression.

---

Global score. Each rater provided a subjective judgement of the overall quality of a site (score out of 10). An average score was then computed for each site. There was a significant correlation between the global ratings of the two judges (r=0.69, p=.001) and the mean ratings for the two judges did not differ significantly, suggesting acceptable inter-judge agreement despite the unstructured and subjective nature of the task.

Interventions recommended. A range of interventions were rated according to whether they were (i) mentioned; (ii) said or implied to be effective/useful or recommended as a first line, second line or adjunct treatment for all or some groups; and (iii) said to be ineffective or not recommended. Interventions denoted effective but explicitly not recommended were coded as ‘not recommended’.

Sources of help recommended. Potential sources of help for depression were rated as recommended, not recommended or not mentioned.

Accountability

Silberg score. Sites were rated on a 9 point scale according to the Silberg et al criteria of authorship (whether authors and their affiliations and credentials were clearly identified), attribution (whether sources and references were mentioned), disclosure (whether ownership of the site and sponsorship was disclosed) and currency (whether the site has been modified in the past month and year and whether the date the site was created or modified was specified).

Level of evidence score. The stated level of evidence associated with each intervention was recorded using a 5 point scale adapted from a previously published hierarchy of evidence scale¹⁴ (see table 2). Evidence ratings were based only on information explicitly provided by the site and not on the raters’ knowledge of the cited study or relevant literature.

Analyses

Site quality and accountability were assessed as a function of site characteristic using Mann-Whitney tests, Kruskall-Wallis analyses followed by Mann-Whitney tests, or chi-square tests. Intercorrelations between variables were computed using Pearson correlation tests.

Results

Site characteristics

Nineteen sites were US-based, one was European and the origin of the remaining site was unknown. The principal purpose of the sites was to provide information/educational material (10), links (4), a consumer forum (1) or information in combination with either links or consumer forum or both (6). Details of other site characteristics are presented in table 3.

Table 3 Guideline quality as a function of type of site. Values are mean scores (standard deviation).
 

*p£ .05 #p<.05 for one judge

^a None of the identified sites was owned solely by a research centre, university, medical/health professional organisation or hospital although one site was owned by a commercial organization with input from university/hospital personnel. Another site was jointly owned by a commercial and a non-profit organization. Both were included in the commercial category.

^bOrganisation established for profit making such as drug or media company, web company.

^cHealth professional including general practitioner, psychiatrist, psychologist, counsellor.

^dSite established by a consumer, carer/friend or consumer organization.

^e Drug company owned or sponsored.

^fAt least two thirds of site devoted to depression.

^gHealth professionals involved in writing material or on editorial board.

Quality of content

The mean guideline, issues and global scores were 4.7 out of 43 (confidence interval (CI) 3.0 to 6.4), 9.8 out of 17 (CI 8.7 to 10.8 ) and 3 out of 10 (CI 2.2 to 3.7) respectively, indicating little concordance with guideline recommendations, inadequate consideration of management/treatment issues and generally low overall ratings.

In part, the low guideline score reflected poor coverage, with, on average, material relevant to more than two thirds of the guideline items absent. However, the poor quality of the sites was not attributable to inadequate coverage alone. In the case of the 5 core guidelines, the majority of sites (average 58%) contradicted or provided material inconsistent with the guidelines.

Sites usually recognized that antidepressants and psychotherapy are effective. However, the specified indications for these treatments were often inaccurate. For example, many sites emphasised one form of these treatments over the other regardless of the severity of the depression and other important factors; almost half of the sites recommended combined antidepressant and psychotherapy as a first line treatment when this is not recommended by the AHCPR guidelines and sites were often internally inconsistent, especially when material was derived from more than one author or source.

Between 40 and 60% of the sites failed to discuss contraindications for medication, failed to recognise individual differences in the effects of antidepressants or did not identify the importance of switching medication as required. Very few sites acknowledged that chronic and subsequent episodes of depression may require a different approach, that the management and treatment of depression in young people may differ from that for adults or that the availability of treatment may be a factor to consider in selecting treatments. However, side effects and the long term nature of antidepressant treatments were discussed by a substantial majority of the sites and the percentage of sites reporting side effects of herbal or dietary supplements was consistent with the overall level of reporting on these types of treatments.

All sites communicated the message that depression can be treated, most indicated that the depression should be treated and only one site failed to mention the risk of suicide in depression. Although effective treatments such as ‘SSRIs’ and ‘TCAs’, ‘psychotherapy’ and ‘cognitive therapy’ were mentioned by the majority of sites, a number of important evidence-based conventional treatments for depression were mentioned by less than half the sites (eg, newer antidepressants, interpersonal therapy, behaviour therapy, cognitive behaviour therapy). St John’s Wort was recommended by only 6 (less than 30%) of the sites despite level 1 evidence suggesting it is effective for mild depression.

Sources of help. All sites promoted consultation with a health professional for diagnosis and/or treatment, and most provided a list of contact organisations for further information or assistance. All recommended a doctor as a source of help (see table 4). Psychiatrists, psychopharmacologists, psychologists and psychotherapists are professionals with expertise in delivering known effective treatments for depression: antidepressants and psychotherapy. However, six (29%) of the sites did not recommend/mention any of these professionals as potential sources of help. Sites were as likely to recommend web sites, family members, the clergy or friends as they were to recommend psychiatrists.

Table 4. Number (percentage) of sites recommending different sources of help.
 

Mentioned but not recommended by an additional ^a2 (9.5%), ^b3 (14.3%), ^c8 (38.1%) of the sites.

Accountability

The mean Silberg score was 5.4 out of 9 (CI 4.6 to 6.3). At least half of the sites clearly specified the author of the web content (13 sites) and their credentials (11 sites) and affiliations (11 sites). Forty-three percent (9) of the sites mentioned at least some sources and references on the site (although such information was typically not comprehensive). All but one site disclosed an owner of the site and three mentioned sponsors. The majority of sites indicated when the site had been created or modified and had modified the site in the past year. Over 40% (9) of the sites had been modified in the past month.

As many as 53 different interventions were mentioned by at least one site, but sites typically did not provide supporting scientific information or refer in general terms to the level of evidence available to support their recommendations. Since most sites mentioned antidepressants and psychotherapy and these therapies are supported by level 1 evidence, data for these interventions were analysed further (taking the highest level of evidence across generic and individual forms of each treatment type). Only 25% (5) of the sites mentioned any scientific evidence in support of the use of antidepressants and only one of these referred to level 1 evidence. Similarly, only 17% (3) of the sites that recommended/did not recommend psychotherapy cited scientific evidence in support of their conclusions and only one site cited level 1 evidence.

Association between quality of content, accountability and site characteristics

There was a significant correlation between the guideline score and the other two quality of content measures (guideline/global quality: r=.53, p<.05; guideline/issues: r=.74, p<.01). However, none of the measures of quality of content correlated significantly with the Silberg accountability score (r: –0.5 to 0.21). Of the sites offering recommendations concerning psychotherapy, sites quoting scientific evidence were more likely to achieve an above median guideline score (p=.034) and showed a tendency to achieve above median issues scores (p=.058). There was no comparable significant relationship for antidepressants.

Content quality (guideline and issues scores, rating of one judge) was superior for sites owned by organizations and sites with an editorial board (see table 3). Only sites owned by organizations reported scientific evidence in support of their endorsement of antidepressants and psychotherapy. Compared with their counterparts, sites owned by organizations and sites involving drug companies were significantly more likely to cite scientific evidence in support of antidepressants (p<.05). However, there was no significant association between the total Silberg accountability score and site characteristics. In fact, analyses of individual Silberg items, revealed that sites owned by organizations and sites involving drug companies were less likely than their counterparts to indicate the author's identity (p<.05), affiliation (p<.05), and credentials (p<.05).

There was no effect of site type on the total number of sources of help endorsed or whether priority professional groups (see above) were recommended but sites involving health professionals were more likely than other sites to endorse mental health specialists/professionals as a source of help (psychiatrist p<.05; psychologist, p<.05; combined p=.002).

Discussion and Conclusions

The finding that web based depression treatment information is of poor quality reinforces concerns arising from the results of other studies which have found inadequate quality^6-9 or poor coverage of important health issues on the web¹⁵. There is a need to improve the accuracy and coverage of web-information concerning: (i) the relative effectiveness of different treatments; (ii) the main indications for particular treatments; and (iii) important management issues such as duration of treatment, reviewing and changing treatments, and the relevance of professional expertise and patient preferences. Sites also need to communicate that tricyclics are ineffective for adolescents and that antidepressants may not be the first line of treatment for this group¹⁶.

The current findings raise questions about the usefulness of specific Silberg et al accountability criteria as indicators of quality and suggest that further investigation of indicators of quality is warranted. Particular site characteristics (such as ownership by an organization or existence of a professional editorial board) are likely to prove more useful indicators of content quality than disclosure of information per se. In the current study, the number of different types of interventions mentioned was also a predictor of site quality. It may be of interest to ascertain if this finding generalises to other fields of health. The citation of scientific evidence in support of treatment recommendations also warrants further investigation as a potentially useful indicator of site quality.

It is possible that the inadequacies documented in this paper are not restricted to web sites but may in fact reflect the beliefs and level of knowledge of many health and mental health professionals. Interestingly, McLung et al⁷ have reported that even medical teaching centres disseminated inadequate reviews on the web. It is also possible that the AHCPR guidelines are themselves outdated or inadequate. In fact, a review of more recent evidence concluded that the major AHCPR conclusions are still applicable and that when rigorously implemented, the guidelines result in improved outcomes compared with usual care¹⁷. The guidelines have been criticised for their failure to recommend psychotherapy as a first line treatment for severe depression¹⁸. Although there is some evidence to support this criticism,¹⁹ few studies have directly compared the efficacy of different treatments for severe depression and the findings have been inconsistent. By contrast, a large number of randomised controlled trials have demonstrated the efficacy of antidepressants in treating severe depression.

Despite low content quality ratings, many sites did contain important and potentially useful information. It is even possible that a formal evaluation might demonstrate improved outcomes for those who visit such sites. Silberg et al have referred to the importance of distinguishing the flowers from the weeds on the superhighway. However, a single site, whether owned by a consumer or a health professional, may grow both flowers and weeds. The real challenge is to devise strategies which selectively eliminate the weeds but leave the flowers to bloom.

ACKNOWLEDGEMENTS

This work was supported by grant 973302 from the National Health and Medical Research Council. We would like to thank Jo Medway for her contribution to data collection and data organisation and for her useful comments on the manuscript.

REFERENCES

Bower H. Internet sees growth of unverified health claims. BMJ 1996; 313: 381.

² Silberg WM, Lundberg GD, Musacchio RA. Assessing, controlling and assuring the quality of medical information on the internet.JAMA 1997; 277: 1244-5.

³ O’Mahony B. Irish health care web sites: a review. Ir Med J 1999; 92: 334-7.

⁴ Jadad AR, Gagliardi A. Rating health information on the internet: navigating to knowledge or to Babel? JAMA 1998; 279:611-4.

⁵Shon J, Musen MA. The low availability of metadata elements for evaluating the quality of medical information on the world wide web.Proc AMIA Symp 1999; (1-2): 945-9.

⁶Impicciatore P, Pandolfini C, Casella N, Bonati M. Reliability of health information for the public on the World Wide Web: systematic survey of advice on managing fever in children at home. BMJ 1997; 314: 1875-9.

⁷ McClung HJ, Murray RD, Heitlinger LA. The internet as a source for current patient information. BMJ 1998; 101: 1-4.

⁸ Soot LC, Moneta GL, Edwards JM. Vascular surgery and the internet: a poor source of patient-oriented information.J Vasc Surg 1999; 30: 84-91.

⁹ Davison K. The quality of dietary information on the world wide web. Clin Perform Qual Health Care 1997; 5: 64-6.

¹⁰ Wyatt JC. Measuring quality and impact of the world wide web. BMJ 1997; 314: 1879-81.

¹¹ Murray CJL, Lopez AD (eds). The global burden of disease: a comprehensive assessment of mortality and disability, injuries and risk factors in 1990 and projected to 2020. Massachusetts: Cambridge, 1996.

¹² Kessler, RC. Some considerations in making resource allocation decisions for the treatment of psychiatric disorders. In: Andrews G, Henderson S, editors. Unmet need in psychiatry: Problems, resources, responses. Cambridge: Cambridge University Press, 2000. p59-84.

¹³ Depression Guideline Panel. Depression in primary care: vol. 2. treatment of major depression, clinical practice guidelines, no 5. AHCPR Publication No. 93-0551. Maryland: U.S. Dept. Health and Human Services: Maryland, 1993.

¹⁴ National Health and Medical Research Council. Guidelines for the development and implementation of clinical practice guidelines. Canberra: NHMRC, 1995.

¹⁵ Sandvik H. Health information and interaction on the internet: A survey of female urinary incontinence. BMJ 1999; 319: 29-32.

¹⁶ National Health and Medical Research Council. Clinical practice guidelines: Depression in young people. Canberra: NHMRC, 1997.

¹⁷ Schulberg HC, Katon W, Simon GE, Rush AJ. Treating major depression in primary care practice: an update of the Agency for Health Care Policy and Research Practice Guidelines. Archives Gen Psychiatry 1998; 55:1121-7.

¹⁸ Persons JB, Thase ME, Crits-Cristoph P. The role of psychotherapy in the treatment of depression: review of two practice guidelines. Arch Gen Psychiatry 1996; 53: 283-90.

¹⁹ Schulberg, HC, Pilkonis PA & Houck P. The severity of major depression and choice of treatment in primary care practice.J Consult Clin Psychol 1998; 66:932-8.

KEY MESSAGES

• Depression is a major source of disability in the community. Web sites offer an unprecedented opportunity to disseminate information to the public about effective depression treatments. However, little is known about the quality of existing depression sites or about indicators of a good health web site.

• An audit of 21 popular sites revealed that the general quality of web information on the treatment of depression is poor.

• Disclosure of information about a site (Silberg et al accountability) is not in itself an indicator of content quality. However, sites with an editorial board and sites owned by organisations may produce higher quality sites than their counterparts.