Abstract
Objective
To evaluate the clinical, methodological, and reporting aspects of systematic reviews and meta-analyses on the treatment of asthma and to compare those published by the Cochrane Collaboration with those published in paper based journals.
Design
Analysis of studies identified from Medline, CINAHL, HealthSTAR, EMBASE, Cochrane Library, personal collections, and reference lists.
Studies
Articles describing a systematic review or a meta-analysis of the treatment of asthma that were published as a full report, in any language or format, in a peer reviewed journal or the Cochrane Library.
Main outcome measures
General characteristics of studies reviewed and methodological characteristics (sources of articles; language restrictions; format, design, and publication status of studies included; type of data synthesis; and methodological quality).
Results
50 systematic reviews and meta-analyses were included. More than half were published in the past two years. Twelve reviews were published in the Cochrane Library and 38 were published in 22 peer reviewed journals. Forced expiratory volume in one second was the most frequently used outcome, but few reviews evaluated the effect of treatment on costs or patient preferences. Forty reviews were judged to have serious or extensive flaws. All six reviews associated with industry were in this group. Seven of the 10 most rigorous reviews were published in the Cochrane Library.
Conclusions
Most reviews published in peer reviewed journals or funded by industry have serious methodological flaws that limit their value to guide decisions. Cochrane reviews are more rigorous and better reported than those published in peer reviewed journals.
Introduction
One of the most important challenges for anyone treating patients with asthma is the need to cope with vast amounts of information about an ever increasing number of new devices and drugs. The abundance of information often makes it difficult to establish the most appropriate drug or regimen to use in a given clinical situation.1
During the past two decades, interest has risen in the use of systematic reviews as tools to help decision makers cope with information overload. These reviews, in their ideal form, include an explicit description of how they were conducted and incorporate strategies to minimise bias and maximise precision.2,3 These reviews can also include meta-analysis (the statistical combination of the results of several independent studies to produce a single estimate of the effect of a particular intervention or healthcare situation).4 However, most systematic reviews and meta-analyses published in peer reviewed journals have methodological deficiencies that may limit their validity.5–12
Recently, important initiatives have emerged to improve the methodological quality of systematic reviews and meta-analyses. Perhaps the most prominent is the Cochrane Collaboration, a large international group11,13 that seems to be producing reviews that are more rigorous and better reported than those published in peer reviewed journals.9,14 We evaluated the quality of published systematic reviews and meta-analyses on the treatment of asthma and compared the characteristics of Cochrane reviews with those in peer reviewed journals.
Methods
Inclusion criteria
To be included, a report had to be described as a systematic review or a meta-analysis of the treatment of asthma and to be published in full, in any language or format, in a journal or the Cochrane Library.
Literature search and study selection
We identified eligible studies by a search up to July 1998 in Medline (from 1966), CINAHL (from 1982), HealthSTAR (from 1975), and EMBASE (from 1984) using a refined search strategy (see BMJ's website for details).9 This was complemented by a search of the Cochrane Library (issue 3, 1998) with the term “asthma” and searches of personal collections and the reference lists of eligible studies.
Independently, two of us (LB and ARJ) reviewed the citations identified by the search strategy to judge their eligibility. The final decision on inclusion in the review was made by consensus.
Data extraction
Three trained reviewers (MF, LB, RS) extracted data independently using unmasked reports.15 The final set was agreed by consensus, and differences were resolved by a fourth reviewer (ARJ).
Data were extracted on the general characteristics of the studies (authors, sources of funding, and source, year, place, and language of publication); clinical issues (population, intervention, condition, and outcomes reviewed), methodological characteristics (language restrictions; number, format, design, and publication status of the studies included; data synthesis; heterogeneity testing; and methodological quality); and conclusions. Methodological quality was assessed with Oxman and Guyatt's index, a validated tool that scores reviews on a seven point scale.8,16 Reports were regarded as having serious or extensive flaws if they received a score of 1 to 3, and as having minimal or minor flaws if they received scores from 4 to 7.
Data analysis
Reviews were divided according to publication in the Cochrane Library or journals and to the presence or absence of association with industry. The Wilcoxon rank-sum test (two sided) was used to compare the number of authors, included studies, and sources and the overall quality scores. Fisher's exact test (two sided) was used to compare the responses to the first nine items in Oxman and Guyatt's index and also the proportion of reviews that described the primary outcomes, selection criteria, and heterogeneity testing and the number of reviews with language restrictions between the Cochrane Library and journals. Values of P<0.05 were regarded as significant.
Results
General characteristics
We identified 50 eligible reviews (see BMJ's website for full list). The first review identified was published in 1988. Twenty nine (58%) of the reviews were published in 1997 and 1998. Twelve reviews were published in the Cochrane Library, and the remaining 38 were published in 22 peer reviewed journals. Only one review appeared in both groups. None of the reviews published in journals represented updated versions of earlier work. The journal with the highest number of reviews was Annals of Pharmacotherapy (seven) followed by the Journal of Allergy and Clinical Immunology and Respiratory Medicine (four each). Table 1 gives other characteristics of the reviews.
Table 1.
Characteristics of 50 reviews of treatment of asthma
| No of reviews | |
|---|---|
| Sources of funding* | |
| Government agencies | 15 |
| Industry | 6 |
| Charities | 2 |
| Consumer groups | 1 |
| Professional organisations | 1 |
| Not reported | 27 |
| Country of corresponding author | |
| United States | 17 |
| Canada | 10 |
| United Kingdom | 8 |
| Australia | 4 |
| Netherlands | 3 |
| Belgium | 2 |
| Germany | 2 |
| Others | 3 |
| Not reported | 1 |
| Sources of studies used* | |
| Medline | 33 |
| Reference lists | 27 |
| Corresponding authors | 14 |
| Journal handsearching | 12 |
| Interventions evaluated | |
| Steroids (inhaled or systemic) | 16 |
| β agonists | 16 |
| Xanthines | 6 |
| Antileukotrienes | 5 |
| Complementary therapies | 3 |
| Immunotherapy | 2 |
| Placebo | 27 |
| Other | 24 |
| Outcomes evaluated* | |
| Forced expiratory volume | 41 |
| Peak expiratory flow | 39 |
| Rescue treatment | 22 |
| Hospital admissions | 13 |
| Quality of life | 13 |
| Cost | 4 |
| Patient preferences | 3 |
| Data synthesis | |
| Meta-analysis | 27 |
| Qualitative review | 15 |
| Unclear | 8 |
More than one category was possible.
Methodological characteristics
The number of studies included in the reviews varied from 2 to 97. Most of the reviews (37) included randomised controlled trials, but three included non-randomised trials with contemporaneous controls and one included case series. In 14 reviews (including some of those mentioned in the previous sentence) the exact type of studies was either unclear or not reported. Seven of the 50 reviews included unpublished articles and nine included articles available only in abstract form. None of the reviews published in peer reviewed journals had been updated.
Methodological quality
Forty reviews obtained quality scores of 3 or less, which is compatible with serious or extensive flaws. All six reviews associated with industry were in this group (table 2). None of these reviews described efforts to reduce selection bias; one described the search strategy, one described the methods used to assess the quality of the studies, and three reported the methods used to combine data across studies (table 2). All but one17 of these studies had results and conclusions that favoured the interventions related to the companies sponsoring the reviews. The number of reviews associated with industry was too small to warrant meaningful statistical comparisons with other reviews.
Table 2.
Methodological quality of systematic reviews and meta-analyses of the treatment of asthma. Values are number of affirmative answers/number of reviews relevant to each question
| Methodological quality (Oxman and Guyatt index16) | All | Cochrane reviews | Peer reviewed journals | P value* | Associated with industry† |
|---|---|---|---|---|---|
| Were the search methods used to find evidence on the primary question stated? | 33/50 | 12/12 | 21/38 | <0.005 | 1/6 |
| Was the search for evidence reasonably comprehensive? | 26/33 | 12/12 | 14/21 | 0.032 | 1/1 |
| Were the criteria used for deciding which studies to include reported? | 30/50 | 12/12 | 18/38 | <0.005 | 4/6 |
| Was bias in the selection of studies avoided? | 15/30 | 9/12 | 5/18 | 0.024 | 0/4 |
| Were the criteria used for assessing the validity of the included studies reported? | 14/50 | 11/12 | 3/38 | <0.005 | 1/6 |
| Was the validity of all studies referred to in the text assessed using appropriate criteria? | 12/14 | 9/11 | 3/3 | 1.000 | 1/1 |
| Were the methods used to combine the findings of the relevant studies reported? | 26/50 | 12/12 | 14/38 | <0.005 | 3/6 |
| Were the findings of the relevant studies combined appropriately? | 24/26 | 11/12 | 11/14 | 0.598 | 4/6 |
| Were the conclusions made by the author(s) supported by the data reported? | 25/50 | 11/12 | 14/14 | <0.005 | 3/6 |
| Overall quality (median score)‡ | 3 | 6 | 2 | <0.005 § | 2 |
Fisher's exact test (two tailed) for Cochrane v peer reviewed journals.
These reviews are a subset of the 38 journal reviews.
The score was obtained by analysing the responses to each of the nine questions, using a standardised set of instructions provided by the developers of the index.8
Wilcoxon rank-sum test (two sided) for Cochrane v peer reviewed journals.
Analysis of individual components of Oxman and Guyatt's index showed that 66% of the reviews stated the search methods used to find the evidence, 52% included a reasonably comprehensive search for evidence, 56% reported the inclusion criteria, 30% included measures to avoid selection bias, 28% reported the criteria used to assess the validity of the included studies, 24% used appropriate criteria for validity assessment, 52% reported the methods used to synthesise findings, and 50% had conclusions supported by the data or the analysis reported in the review (table 2). In addition, 20 reviews (40%) included heterogeneity testing, 16 (32%) stated the primary outcome of interest to the reviewers, and eight (16%) included assessment of publication bias. We found no significant difference in the proportion of reviews with favourable results between those with minimal or minor flaws and those with serious or extensive flaws (1/10 v 11/40, P=0.416).
Cochrane reviews versus reviews in peer reviewed journals
Of the 10 reviews with scores of 4 or more, seven were published in the Cochrane Library. Cochrane reviews had higher overall quality scores than those published in peer reviewed journals (median 6 v 2, P<0.005). Compared with reviews published in journals, more Cochrane reviews had adequate reporting of the search strategy, had comprehensive search strategies, described the inclusion criteria, made efforts to avoid bias during the selection of the studies, reported the criteria for assessing the quality of the studies, reported the methods used to combine study findings, and combined the findings of the relevant studies appropriately (table 2). Fewer Cochrane reviews had language restrictions (0/12 v 21/38, P<0.005) and more included heterogeneity testing (9 v 11, P=0.007).
There were no significant differences between the groups in the number of authors, the number of studies reviewed, assessment of publication bias, and identification of primary outcomes.
Findings of reviews
Twelve reviews identified an intervention as better than the control treatment. None of the reviews reported patient preferences. The box gives the conclusions of the reviews with minimal flaws.
Findings of studies with minimal flaws
Peer reviewed journals
Use of oral and intravenous steroids early in the treatment of asthma exacerbations reduces hospital admissions in adults and children; steroids are effective in preventing relapse in outpatient management of exacerbations; and oral and intravenous steroids seem to have equivalent effects on pulmonary function during acute exacerbations18
Addition of ipratropium bromide to β agonist significantly improves the predicted forced expiratory volume in one second but gives no clinical improvement19
Bronchodilator delivery through a meter dosed inhaler or wet nebuliser is equivalent in acute treatment of asthma20
Cochrane Library
Limited asthma education is unlikely to improve health outcomes in adults with asthma21
Addition of a single inhalation of anticholinergics to a β agonist regimen may improve lung function but has no notable effect on hospital admissions. Multiple doses may avoid hospital admission in 1 of 11 treated patients22
A short course of corticosteroids (oral or intramuscular) after an acute exacerbation of asthma (of variable and poorly reported severity at baseline) reduces the number of relapses and decreases use of β agonist without apparent increase in adverse effects23
There is no indication for routine use of methotrexate in patients receiving long term treatment with oral steroids24
Prophylaxis with single doses of nedocromil sodium1-8 mg inhibits the severity and duration of exercise induced bronchoconstriction. Benefits seem greatest in patients with more severe symptoms25
Allergen specific immunotherapy reduces the symptoms and drug requirements but has no consistent effect on lung function. The effects are greater in patients with allergen specific bronchial hyperreactivity than in those with non-specific hyperreactivity26
Chemical and physical measures to reduce exposure to mites or their products at home seem to be ineffective27
Discussion
The number of systematic reviews and meta-analyses evaluating the treatment of asthma is increasing. Most of the reviews have clinical relevance but have methodological limitations that could have been avoided easily. Future reviews should include comprehensive and well reported search strategies, attempts to reduce bias during the selection of the studies, assessment of the validity of the studies, and clear description of the methods used to synthesise the data available. Few reviews address safety issues or the effect of treatment on quality of life, costs, or patient preferences.
This study also confirms that Cochrane reviews are more rigorous than reviews published in peer reviewed journals.9,14 Several factors may explain the differences. Cochrane reviews follow standardised instructions, are peer reviewed at several levels, and receive input from different groups at different stages.13 One limitation of our analyses is that the tool used for quality assessments was developed by Andrew Oxman, who led the development of the methodology of Cochrane reviews.
Perhaps peer reviewed journals could improve the quality of the systematic reviews they publish by providing authors and peer reviewers with clear reporting criteria, including those followed by Cochrane reviewers,28,29 by influencing the review process at a much earlier stage, and by promoting more frequent updates or correction of published material on the internet. Authors should be particularly careful when doing reviews in association with industry, as such reviews have been shown to have an increased risk of producing results in favour of the interventions promoted by the sponsors.12 This was the case for all but one of the reviews associated with industry in our study. The only review that did not favour an intervention was designed to evaluate the effects of vitamin C on asthma, not a new proprietary compound.17
Bridging the gap between methodological research and methodological practice will require unprecedented efforts by researchers, consumers, funders, journal editors, and peer reviewers. Efforts made by members of the Cochrane Collaboration are already contributing to filling the gap and should be emulated by peer reviewed journals.
What is already known on this topic
Systematic reviews and meta-analyses could help decision makers cope with information overload
The number of systematic reviews and meta-analyses evaluating asthma treatments is increasing
What this study adds
Most reviews of asthma treatment published in peer reviewed journals or funded by industry have serious methodological flaws that limit their usefulness
Cochrane reviews are more rigorous and better reported than those published in peer reviewed journals
The clinically relevant messages from the 10 most rigorous reviews are summarised
Supplementary Material
Acknowledgments
We thank Ann Murray for editorial assistance. The review was coordinated by Foresight Consultants, a division of Foresight Links Corporation.
Footnotes
Funding: Foresight Consultants was commissioned by AstraZeneca Research and Development Lund, Sweden, and AstraZeneca US to perform an independent review of this topic through an unrestricted contract. Representatives from AstraZeneca did not participate in the study selection or appraisal or in the analysis of the data. ARJ was supported in part by a National Health Research Scholar Award from Health Canada, Ottawa, Ontario.
Competing interests: ARJ is codirector of the Canadian Cochrane Network and Centre.
website extra: Details of the search strategy and all included studies are available on the BMJ's website www.bmj.com
References
- 1.O'Byrne PM. Clinical comparisons of inhaler systems—what are the important aspects. J Aer Med. 1995;8:S39–S47. doi: 10.1089/jam.1995.8.suppl_3.s-39. [DOI] [PubMed] [Google Scholar]
- 2.Mulrow CD. Rationale for systematic reviews. In: Chalmers I, Altman DG, editors. Systematic reviews. London: BMJ Publishing; 1995. pp. 1–8. [Google Scholar]
- 3.Oxman AD, Guyatt GH. Guidelines for reading literature reviews. Can Med Ass J. 1988;138:687–703. [PMC free article] [PubMed] [Google Scholar]
- 4.Lau J, Ioannidis JPA, Schmid CH. Quantitative synthesis in systematic reviews. Ann Intern Med. 1997;127:820–826. doi: 10.7326/0003-4819-127-9-199711010-00008. [DOI] [PubMed] [Google Scholar]
- 5.Sacks HS, Berrier J, Reitman D, Pagano D, Chalmers T. Meta-analyses of randomized control trials: an update of the quality and methodology. In: Bailar JC III, Mosteller F, editors. Medical uses of statistics. Boston: New England Medical Journal Publications; 1992. pp. 427–442. [Google Scholar]
- 6.Sacks HS, Reitman D, Pagano D, Kupelnick B. Meta-analysis: an update. Mount Sinai J Med. 1996;3-4:216–224. [PubMed] [Google Scholar]
- 7.Assendelft WJJ, Koes BW, Knipschild PG, Bouter LM. The relationship between methodological quality and conclusions in reviews of spinal manipulation. JAMA. 1995;274:1942–1948. [PubMed] [Google Scholar]
- 8.Jadad AR, McQuay HJ. Meta-analyses to evaluate analgesic interventions: a systematic qualitative review of their methodology. J Clin Epidemiol. 1996;49:235–243. doi: 10.1016/0895-4356(95)00062-3. [DOI] [PubMed] [Google Scholar]
- 9.Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA. 1998;280:278–280. doi: 10.1001/jama.280.3.278. [DOI] [PubMed] [Google Scholar]
- 10.Moher D, Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, et al. Does the poor quality of reports of randomized trials exaggerate estimates of intervention effectiveness reported in meta-analyses? Lancet. 1998;352:609–613. doi: 10.1016/S0140-6736(98)01085-X. [DOI] [PubMed] [Google Scholar]
- 11.Bero LA, Rennie D. Influences on the quality of published drug trials. Int J Technol Assess Health Care. 1996;12:209–237. doi: 10.1017/s0266462300009582. [DOI] [PubMed] [Google Scholar]
- 12.Barnes EC, Bero L. Why review articles on the health effects of passive smoking reach different conclusions. JAMA. 1998;279:1566–1570. doi: 10.1001/jama.279.19.1566. [DOI] [PubMed] [Google Scholar]
- 13.Jadad AR, Haynes RB. The Cochrane Collaboration—advances and challenges in improving evidence-based decision making. Med Decis Making. 1998;18:2–9. doi: 10.1177/0272989X9801800102. [DOI] [PubMed] [Google Scholar]
- 14.Egger M, Davey Smith G, Schneider M, Minder C. Bias in meta-analysis detected by a simple, graphical test. BMJ. 1997;315:629–634. doi: 10.1136/bmj.315.7109.629. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Berlin JA.for the University of Pennsylvania Meta-analysis Blinding Study Group. Does blinding of readers affect the results of meta-analyses? Lancet 1997350185–186. [DOI] [PubMed] [Google Scholar]
- 16.Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44:1271–1278. doi: 10.1016/0895-4356(91)90160-b. [DOI] [PubMed] [Google Scholar]
- 17.Bielory L, Gandhi R. Asthma and vitamin C. Ann Allergy. 1994;73:89–96. [PubMed] [Google Scholar]
- 18.Rowe BH, Keller JL, Oxman AD. Effectiveness of steroid therapy in acute exacerbations of asthma: a meta-analysis. Am J Emerg Med. 1992;10:301–310. doi: 10.1016/0735-6757(92)90007-k. [DOI] [PubMed] [Google Scholar]
- 19.Osmond MH, Klassen TP. Efficacy of ipratropium bromide in acute childhood asthma: a meta-analysis. Acad Emerg Med. 1995;2:651–656. doi: 10.1111/j.1553-2712.1995.tb03607.x. [DOI] [PubMed] [Google Scholar]
- 20.Turner MO, Patel A, Ginsburg S, FitzGerald JM. Bronchodilator delivery in acute airflow obstruction. A meta-analysis. Arch Intern Med. 1997;157:1736–1744. [PubMed] [Google Scholar]
- 21.Gibson PG, Coughlan J, Wilson AJ, Hensley MJ, Abramson M, Bauman A, et al. Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. The effects of limited (information only) patient education programs on the health outcomes of adults with asthma. [Google Scholar]
- 22.Plotnick LH, Ducharme FM Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Efficacy and safety of combined inhaled anticholinergics and beta-s-agonists in the initial management of acute pediatric asthma. [Google Scholar]
- 23.Rowe BH, Spooner CH, Ducharme FM, Bretzlaff JA, Bota GW Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. The effectiveness of corticosteroids in the treatment of acute exacerbations of asthma: a meta-analysis of their effect on relapse following acute assessment. [Google Scholar]
- 24.Davies H, Olson L, Gibson P Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Methotrexate as a steroid sparing agent in adult asthma. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25.Spooner CH, Saunders LD, Rowe BH Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Nedocromil sodium as single dose prophylactic treatment of exercise-induced bronchoconstriction. [Google Scholar]
- 26.Abramson MJ, Puy RM, Weiner JM Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. Allergen specific immunotherapy for asthma. [Google Scholar]
- 27.Hammerquist C, Burr ML, Gotzsche PC Cochrane Collaboration, editors. Cochrane Library. Issue 3. Oxford: Update Software; 1998. House dust mite control measures in the management of asthma. [Google Scholar]
- 28.Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF, for the QUOROM Group. Improving the quality of reports of meta-analyses of randomized controlled trials: the QUOROM statement [Eprint, unreviewed]. Lancet Interactivewww.thelancet.com/newlancet/reg/issues/vol354no9172/commentary2.html (accessed 18 October, 1999). [DOI] [PubMed]
- 29.Shea B, Dube C, Moher D. Assessing the quality of reports of systematic reviews and meta-analyses: a systematic review of checklists and scales. Abstract presented at the VII Cochrane Colloquium. 1999.
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.