Skip to main content
PMC home page
PLOS ONE logoLink to PLOS ONE
. 2007 Dec 26;2(12):e1350. doi: 10.1371/journal.pone.0001350

External Validation of a Measurement Tool to Assess Systematic Reviews (AMSTAR)

Beverley J Shea 1,2,*, Lex M Bouter 3, Joan Peterson 4, Maarten Boers 5, Neil Andersson 1,6, Zulma Ortiz 7, Tim Ramsay 4, Annie Bai 8, Vijay K Shukla 8, Jeremy M Grimshaw 4
Editor: Joel Gagnier9
PMCID: PMC2131785  PMID: 18159233

Abstract

Background

Thousands of systematic reviews have been conducted in all areas of health care. However, the methodological quality of these reviews is variable and should routinely be appraised. AMSTAR is a measurement tool to assess systematic reviews.

Methodology

AMSTAR was used to appraise 42 reviews focusing on therapies to treat gastro-esophageal reflux disease, peptic ulcer disease, and other acid-related diseases. Two assessors applied the AMSTAR to each review. Two other assessors, plus a clinician and/or methodologist applied a global assessment to each review independently.

Conclusions

The sample of 42 reviews covered a wide range of methodological quality. The overall scores on AMSTAR ranged from 0 to 10 (out of a maximum of 11) with a mean of 4.6 (95% CI: 3.7 to 5.6) and median 4.0 (range 2.0 to 6.0). The inter-observer agreement of the individual items ranged from moderate to almost perfect agreement. Nine items scored a kappa of >0.75 (95% CI: 0.55 to 0.96). The reliability of the total AMSTAR score was excellent: kappa 0.84 (95% CI: 0.67 to 1.00) and Pearson's R 0.96 (95% CI: 0.92 to 0.98). The overall scores for the global assessment ranged from 2 to 7 (out of a maximum score of 7) with a mean of 4.43 (95% CI: 3.6 to 5.3) and median 4.0 (range 2.25 to 5.75). The agreement was lower with a kappa of 0.63 (95% CI: 0.40 to 0.88). Construct validity was shown by AMSTAR convergence with the results of the global assessment: Pearson's R 0.72 (95% CI: 0.53 to 0.84). For the AMSTAR total score, the limits of agreement were −0.19±1.38. This translates to a minimum detectable difference between reviews of 0.64 ‘AMSTAR points’. Further validation of AMSTAR is needed to assess its validity, reliability and perceived utility by appraisers and end users of reviews across a broader range of systematic reviews.

Introduction

High quality systematic reviews are increasingly recognized as providing the best evidence to inform health care practice and policy [1]. The quality of a review, and so its worth, depends on the extent to which, scientific review methods were used to minimize the risk of error and bias. The quality of published reviews can vary considerably, even when they try to answer the same question [2]. As a result, it is necessary to appraise their quality (as is done for any research study) before the results are implemented into clinical or public health practice. Much has been written on how best to appraise systematic reviews, and while there is some variation on how this is achieved, most agree on key components of the critical appraisal [3]. Methodological quality can be defined as the extent to which the design of a systematic review will generate unbiased results [4].

Several instruments exist to assess the methodological quality of systematic reviews [5], but not all of them have been developed systematically or empirically validated and have achieved general acceptance. The authors of this paper acknowledge that the methodological quality and reporting quality for systematic reviews is very different. The first, methodological quality, considers how well the systematic review was conducted (literature searching, pooling of data, etc.). The second, reporting quality, considers how well systematic reviewers have reported their methodology and findings. Existing instruments often try to include both types of methods without being conceptually clear about the differences.

In an attempt to achieve some consistency in the evaluation of systematic reviews we have developed a tool to assess their methodological quality. This builds on previous work [6], and is based on empirical evidence and expert consensus. A measurement tool to assess systematic reviews (AMSTAR) was highly rated in a recent review (personal communication) of quality assessment instruments performed by the Canadian Agency for Drugs and Technologies in Health (CADTH). In this study we present the results of an external validation of AMSTAR using data from a series of systematic reviews obtained from the gastroenterology literature.

Methods

The characteristics and basic properties of the instrument have been described elsewhere [7]. Briefly, a 37-item initial assessment tool was formed by combining a) the enhanced Overview Quality Assessment Questionnaire (OQAQ) scale, b) a checklist created by Sacks, and c) three additional items recently judged by experts in the field to be of methodological importance. In its development phase the instrument was applied to 99 paper-based and 52 electronic systematic reviews [6] [7]. Exploratory factor analysis was used to identify underlying components. The results were considered by methodological experts using a nominal group process to reduce the number of items and design an assessment tool with face and content validity. This process lead to an 11-item instrument [7]. A description of the instrument is provided in Annex S1.

External validity

For our validation test set we chose to use systematic reviews or meta-analyses in the area of gastroenterology, specifically upper gastrointestinal. CADTH's informational specialist searched electronic bibliographic databases (i.e. Medline, Central and EMBASE) up to and including 2005. A total of 42 systematic reviews met the a priori criteria and were included [8]. This sample included seven electronic Cochrane systematic reviews and 35 paper-based non-Cochrane reviews. The topics of the reviews ranged across the spectrum of GI problems like dyspepsia, gastro-esophageal reflux disease (GERD), peptic ulcer disease (PUD), and also GI drug interventions such as H2 receptor antagonists and proton pump inhibitors [9]–[50].

Two CADTH assessors from two review groups (SS and FA, AL and CY) independently applied AMSTAR to each review and reached agreement on the assessment results. To assess construct validity, two reviewers (JP, ZO) plus a clinician and/or methodologist (MB, DF, DP, MO, and DH) applied a global assessment to each review [51] (Annex S2).

Agreement and reliability

We calculated an overall agreement score using the weighted Cohen's kappa, as well as one for each item [52] (Table 1). Bland and Altman's limits of agreement methods were used to display agreement graphically [53], [54] (Fig. 1). We calculated the percentage of the theoretical maximum score. Pearson's Rank correlation coefficients were used to assess reliability of this total score. For comparisons of rating the methodological quality we calculated chance-corrected agreement (using kappa) and chance-independent agreement (using Φ) [52], [55], [56]. We accepted a correlation of >0.66. We further scrutinized items and reviews with kappa scores below 0.66 [52]. Kappa values of less than 0 rate as less than chance agreement; 0.01–0.20 slight agreement; 0.21–0.40 fair agreement; 0.41–0.60 moderate agreement; 0.61–0.80 substantial agreement; and 0.81–0.99 almost perfect agreement [52], [57]. We calculated PHI Φ for each question [55], [58].

Table 1. Assessment of the inter-rater agreement for AMSTAR.

Items Kappa (95% CI) PHI Φ
1. Was an ‘a priori’ design provided? 0.75 (0.55 to 0.96) 0.76
2. Was there duplicate study selection and data extraction? 0.81 (0.63 to 0.99) 0.83
3. Was a comprehensive literature search performed? 0.88 (0.73 to 1.00) 0.89
4. Was the status of publication (i.e. grey literature) used as an inclusion criterion? 0.64 (0.40 to 0.88) 0.64
5. Was a list of studies (included and excluded) provided? 0.84 (0.67 to 1.00) 0.84
6. Were the characteristics of the included studies provided? 0.76 (0.55 to 0.96) 0.76
7. Was the scientific quality of the included studies assessed and documented? 0.90 (0.77 to 1.00) 0.91
8. Was the scientific quality of the included studies used appropriately in formulating conclusions? 0.51 (0.25 to 0.78) 0.56
9. Were the methods used to combine the findings of studies appropriate? 0.80 (0.63 to 0.99) 0.80
10. Was the likelihood of publication bias assessed? 0.85 (0.64 to 1.00) 0.85
11. Were potential conflicts of interest included? 1.00 (100% no) 1.00
Overall Score 0.84 (0.67 to 1.00) 0.85
Open in a new tab

Figure 1. Bland and Altman limits of agreement plot for AMSTAR scores.

Figure 1

Open in a new tab

Construct validity

We assessed construct validity (i.e. evaluation of a hypothesis about the expected performance of an instrument) by converting the total mean score (mean of the two assessors) for each of the 42 reviews to a percentage of the maximum score for AMSTAR and of the maximum score of the global assessment instrument. We used Pearson's Rank correlation coefficients, Pearson's R and Kruskal-Wallis test to further explore the impact of the following items on the construct validity of AMSTAR: a) Cochrane systematic review vs. non-Cochrane systematic reviews [59], [60], b) journal type [61], c) year of publication [62], d) conflict of interest [63], e) impact factor [64], and number of pages [64]. We studied these in the context of a priori hypotheses concerning the correlation of AMSTAR scores. Because of the nature of their development, we anticipated that Cochrane systematic reviews would have higher quality scores than non-Cochrane systematic reviews and those electronic or general journals would score higher than specialist journals. We reported on impact factors for these journals. We hypothesized that reviews published more recently would be of higher quality than those published earlier. In addition, we anticipated that reviews declaring a conflict of interest might have lower quality scores [63], [64].

We assessed the practicability of the new instrument by recording the time it took to complete scoring and the instances where scoring was difficult. We interviewed assessors (N = 6) to obtain data on clarity, ambiguity, completeness and user-friendliness.

We used SPSS (versions 13 and 15) and MedCalc for Windows, version 8.1.0.0.

Results

The 42 reviews included in the study had a wide range of quality scores. The overall scores estimated by the AMSTAR instrument ranged from 0 to 10 (out of a maximum of 11) with a mean of 4.6 (95% CI: 3.7 to 5.6; median 4.0 (range 2.0 to 6.0). The overall scores for the global assessment instrument ranged from 2 to 7 (out of a maximum score of seven) with a mean of 4.43 (95% CI: 3.6 to 5.3) and median 4.0 (range 2.5 to 5.3).

Agreement and Reliability

The reliability of the total AMSTAR score between two assessors (the sum of all items answered ‘yes’ scored as 1, all others as 0) was (kappa 0.84 (95% CI: 0.67 to 1.00, Φ = 0.85) and Pearson's R 0.96 (95% CI: 0.92 to 0.98). The inter-rater agreement (kappa) between two raters, for the global assessment was 0.63 (95% CI: 0.40 to 0.88).

Items in AMSTAR displayed levels of agreement that ranged from moderate to almost perfect; nine items scored a kappa of >0.75 (0.55 to 0.96 (and Φ >0.76). Item 4 had a kappa of 0.64 (0.40 to 0.88) Φ = 0.64 and item 8 a kappa of 0.51(0.25 to 0.78 Φ = 0.56). The reliability of the total AMSTAR score was excellent (kappa 0.84 (95% CI: 0.67 to 1.00 and Pearson's R 0.96 (95% CI: 0.92 to 0.98). For the AMSTAR total score, the limits of agreement were −0.19±1.38 (Fig. 1).

The mean age of our reviewers was 40.57, median 43. Fifty-seven percent were identified as experts in methodology and 43% were identified as content experts in the field.

Construct validity

Expressed as a percentage of the maximum score, the results of AMSTAR converged with the results of the global assessment instrument [Pearson's Rank Correlation Coefficient 0.72 (95% CI: 0.53 to 0.84)]. AMSTAR scoring also upheld our other a priori hypotheses. The sub-analysis revealed that Cochrane reviews had significantly higher scores than paper-based reviews with a (R = 37.21 n = 7) for Cochrane reviews and (R = 18.36 n = 35) for paper-based (P<0.0002). Cochrane reviews (R = 37.21 n = 7) also scored higher than reviews published in general journals (R = 25.77 n = 11) and specialty journals (R = 14.96, n = 24) (P<0.0001). Reviews published from 2000 onward had higher AMSTAR scores than earlier reviews (R = 25.20, n = 25 vs. R = 13.12, n = 17; P = 0.0002).

The journals had the following overall summary statistics for the impact factors: mean 5.88 (95% CI: 3.9 to 7.9) median 3.3 (lowest value 1.4, highest value 23.9). There is no statistical association between AMSTAR score and impact factor (Pearson's R (0.555 P = 0.7922)). There was however a significant association found with the number of pages and AMSTAR scores (Pearson's R (0.5623 P = 0.0001 n = 42). We found no association (R 0.1773 P = 0.0308) when we removed the outliers (i.e. systematic reviews with higher page numbers).

Conflict of interest was poorly presented. Of the 42 reviews assessed, no study had appropriately declared their conflict of interest. Therefore, we were unable to assess whether or not funding had a positive or negative effect on the AMSTAR score.

Practicability

Both AMSTAR and the global assessment required on average 15 minutes to complete, but with the latter, assessors expressed difficulty in reaching a final decision in the absence of comprehensive guidelines. In contrast, AMSTAR was well received.

Discussion

Principal findings

This paper describes an external validation of AMSTAR. This new measurement tool to assess methodological quality of systematic reviews showed satisfactory inter-observer agreement, reliability and construct validity in this study. Items in AMSTAR displayed levels of agreement that ranged from moderate to almost perfect. The reliability of the total AMSTAR score was excellent. Construct validity was shown by AMSTAR convergence with the results of the global assessment instrument.

We found a significant association between number of published pages and overall AMSTAR score, suggesting that the longer the manuscript, the higher the quality score. It should be interpreted with caution given the fact that only a couple of the longer reviews largely drive the hypothesis tests. We found no association when the outliers were removed from the dataset. We did not find an association between AMSTAR score and impact factor.

The AMSTAR instrument was developed pragmatically using previously published tools and expert consensus. The original 37 items were reduced to an 11- item instrument addressing key domains; the resulting instrument was judged by the expert panel to have face and content validity [7].

Strengths and weaknesses of the study

This is a prospective external validation study. We compared the new instrument to an independent and reliable gold standard designed for assessing the quality of systematic reviews, allowing multiple testing of convergent validity.

The analytical methods for assessing quality and measuring agreement amongst assessors need further discussion and development. We calculated chance-corrected agreement, using the kappa statistic [57], [65]. While avoiding high levels of agreement due to chance, kappa has its own limitations that have lead to academic criticism [66], [67]. One of the major difficulties with kappa is that when the proportion of positive ratings is extreme, the possible agreement above chance agreement is small and it is difficult to achieve even moderate values of kappa. Thus, if one uses the same raters in a variety of settings, as the proportion of positive ratings becomes extreme, kappa will decrease even if the manner in which the assessors rate the quality does not change. To address this limitation, we also calculated chance-independent agreement using PHIΦ, a relatively new approach to assessing observer agreement [55], [58].

We were unable to test our convergent validity hypothesis about conflict of interest because of missing data in the systematic reviews and primary studies. This highlights the need for journals and journal editors to require that the information is provided.

Our results are based on a small sample of systematic reviews in a particular clinical area and a relatively small number of AMSTAR assessors. There is a need for replication in larger and different data sets with more diverse appraisers.

Possible mechanisms and implications for clinicians or policymakers

Existing systematic review appraisal instrument did not reflect current evidence on potential sources of bias in systematic reviews and were generally not validated. The best available instrument prior to the development of AMSTAR was OQAQ which was formally validated. However, users of OQAQ frequently had to develop their own rules for operationalizing the instrument and OQAQ does not reflect current evidence on sources of potential bias in systematic reviews (for example funding source and conflict of interest [68], [69], [70]).

Quality assessment instruments can focus on either reporting quality (how well systematic reviewers have reported their methodology and findings (internal validity) or methodological quality (how well the systematic review was conducted (literature searching, pooling of data, etc.). It is possible for a systematic review with poor methodological quality to have good reporting quality. For this reason, the AMSTAR items focus on methodological quality.

Decision-makers have spent the last ten years trying to work out the best way to use the enormous amounts of systematic reviews available to them. They can hardly know where to start when deciding whether the relevant literature is valid and of the highest quality. AMSTAR is a user friendly methodological quality assessment that has the potential to standardize appraisal of systematic reviews. Early experience suggests that relevant groups are finding the instrument useful.

Unanswered questions and future research

Further validation of AMSTAR is needed to assess its validity, reliability and perceived utility by appraisers and end users of reviews across a broader range of systematic reviews. We need to assess the responsiveness of AMSTAR looking at its sensitivity to discriminate between high and low methodological quality reviews.

We need to assess the applicability of AMSTAR for reviews of observational (diagnostic, etiological and prognostic) studies and if necessary develop AMSTAR extensions for these reviews.

We plan to update AMSTAR as new evidence regarding sources of bias within systematic reviews becomes available.

Supporting Information

Annex S1

AMSTAR is a measurement tool created to assess the methodological quality of systematic reviews.

(0.04 MB DOC)

Click here for additional data file. (40.5KB, doc)
Annex S2

Global assessment rating

(0.03 MB DOC)

Click here for additional data file. (27.5KB, doc)

Acknowledgments

We would like to thank our International panel of assessors: Daniel Francis, David Henry, Marisol Betancourt, Dana Paul, Martin Olmos, and our local team of assessors: Sumeet Singh, Avtar Lal, Changhua Yu, Fida Ahmed. We also thank Dr. Giuseppe G.L. Biondi-Zoccai and Crystal Huntly-Ball for their helpful suggestions on this manuscript.

Footnotes

Competing Interests: The authors have declared that no competing interests exist.

Funding: The authors have no support or funding to report.

References

  • 1.Young D. Policymakers, experts review evidence-based medicine. Am. J. Health Syst Pharm. 2005;62(4):342–343. doi: 10.1093/ajhp/62.4.0342. [DOI] [PubMed] [Google Scholar]
  • 2.Dolan-Mullen P, Ramírez G. The Promise and Pitfalls of Systematic Reviews. Annual Review of Public Health. 2006;27:81–102. doi: 10.1146/annurev.publhealth.27.021405.102239. [DOI] [PubMed] [Google Scholar]
  • 3.Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44(11):1271–78. doi: 10.1016/0895-4356(91)90160-b. [DOI] [PubMed] [Google Scholar]
  • 4.Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, et al. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials. 1995;16(1):62–73. doi: 10.1016/0197-2456(94)00031-w. [DOI] [PubMed] [Google Scholar]
  • 5.Shea B, Dube C, Moher D. Assessing the quality of reports of systematic reviews: the QUOROM statement compared to other tools. Systematic review in health care meta-analysis in context. London: BMJ Books. 2001;(7):122–39. [Google Scholar]
  • 6.Shea B. University of Ottawa, Department of Epidemiology and Community Medicine; 1999. Assessing the quality of reporting meta-analyses of randomized controlled trials. MSc thesis. [Google Scholar]
  • 7.Shea B, Grimshaw JM, Wells GA, Boers M, Andersson N, et al. Development of AMSTAR: A Measurement Tool to Assess Systematic Reviews. BMC Medical Research Methodology. 2007;7:10. doi: 10.1186/1471-2288-7-10. doi:10.1186/1471-2288-7-10. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 8.Singh S, Bai A, Lal A, Yu C, Ahmed F, et al. Ottawa, Canada: The Canadian Agency for Drugs and Technologies in Health (CADTH); 2006. Developing evidence-based best practices for the prescribing and use of proton pump inhibitors in Canada. [Google Scholar]
  • 9.Chiba N, De Gara CJ, Wilkinson JM, Hunt RH. Speed of healing and symptom relief in grade II to IV gastroesophageal reflux disease: a meta-analysis. Gastroenterology. 1997;112(6):1798–810. doi: 10.1053/gast.1997.v112.pm9178669. [DOI] [PubMed] [Google Scholar]
  • 10.Caro JJ, Salas M, Ward A. Healing and relapse rates in gastroesophageal reflux disease treated with the newer proton-pump inhibitors lansoprazole, rabeprazole, and pantoprazole compared with omeprazole, ranitidine, and placebo: evidence from randomized clinical trials. Clin Ther. 2001;23(7):998–1017. doi: 10.1016/s0149-2918(01)80087-4. [DOI] [PubMed] [Google Scholar]
  • 11.Klok RM, Postma MJ, van Hout BA, Brouwers JR. Meta-analysis: comparing the efficacy of proton pump inhibitors in short-term use. Aliment Pharmacol Ther. 2003;17(10):1237–45. doi: 10.1046/j.1365-2036.2003.01562.x. [DOI] [PubMed] [Google Scholar]
  • 12.Van Pinxteren B, Numans ME, Lau J, de Wit NJ, Hungin AP, et al. Short-term treatment of gastroesophageal reflux disease. J Gen Intern Med. 2003;18(9):755–63. doi: 10.1046/j.1525-1497.2003.20833.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Van Pinxteren B, Numans ME, Bonis PA, Lau J. Short-term treatment with proton pump inhibitors, H2-receptor antagonists and prokinetics for gastro-oesophageal reflux disease-like symptoms and endoscopy negative reflux disease. Cochrane Database Syst Rev. 2004;(3):CD002095. doi: 10.1002/14651858.CD002095.pub2. [DOI] [PubMed] [Google Scholar]
  • 14.Rostom A, Dube C, Wells G, Tugwell P, Welch V, et al. Prevention of NSAID-induced gastroduodenal ulcers. Cochrane Database Syst Rev. 2002;(4):CD002296. doi: 10.1002/14651858.CD002296. [DOI] [PubMed] [Google Scholar]
  • 15.Laheij RJ, van Rossum LG, Jansen JB, Straatman H, Verbeek AL. Evaluation of treatment regimens to cure Helicobacter pylori infection: a meta-analysis. Aliment Pharmacol Ther. 1999;13(7):857–64. doi: 10.1046/j.1365-2036.1999.00542.x. [DOI] [PubMed] [Google Scholar]
  • 16.Carlsson R, Galmiche JP, Dent J, Lundell L, Frison L. Prognostic factors influencing relapse of oesophagitis during maintenance therapy with antisecretory drugs: a meta-analysis of long-term omeprazole trials. Aliment Pharmacol Ther. 1997;11(3):473–82. doi: 10.1046/j.1365-2036.1997.00167.x. [DOI] [PubMed] [Google Scholar]
  • 17.Chiba N. Proton pump inhibitors in acute healing and maintenance of erosive or worse esophagitis: a systematic overview. Can J Gastroenterol. 1997;11(Suppl B):66B–73B. [PubMed] [Google Scholar]
  • 18.Delaney B, Moayyedi P, Deeks J, Innes M, Soo S, et al. 2000. The management of dyspepsia: a systematic review. Health Technol Assess 4(39); i,iii-189. Available: http://www.ncchta.org/execsumm/summ439.htm. [DOI] [PubMed] [Google Scholar]
  • 19.Moayyedi P, Soo S, Deeks J, Delaney B, Harris A, et al. Eradication of Helicobacter pylori for non-ulcer dyspepsia. Cochrane Database Syst Rev. 2005;(1):CD002096. doi: 10.1002/14651858.CD002096.pub2. [DOI] [PubMed] [Google Scholar]
  • 20.Moayyedi P, Soo S, Deeks J, Delaney B, Innes M, et al. Pharmacological interventions for non-ulcer dyspepsia. Cochrane Database Syst Rev. 2005;(1):CD001960. doi: 10.1002/14651858.CD001960. Available: http://www.mrw.interscience.wiley.com/cochrane/clsysrev/articles/CD001960/pdf_fs.html (accessed 2006 Feb 16) [DOI] [PubMed] [Google Scholar]
  • 21.Delaney BC, Moayyedi P, Forman D. Initial management strategies for dyspepsia. Cochrane Database Syst Rev. 2003;(2):CD001961. doi: 10.1002/14651858.CD001961. [DOI] [PubMed] [Google Scholar]
  • 22.Hopkins RJ, Girardi LS, Turney EA. Relationship between Helicobacter pylori eradication and reduced duodenal and gastric ulcer recurrence: a review. Gastroenterology. 1996;110(4):1244–52. doi: 10.1053/gast.1996.v110.pm8613015. [DOI] [PubMed] [Google Scholar]
  • 23.Huang JQ, Sridhar S, Hunt RH. Role of Helicobacter pylori infection and non-steroidal anti-inflammatory drugs in peptic-ulcer disease: a meta-analysis. Lancet. 2002;359(9300):14–22. doi: 10.1016/S0140-6736(02)07273-2. [DOI] [PubMed] [Google Scholar]
  • 24.Moayyedi P, Soo S, Deeks J, Forman D, Mason J, et al. Systematic review and economic evaluation of Helicobacter pylori eradication treatment for non-ulcer dyspepsia. Dyspepsia Review Group. BMJ. 2000;321(7262):659–64. doi: 10.1136/bmj.321.7262.659. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Jovell AJ, Aymerich M, Garcia Altes A, Serra Prat M. Barcelona: Catalan Agency for Health Technology Assessment; 1998. Clinical practice guideline for the eradicating therapy of Helicobacter pylori infections associated to duodenal ulcer in primary care. Available: http://www.gencat.net/salut/depsan/units/aatrm/pdf/gp9802en.pdf. [Google Scholar]
  • 26.Gisbert JP, González L, Calvet X, García N, López T. Proton pump inhibitor, clarithromycin and either amoxycillin or nitroimidazole: a meta-analysis of eradication of Helicobacter pylori. Aliment Pharmacol Ther. 2000;14(10):1319–28. doi: 10.1046/j.1365-2036.2000.00844.x. [DOI] [PubMed] [Google Scholar]
  • 27.Calvet X, García N, López T, Gisbert JP, Gené E, et al. A meta-analysis of short versus long therapy with a proton pump inhibitor, clarithromycin and either metronidazole or amoxycillin for treating Helicobacter pylori infection. Aliment Pharmacol Ther. 2000;14(5):603–09. doi: 10.1046/j.1365-2036.2000.00744.x. [DOI] [PubMed] [Google Scholar]
  • 28.Gené E, Calvet X, Azagra R, Gisbert JP. Triple vs. quadruple therapy for treating Helicobacter pylori infection: a meta-analysis. Aliment Pharmacol Ther. 2003;17(9):1137–43. doi: 10.1046/j.1365-2036.2003.01566.x. [DOI] [PubMed] [Google Scholar]
  • 29.Huang J, Hunt RH. The importance of clarithromycin dose in the management of Helicobacter pylori infection: a meta-analysis of triple therapies with a proton pump inhibitor, clarithromycin and amoxycillin or metronidazole. Aliment Pharmacol Ther. 1999;13(6):719–29. doi: 10.1046/j.1365-2036.1999.00530.x. [DOI] [PubMed] [Google Scholar]
  • 30.Leodolter A, Kulig M, Brasch H, Meyer Sabellek W, Willich SN, et al. A meta-analysis comparing eradication, healing and relapse rates in patients with Helicobacter pylori-associated gastric or duodenal ulcer. Aliment Pharmacol Ther. 2001;15(12):1949–58. doi: 10.1046/j.1365-2036.2001.01109.x. [DOI] [PubMed] [Google Scholar]
  • 31.Moayyedi P, Murphy B. Helicobacter pylori: a clinical update. J Appl Microbiol. 2001;(30):126S–33S. doi: 10.1046/j.1365-2672.2001.01361.x. [DOI] [PubMed] [Google Scholar]
  • 32.Oderda G, Rapa A, Bona G. A systematic review of Helicobacter pylori eradication treatment schedules in children. Aliment Pharmacol Ther. 2000;14(Suppl 3):59–66. doi: 10.1046/j.1365-2036.2000.03102.x. [DOI] [PubMed] [Google Scholar]
  • 33.Schmid CH, Whiting G, Cory D, Ross SD, Chalmers TC. Omeprazole plus antibiotics in the eradication of Helicobacter pylori infection: a meta-regression analysis of randomized, controlled trials. Am J Ther. 1999;6(1):25–36. doi: 10.1097/00045391-199901000-00005. [DOI] [PubMed] [Google Scholar]
  • 34.Unge P, Berstad A. Pooled analysis of anti-Helicobacter pylori treatment regimens. Scand J Gastroenterol Suppl. 1996;220:27–40. [PubMed] [Google Scholar]
  • 35.Unge P. Antimicrobial treatment of H. pylori infection: a pooled efficacy analysis of eradication therapies. Eur J Surg Suppl. 1998;582:16–26. doi: 10.1080/11024159850191391. [DOI] [PubMed] [Google Scholar]
  • 36.Unge P. What other regimens are under investigation to treat Helicobacter pylori infection? Gastroenterology. 1997;113(6 Suppl):S131–S148. doi: 10.1016/s0016-5085(97)80027-3. [DOI] [PubMed] [Google Scholar]
  • 37.Vallve M, Vergara M, Gisbert JP, Calvet X. Single vs. double dose of a proton pump inhibitor in triple therapy for Helicobacter pylori eradication: a meta-analysis. Aliment Pharmacol Ther. 2002;16(6):1149–56. doi: 10.1046/j.1365-2036.2002.01270.x. [DOI] [PubMed] [Google Scholar]
  • 38.Veldhuyzen van Zanten SJ, Sherman PM. Indications for treatment of Helicobacter pylori infection: a systematic overview. CMAJ. 1994;150(2):189–98. [PMC free article] [PubMed] [Google Scholar]
  • 39.Trépanier EF, Agro K, Holbrook AM, Blackhouse G, Goeree R, et al. Meta-analysis of H pylori (HP) eradication rates in patients with duodenal ulcer (DU). Can J Clin Pharmacol. 1998;5(1):67. [Google Scholar]
  • 40.Bamberg P, Caswell CM, Frame MH, Lam SK, Wong EC. A meta-analysis comparing the efficacy of omeprazole with H2-receptor antagonists for acute treatment of duodenal ulcer in Asian patients. J Gastroenterol Hepatol. 1992;7(6):577–85. doi: 10.1111/j.1440-1746.1992.tb01489.x. [DOI] [PubMed] [Google Scholar]
  • 41.Di Mario F, Battaglia G, Leandro G, Grasso G, Vianello F, et al. Short-term treatment of gastric ulcer: a meta-analytical evaluation of blind trials. Dig Dis Sci. 1996;41(6):1108–31. doi: 10.1007/BF02088227. [DOI] [PubMed] [Google Scholar]
  • 42.Eriksson S, Langstrom G, Rikner L, Carlsson R, Naesdal J. Omeprazole and H2-receptor antagonists in the acute treatment of duodenal ulcer, gastric ulcer and reflux oesophagitis: a meta-analysis. Eur J Gastroenterol Hepatol. 1995;7(5):467–75. [PubMed] [Google Scholar]
  • 43.Poynard T, Lemaire M, Agostini H. Meta-analysis of randomized clinical trials comparing lansoprazole with ranitidine or famotidine in the treatment of acute duodenal ulcer. Eur J Gastroenterol Hepatol. 1995;7(7):661–65. [PubMed] [Google Scholar]
  • 44.Laine L, Schoenfeld P, Fennerty MB. Therapy for Helicobacter pylori in patients with nonulcer dyspepsia: a meta-analysis of randomized, controlled trials. Ann Intern Med. 2001;134(5):361–9. doi: 10.7326/0003-4819-134-5-200103060-00008. [DOI] [PubMed] [Google Scholar]
  • 45.Mulder CJ, Schipper DL. Omeprazole and ranitidine in duodenal ulcer healing. Analysis of comparative clinical trials. Scand J Gastroenterol Suppl. 1990;178:62–6. doi: 10.3109/00365529009093152. [DOI] [PubMed] [Google Scholar]
  • 46.Shiau JY, Shukla VK, Dubé C. Ottawa: Canadian Coordinating Office for Health Technology Assessment; 2002. The efficacy of proton pump inhibitors in adults with functional dyspepsia. [Google Scholar]
  • 47.Danesh J, Lawrence M, Murphy M, Roberts S, Collins R. Systematic review of the epidemiological evidence on Helicobacter pylori infection and non-ulcer or uninvestigated dyspepsia. Arch Intern Med. 2005;160(8):1192–98. doi: 10.1001/archinte.160.8.1192. [DOI] [PubMed] [Google Scholar]
  • 48.Gibson PG, Henry RL, Coughlan JL. Gastro-esophageal reflux treatment for asthma in adults and children. Cochrane Database Syst Rev. 2005;(3):1–27. doi: 10.1002/14651858.CD001496. [DOI] [PubMed] [Google Scholar]
  • 49.Fischbach LA, Goodman KJ, Feldman M, Aragaki C. Sources of variation of helicobacter pylori treatment success in adults worldwide: a meta-analysis. Int J Epidemiol. 2002;31(1):128–39. doi: 10.1093/ije/31.1.128. [DOI] [PubMed] [Google Scholar]
  • 50.Ford A, Delaney B, Moayyedi P. Eradication therapy for peptic ulcer disease in helicobacter pylori positive patients. Cochrane Database Syst Rev. 2003;(4):CD003840. doi: 10.1002/14651858.CD003840. [DOI] [PubMed] [Google Scholar]
  • 51.Oxman AD, Guyatt GH. Validation of an index of the quality of review articles. J Clin Epidemiol. 1991;44(11):1271–78. doi: 10.1016/0895-4356(91)90160-b. [DOI] [PubMed] [Google Scholar]
  • 52.Cohen J. A coefficient of agreement for nominal scales. Educ Psychol Measure. 1960;0(1):622–26. [Google Scholar]
  • 53.Bland JM, Altman DG. Statistical methods for assessing agreement between two methods of clinical Measurement. Lancet. 1986;i:307–10. [PubMed] [Google Scholar]
  • 54.Bland JM, Altman DG. Statistical methods for assessing agreement between measurement. Biochimica Clinica. 1987;11:399–404. [Google Scholar]
  • 55.Meade M, Cook R, Guyatt G, Groll R, Kachura J, et al. Interobserver Variation in Interpreting Chest Radiographs for the Diagnosis of Acute Respiratory Distress Syndrome. Am. J. Respir. Crit. Care Med. 2000;161(1):185–90. doi: 10.1164/ajrccm.161.1.9809003. [DOI] [PubMed] [Google Scholar]
  • 56.Uebersax JS. Diversity of decision-making models and the measurement of inter-rater agreement. Psychological Bulletin. 1987;101:140–46. [Google Scholar]
  • 57.Cohen J. Weighted kappa: Nominal scale agreement with provision for scaled disagreement or partial credit. Psychol Bull. 1968;70:213–220. doi: 10.1037/h0026256. [DOI] [PubMed] [Google Scholar]
  • 58.McGinn T, Guyatt G, Cook R, Meade M. Diagnosis: measuring agreement beyond chance. In: Guyatt G, Rennie D, editors. Users' guide to the medical literature. A manual for evidence-based clinical practice. Chicago, IL: AMA Press; 2002. pp. 461–70. [Google Scholar]
  • 59.Moja LP, Telaro E, D'Amico R, Moschetti I, Coe L, et al. Assessment of methodological quality of primary studies by systematic reviews: results of the metaquality cross sectional study. BMJ Publishing Group Ltd. 2005;330(7499):1053. doi: 10.1136/bmj.38414.515938.8F. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 60.Shea B, Moher D, Graham I, Pham B, Tugwell P. A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Evaluation & the Health Professions. 2002;25(1):116–29. doi: 10.1177/0163278702025001008. [DOI] [PubMed] [Google Scholar]
  • 61.Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and Reporting Characteristics of Systematic Reviews. PLoS Med. 2007;4(3):e78. doi: 10.1371/journal.pmed.0040078. doi:10.1371/journal.pmed.0040078. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 62.Sacks HS, Berrier J, Reitman D, Ancona-Berk VA, Chalmers TC. Meta-analyses of randomized controlled trials. New England Journal of Medicine. 1987;316:450–54. doi: 10.1056/NEJM198702193160806. [DOI] [PubMed] [Google Scholar]
  • 63.Bero LA. Managing financial conflicts of interest in research. Journal of the American College of Dentists. 2005;72(2):4–9. [PubMed] [Google Scholar]
  • 64.Biondi-Zoccai G, Lotrionte M, Abbate A, Testa L. Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ. 2006;332:202–6. doi: 10.1136/bmj.38693.516782.7C. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 65.Fleiss JL. Measuring nominal scale agreement among many raters. Psychol. Bull. 1971;76:378–382. [Google Scholar]
  • 66.McClure M, Willett W. Misinterpretation and misuse of the kappa statistic. Am. J. Epidemiol. 1987;126:161–169. doi: 10.1093/aje/126.2.161. [DOI] [PubMed] [Google Scholar]
  • 67.Cook RJ, Farewell VT. Conditional inference for subject-specific and marginal agreement: two families of agreement measures. Can. J. Stat. 1995;23:333–344. [Google Scholar]
  • 68.Barnes DE, Bero LA. Why review articles on the health effects of passive smoking reach different conclusions. JAMA. 1998;279:1566–1570. doi: 10.1001/jama.279.19.1566. [DOI] [PubMed] [Google Scholar]
  • 69.Cho MK, Bero LA. The quality of drug studies published in symposium proceedings. Ann Intern Med. 1996;124:485–489. doi: 10.7326/0003-4819-124-5-199603010-00004. [DOI] [PubMed] [Google Scholar]
  • 70.Lexchin J, Bero LA, Djulbegovic B, Clark O. Pharmaceutical industry sponsorship and research outcome and quality: systematic review. BMJ. 2003;326:1167–1170. doi: 10.1136/bmj.326.7400.1167. [DOI] [PMC free article] [PubMed] [Google Scholar]

Associated Data

This section collects any data citations, data availability statements, or supplementary materials included in this article.

Supplementary Materials

Annex S1

AMSTAR is a measurement tool created to assess the methodological quality of systematic reviews.

(0.04 MB DOC)

Click here for additional data file. (40.5KB, doc)
Annex S2

Global assessment rating

(0.03 MB DOC)

Click here for additional data file. (27.5KB, doc)

Articles from PLoS ONE are provided here courtesy of PLOS

RESOURCES