An AMSTAR assessment of the methodological quality of systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science (JAOS)

Abstract

Background

Systematic reviews are not an assembly of anecdotes but a distillation of current best available evidence on a particular topic and as such have an important role to play in evidence-based healthcare. A substantial proportion of these systematic reviews focus on interventions, and are able to provide clinicians with the opportunity to understand and translate the best available evidence on the effects of these healthcare interventions into clinical practice. The importance of systematic reviews in summarising and identifying the gaps in evidence which might inform new research initiatives is also widely acknowledged. Their potential impact on practice and research makes their methodological quality especially important as it may directly influence their utility for clinicians, patients and policy makers. The objectives of this study were to identify systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science (JAOS) and to evaluate their methodological quality using the evaluation tool, AMSTAR.

Methods

Potentially eligible systematic reviews in JAOS were identified through an electronic search of the Scientific Electronic Library Online (SciELO). Details of the relevant aspects of methodology as reported in these systematic reviews were extracted from the full text publications. Methodological quality was assessed independently by two reviewers using the AMSTAR questionnaire.

Results

Five systematic reviews were identified, one of which was subsequently excluded as it was a review of a diagnostic test. Summary AMSTAR scores for the four included reviews were: 1, 5, 2 and 4 out of a maximum score of 11 (range 1-5, mean 3) with only one of the reviews scoring 5.

Conclusion

AMSTAR evaluation of the methodological quality of the relatively small number of systematic reviews published in JAOS illustrated that there was room for improvement. Pre-publication and editorial appraisal of future systematic reviews might benefit from the application of tools such as AMSTAR and is to be recommended.

INTRODUCTION

Substantial progress appears to have been made recently in Brazil in the field of healthcare research. Haddad⁸ (2008) reported that this has placed it among the 25 most productive countries in the world and Brazilian dentistry now ranks 5^th in recognized world scientific production. However whilst there has been a significant increase in the number of research publications concerns have been expressed as to whether quality is keeping pace with quantity¹⁵. As published research increases there is an increasing requirement not only for periodic assessments of its quality but also for its integration and synthesis, so that it can be used as the basis for more well-informed healthcare policy making, and ultimately guide best practice in the provision of healthcare.

Systematic reviews have an important role to play in evidence-based healthcare. For example, reviews of healthcare interventions help clinicians to understand and translate the current best evidence of the effects of these interventions into their daily clinical practice. The importance of systematic reviews in summarising evidence and identifying gaps which might inform new research initiatives is widely acknowledged. This impact of systematic reviews on practice and research makes their methodological quality especially important, as it may influence their utility for clinicians, patients and policy makers¹².

An assessment of the methodological quality of systematic reviews examining the effectiveness of oral and dental healthcare interventions reported that a substantial majority of these reviews were of mediocre quality⁷. A total of 65 reviews were included for assessment spanning publication years of 1990-2001. Major weaknesses highlighted by the investigators were inadequate search strategies; a lack of quality assessment of included studies, no examination of heterogeneity and inappropriate pooling of data.

Six years on from the Glenny⁷ (2003) article and a thought provoking editorial Systematic reviews in endodontics - examples of GIGO? echoed similar concerns and indicated that at least in the field of Endodontics there had been limited improvement and that somewhat alarmingly, the quality of these recent publications is negatively correlated to the frequency²³. In an attempt to provide robust guidance the National Institute for Health and Clinical Excellence (NICE) in UK published a review examining the key components that comprise a methodologically sound systematic review. This study covered healthcare in general and included relevant work in oral healthcare¹⁹.

In 2002, the Agency for Healthcare Research and Quality (AHRQ) undertook a review of methods that could be used to rate the strength of scientific evidence, which included those used to evaluate the quality of systematic reviews. A number of such tools were identified in which the key domains were: study questions, search strategy, inclusion and exclusion criteria, data abstraction, study quality and validity, data synthesis and analysis and funding or sponsorship of reviews²⁴. One of these tools, the Overview Quality Assessment Questionnaire (OQAQ) had been developed several years previously by Oxman and Guyatt¹⁴(1991). Integration of this questionnaire with a quality checklist¹⁷led to the development of a new instrument which could be used to assess the methodological quality of systematic reviews (AMSTAR)²¹.

AMSTAR is an 11-item assessment tool with good face and construct validity, has recently been validated and is being used increasingly by healthcare policy makers, health technology assessment agencies and some authors and journal editors²².

The main use of AMSTAR has been in the assessment of reviews of interventions rather than those covering other aspects of health or health care such as diagnosis, prognosis and etiology. It is used by agencies such as the Canadian Agency for Drugs and Technologies in Health³ in the evaluation of a range of systematic reviews but its use in the assessment of systematic reviews of oral healthcare interventions appears to have been sporadic and limited². However concerns have been expressed recently about the inappropriate use of AMSTAR in an assessment of the methodological quality of systematic reviews in Endodontology⁶.

Brazil has a prodigious number of dental schools, increasing from 32 in 1960 to 191 in 2009¹⁸. Many of these dental schools publish a dental journal and there are more than 90 such journals. Three of these; Brazilian Dental Journal; Journal of Applied Oral Science; Brazilian Oral Research are currently indexed to MEDLINE.

There has been an increased level of interest in evidence-based dentistry in Brazil, which has occurred alongside a steady rise in the number of systematic reviews published in some Brazilian dental journals; but little is known about the methodological quality of these reviews.

OBJECTIVES

To assess the methodological quality of systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science using the AMSTAR tool.

METHODS

Systematic reviews published in JAOS in recent years were identified through the electronic searching of the Scientific Electronic Library Online²⁰. This electronic virtual library is devoted to ensuring access to scientific information generated in low- and middle-income countries and provides access to full text articles of a substantial number of e-journals from Latin America, South Africa and elsewhere. The methodological quality of the included reviews was assessed using the AMSTAR quality assessment tool (A Measurement Tool to Assess Systematic Reviews) (Figure 3).

Figure 3.

AMSTAR quality assessment tool

1. Was an "a priori" design provided?
The research question and inclusion criteria should be established before the conduct of the review.
□ Yes	□ No	□ Can't answer	□ Not applicable
2. Was there duplicate study selection and data extraction?
There should be at least two independent data extractors and a consensus procedure for disagreements should be in place.
□ Yes	□ No	□ Can't answer	□ Not applicable
3. Was a comprehensive literature search performed?
At least two electronic sources should be searched. The report must include years and databases used (e.g. Central, EMBASE, and MEDLINE). Key words and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found.
□ Yes	□ No	□ Can't answer	□ Not applicable
4. Was the status of publication (i.e. grey literature) used as an inclusion criterion?
The authors should state that they searched for reports regardless of their publication type. The authors should state whether or not they excluded any reports (from the systematic review), based on their publication status, language etc.
□ Yes	□ No	□ Can't answer	□ Not applicable
5. Was a list of studies (included and excluded) provided?
A list of included and excluded studies should be provided.
□ Yes	□ No	□ Can't answer	□ Not applicable
6. Were the characteristics of the included studies provided?
In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed e.g. age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases should be reported.
□ Yes	□ No	□ Can't answer	□ Not applicable
7. Was the scientific quality of the included studies assessed and documented?
'A priori' methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant.
□ Yes	□ No	□ Can't answer	□ Not applicable
8. Was the scientific quality of the included studies used appropriately in formulating conclusions?
The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations.
□ Yes	□ No	□ Can't answer	□ Not applicable
9. Were the methods used to combine the findings of studies appropriate?
For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (i.e. Chi-squared test for homogeneity, P). If heterogeneity exists a random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e. is it sensible to combine?).
□ Yes	□ No	□ Can't answer	□ Not applicable
10. Was the likelihood of publication bias assessed?
An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger regression test).
□ Yes	□ No	□ Can't answer	□ Not applicable
11. Was the conflict of interest stated?
Potential sources of support should be clearly acknowledged in both the systematic review and the included studies.
□ Yes	□ No	□ Can't answer	□ Not applicable

Inclusion and exclusion criteria

Reports of systematic reviews were considered eligible for inclusion if they met the following criteria: the terms or phrases systematic review, meta-analysis or overview were used in the title or abstract, or if the main text provided a clear indication that a systematic review had been carried out. In view of the wide and at times indiscriminate use of the terms systematic review, only studies whose overall methods and conduct defined them as a systematic review were included whereas studies which contained for example only a literature search, systematic or otherwise, were excluded. The key components that comprise a methodologically robust systematic review were identified, and then discussed and agreed on by the authors through consensus and were used as criteria for selection of reviews to be included in this study. These comprised; a focused research question, a comprehensive search, clearly defined study inclusion/exclusion criteria, transparent and reproducible selection of studies, a full assessment and reporting of the methodological quality of the included studies, independent data extraction, and plausible analysis and synthesis of data¹⁹. Reviews of a diagnostic test were excluded because the applicability of AMSTAR for reviews of observational (diagnostic, etiological and prognostic) studies has not yet been assessed²¹. If a systematic review did not involve human dental or oral tissues it was also excluded.

Electronic searches and assessment of eligibility

Two investigators (PS and ZF) independently examined the titles and abstracts of all 37 issues (volume 11; Jan/Mar 2003 to volume 17) of the JAOS available on-line and up to December 2009.

Full text copies of potentially eligible studies were obtained and assessed independently against the inclusion criteria by two investigators (ZF and VJ). Any disagreements on eligibility were discussed and if required were resolved by consulting with the third investigator (MS).

Assessment of methodological quality

AMSTAR was applied independently by two assessors (VJ and ZF) to each of the included systematic reviews and any differences in their evaluations were discussed and agreed through consensus. Individual items were categorized as yes, no; can't answer or not applicable and the assessment data were tabulated. Although weights are not attached to any of the individual items in AMSTAR, summary quality scores were calculated, in keeping with other assessments that have used AMSTAR^2,21. Final grading of the methodological quality of each systematic review was based on the overall score and reported as either high, medium or low; in concordance with the rating system used by the CADTH³.

RESULTS

The 37 issues of the JAOS provided a total of 604 titles and abstracts to publications, and most of which were original research reports. Five systematic reviews were identified, one of which was subsequently excluded¹⁶ as it was a review of a diagnostic test. Four systematic reviews (Figure 1) were included and provided data for this study^1,4,5,11. The individual assessments for these reviews are presented in (Figure 2).

Figure 1.

Characteristics of included systematic reviews

Study ID	Review topic
Alves, et al.1 (2005)	Titanium Tetrafluoride and dental caries: a systematic review
Mickenautsch, et al.11 (2007)	Sugar-free chewing gum and dental caries - a systematic review
Estrela, et al.4 (2008)	Efficacy of sodium hypochlorite and chlorhexidine against Enterococcus faecalis
Estrela, et al.5 (2009)	Antibacterial efficacy of intracanal medicaments on bacterial biofilm: a critical review

Figure 2.

Quality assessment of included systematic reviews

AMSTAR Question	YES	NO	Can't Say	Not applicable
"A priori" design provided		*
Duplicate study selection and data extraction		*
Comprehensive literature search			*
Publication status as an inclusion criterion		*
List of studies (included and excluded) provided		*
Characteristics of the included studies provided	*
Quality assessment			*
Quality used appropriately			*
Methods used to combine appropriate			*
Publication bias assessed		*
Conflict of interest stated		*

AMSTAR Question	YES	NO	Can't Say	Not applicable
"A priori" design provided		*
Duplicate study selection and data extraction	*
Comprehensive literature search	*
Publication status as an inclusion criterion	*
List of studies (included and excluded) provided		*
Characteristics of the included studies provided	*
Quality assessment	*
Quality used appropriately			*
Methods used to combine appropriate		*
Publication bias assessed		*
Conflict of interest stated		*

AMSTAR Question	YES	NO	Can't Say	Not applicable
"A priori" design provided		*
Duplicate study selection and data extraction		*
Comprehensive literature search	*
Publication status as an inclusion criterion		*
List of studies (included and excluded) provided		*
Characteristics of the included studies provided	*
Quality assessment		*
Quality used appropriately		*
Methods used to combine appropriate		*
Publication bias assessed		*
Conflict of interest stated		*

AMSTAR Question	YES	NO	Can't Say	Not applicable
"A priori" design provided		*
Duplicate study selection and data extraction	*
Comprehensive literature search	*
Publication status as an inclusion criterion	*
List of studies (included and excluded) provided	*
Characteristics of the included studies provided				*
Quality assessment			*
Quality used appropriately			*
Methods used to combine appropriate				*
Publication bias assessed		*
Conflict of interest stated		*

Alves, et al.¹ (2005): Summary Score (1/11)

Mickenautsch, et al.¹¹ (2007): Summary Score (5/11)

Estrela, et al.⁴ (2008): Summary Score (2/11)

Estrela, et al.⁵ (2009): Summary Score (4/11) Empty Review

Out of the four reviews included in our study, and based on our AMSTAR assessments, three were categorized as of low quality based on their reports in JAOS. Although fairly comprehensive searches i.e. in terms of databases used, were conducted by three of the four reviews, the search strategies were only completetly described in one of the reports. Ideally, selection of studies and data extraction should be conducted by two independent reviewers but this was only adequately reported in two of the reviews. Although all of the reviews provided a list of included and excluded studies, only one review clearly indicated the reasons for exclusion of individual studies. The methods used to combine the results of studies differed quite markedly between the reviews. None of the reviews provided any reassuring evidence that reasonable attempts had been made to assess clinical diversity between the studies i.e. to assess whether the studies are similar enough to combine, and that any pooling of outcomes data across the included studies would be plausible. This evaluation should include an exploration of the characteristics of the eligible studies to assess the similarities and differences among the types of participants, the interventions received and the outcome measures being analysed.

Selection of eligible studies for a review should be based on clearly stated prespecified criteria however the unavailability of protocols for any of these systematic reviews precluded any assessment of whether an a priori design had been used. The possibility of publication bias among eligible studies and its potential effects were not assessed in any of the reviews and there were no disclosures of conflicts of either the authors of these reviews, and no indication of whether they had noted any potential conflicts of interest by the investigators in the studies included in the reviews.

DISCUSSION

The methodological quality of the four systematic reviews published in the JAOS between 2003 and 2009 was rated as low for three of them and medium for Mickenautsch, et al.¹¹ (2007) using the AMSTAR assessment tool.

Only one of the four reviews identified in our study attained a score of 5/11 and few of the reviews adequately satisfied key quality items such as the use of an a priori design; assessment of the quality of included studies and the assessment of publication bias. Yet these are arguably some of the most important prerequisites of a high quality systematic review.

In considering the results of this study it should be kept in mind that there is an important difference between the conduct of a systematic review and its reporting, and which may inadvertently result in a downgrading of any quality assessment. An example where a discrepancy may occur is when a journal in question limits the space available to publish the methods of research studies. In order to compensate some journals provide unlimited space, often online, to publish the methods of the research studies, including systematic reviews, that they contain.

Strengths

One of the strengths of this study is that we were able to identify and examine all systematic reviews in a consecutive sample of issues of JAOS from 2003 to 2009. This was possible because of online access to the relevant issues of the journal.

The simplicity of the design and application of the AMSTAR tool was important, because it made it quick and easy to use. The fact that it has been validated, gives credibility to its use as a tool to assess quality.

Limitations

One of the limitations of our study was the reliance on the summary score from AMSTAR to place articles into different quality categories. Although it is generally agreed that summary scores may mask inherent strengths and weaknesses of systematic reviews²² they can also prove helpful in informing decision making. For example, to help policy makers and clinicians CADTH categorizes AMSTAR quality assessments into three ranges, HIGH (9-11), MEDIUM (5-8), and LOW (0-4)³. Furthermore, the findings from an AMSTAR evaluation can also be used to explore the quality of a review within specific domains.

A further constraint is the very nature of a retrospective study as a point of application for the AMSTAR tool, or indeed any quality assessment. The assessment of the quality of a systematic review should, ideally, be carried out by assessing the methods actually used and not a report (pre-publication or otherwise) of its conduct. In addition the publication of a protocol for a review prior to the full evaluation of the available studies has the advantage that the impact of review authors' biases and the potential for duplication are reduced, the transparency of methods and processes are promoted, and peer review of the planned methods can occur¹⁰.

Context

This study has highlighted that the methodological quality of the reviews published in the JAOS during 2003 to 2009 leaves the way open for improvements in either the reporting of systematic reviews, their conduct or both. This is important if the raised awareness of the concept of evidence-based dentistry within Brazil leads to a continuing increase in the number of systematic reviews being conducted and published by Brazilian researchers not only regionally but also internationally.

Within these reviews, an assessment of the quality of included studies for risk of bias can help to identify flaws and errors in those studies, and the reporting of these assessments can help users of the review to decide for themselves about the weight to give to individual studies and the review as a whole. These biases within included studies can operate in a variety of ways on the results of a study, which can lead to over or underestimation of the true difference between the interventions being compared and potentially a distortion of the overall estimate for the review if data from those studies are synthesized in a meta-analysis. Inappropriate synthesis of data from studies with wide clinical diversity is seen in the literature¹³ and this was exemplified in one of the reviews assessed in this study in which data from different interventions was pooled to provide a summary statistic. Before meta-analysis is considered a thorough appraisal of possible sources of clinical heterogeneity should be undertaken.

It was not the objective of our study to compare previous publications of methodological quality assessment of systematic reviews. A systematic review of this topic, including relevant oral healthcare publications has already been undertaken¹⁹.

A number of resources are available to help authors to conduct and report high quality well-designed, transparent and reproducible systematic reviews. One such resource, that is required by several journals, and provides a minimum set of items for reporting in systematic reviews and meta-analyses is PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses)⁹. The conduct of a review is considered to be more important than its reporting quality, and therefore aids which help researchers to do high quality reviews should be welcomed.

Implications for research and publication policy

Tools, such as AMSTAR, exist to assess the quality of reports of systematic reviews, and their conduct if sufficient information is reported in the review. The use of robust methods in the review process will help to ensure that the findings are reliable and that the conclusions drawn can be applied with a degree of confidence. We believe that an AMSTAR assessment of systematic reviews could be used as part of the peer reviewing process for systematic reviews, with decisions on acceptance for publication being influenced by whether or not the essential domains for quality have been adequately satisfied.

Busy clinicians may believe they do not have the time to assess the quality of published systematic reviews. We disagree with this premise and affirm that they should make strident attempts to critically appraise all relevant research when making decisions about patient care. Healthcare providers need reliable information about the beneficial and harmful effects of healthcare interventions if they are to provide the highest quality care to their patients. Therefore it is crucial that published reviews are of high quality and are well-designed, ensuring that ultimately they provide a balanced and impartial summary of the results reported by taking into account any inconsistencies in the totality of the evidence. A quality assessment using AMSTAR may guide this critical appraisal.

We suggest that the use of a tool such as AMSTAR in the pre-publication and editorial appraisal of systematic reviews would help to ensure that those that are published are of high quality and are reported clearly, completely and in such a way that this quality is readily apparent.

CONCLUSION

An AMSTAR evaluation of the methodological quality of the four systematic reviews published in JAOS from 2003 to 2009 showed that there was room for improvement in their reports and, possibly, their conduct.

Disclaimer

Although some of the authors are members of The Cochrane Collaboration, funded by the National Institute of Health Research (NIHR) or employed by the Universities of Manchester or Bern, the views expressed in this paper are not necessarily the views or the official policy of The Cochrane Collaboration, The NIHR, The University of Manchester or the University of Bern.