Reporting quality of systematic reviews of interventions aimed at improving vaccination coverage: compliance with PRISMA guidelines

ABSTRACT

Systematic reviews have become increasingly important for informing clinical practice and policy; however, little is known about the reporting characteristics and quality of SRs of interventions to improve immunization coverage in different settings. The aim of this study was to assess the reporting quality of systematic reviews of interventions aimed at improving vaccination coverage using the recommended Preferred Reporting Items for Systematic Reviews and Meta-analysis (PRISMA) guideline.

PubMed and Cochrane Library were searched to identify SRs of interventions to improve immunization coverage, indexed up to May 2016. Two authors independently screened the search output, assessed study eligibility, and extracted data from eligible SRs using a 27-item data collection form derived from PRISMA. Discrepancies in reviews assessments were resolved by discussion and consensus.

A total of 57 reviews were included in this study with a mean percentage of applicable PRISMA items that were met across all studies of 66% (range 19–100%) and median compliance of 70%. 39 out of the 57 reviews were published after the release of the PRISMA statement in 2009. Highest compliance was observed in items related to the “description of rational”, “description of eligibility criteria”, “synthesis of results” and “provision of a general interpretation of the results” (items #3, #6, #14 and #26, respectively). Compliance was poorest in the items “describing summary of evidence” (item 24, 19%), “describing indication of review protocol and registration” (item 5, 26%) and “describing results of risk of bias across studies (item 22, 33%).

The overall reporting quality of systematic reviews of interventions to improve vaccination coverage requires significant improvement. There remains a need for additional research targeted at addressing potential barriers to compliance and strategies to improve compliance with PRISMA guideline.

Introduction

Systematic reviews and meta-analyses have the highest possible level of evidence in medical literature by combining the results of a number of trials¹. Poorly conducted primary trials, even randomized controlled trials (RCTs), could lead to the introduction of bias which may decrease the usefulness of systematic reviews and meta-analyses.² It is crucial to value findings in systematic reviews and meta-analyses because clinicians use the results directly in clinical practice guidelines and protocols.³ Granting agencies may require a systematic review to ensure there is justification for further research (http://www.cihr-irsc.gc.ca/e/39187.html). As with all research, the value of a systematic review depends on the methodology, outcome, and the clarity of reporting. As with other publications, the reporting quality of systematic reviews differs in quality and appraisal, limiting readers’ ability to assess the strength and weaknesses of those reviews and ultimately reducing its relevance to patient population and clinical relevance.⁴

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement was published in 2009 and consists of a 27-item checklist that covers each section of an article and includes a flow diagram. Its purpose is to help authors improve the quality of reporting in systematic reviews and meta-analyses.⁵ Similar studies that have used the PRISMA statement in evaluating systematic reviews reporting quality have been reported in different fields, including orthorhino-laryngology,² orthodontics,⁶ radiology,⁷ gastroenterology and hepatology,⁸ orthopedics,⁹ plastic surgery, and ophthalmology.¹⁰ Irrespective of the field of research, poorly reported systematic reviews can be misleading and potentially dangerous in clinical practice and policy implementation.

Studies in vaccinology may present additional challenges to both researchers and readers because of the need for complexity in study designs. For example, interventions to improve immunization are multifaceted coupled with the dozens of vaccines available for different groups and settings may require alternative designs and methods of analysis. As such, the results and conclusions of such studies may be confusing if they do not provide an appropriate level of explanation. Systematic reviews play a critical role in enabling the accurate appraisal of the literature because they allow large pools of data to be simultaneously interpreted, which can resolve some of the complexity in this field of vaccinology. Adequate reporting of the findings of systematic reviews and meta-analyses using an objective index such as the PRISMA statement is thus essential for clinical practice and patient care.¹¹ Inherently, authors of these studies are required to provide complete, clear and transparent information by using good reporting methods.^5,12

To our knowledge, the quality of reporting of systematic reviews and meta-analyses has not been assessed in intervention reviews aimed at improving vaccination coverage. Therefore, our objective was to assess the quality of reporting in systematic reviews of interventions aimed at improving vaccination coverage.

Results

The search yielded a total of 2322 records to be screened and of these, 2247 were considered ineligible and therefore excluded. Subsequently, full texts of the remaining 75 studies were retrieved, 6 of which were protocols, 11 Cochrane reviews, 55 non-Cochrane reviews and 3 overviews; 57 systematic reviews met our predefined inclusion criteria (Figure 1).

Figure 1.

PRISMA selection of eligible studies flow diagram.

Among the 57 included reviews, 11 (19.3%) were Cochrane reviews and 46 were non-Cochrane reviews. 18 reviews were published before the release of the PRISMA statement in 2009 while 39 were published post PRISMA release. Among the 57 included reviews, 11 (19%) were updates. The corresponding authors of the systematic reviews were from the following countries: UK (12 reviews, 21%),^13–24 Canada (11 reviews, 19%),^25–33 Norway (3 reviews, 5%),^34–36 Greece (1 review, 2%),³⁷ Australia (2 reviews, 4%),^38–40 Switzerland (2 reviews, 4%),^41,42 USA (19 reviews, 33%),^32,43–60 Thailand (1 review, 2%),⁶¹ China (2 reviews, 4%),^62,63 Italy (2 reviews, 4%),^64,65 India (1 review, 2%)⁶⁶ and Nigeria (1 review, 2%)⁶⁷ as shown in Table 2. A greater number of systematic reviews had impact factor ≤5 (36, 63.2%) (Table 2). Most of the systematic reviews of interventions to improve immunization coverage in this study focused on multifaceted interventions (34, 60%) compared to single recipient oriented (10, 18%) or single provider oriented (12, 21%) intervention.

Table 2.

PRISMA score based on the journal type, impact factor, years since publication, country of corresponding author, number of authors, type of immunization intervention.

Category	Characteristic	N (%)
Type of journal	Vaccine specialty	11 (19%)
	General	37 (65%)
	Other specialties	8 (14%)
Journal impact factor by review	0–2	6 (11%)
	2.1–5	31 (55%)
	5.1–10	18 (32%)
	>10	1 (2%)
No. of authors	1	1 (2%)
	2–3	19 (33%)
	4–6	18 (32%)
	>7	19 (33%)
Country of corresponding author	United States	19 (33%)
	Canada	11 (19%)
	United Kingdom	12 (21%)
	Australia	2 (4%)
	Switzerland	2 (4%)
	Norway	3 (5%)
	Italy	2 (4%)
	India	1 (2%)
	Thailand	1 (2%)
	Nigeria	1 (2%)
	Greece	1 (2%)
	China	2 (4%)
Year since Publication	≥1	21 (37%)
	2–3	10 (18%)
	4–6	06 (11%)
	7–9	04 (07%)
	≥10	16 (28%)
Type of immunization intervention	RO	10 (18%)
	PO	12 (21%)
	RO+PO	14 (25%)
	RO+PO+HSO	16 (28%)
	PO+HSO	2 (4%)
	RO+HSO	2 (4%)
Update of a previous review		11 (19%)
N.B: NR, not reported; RO, recipient oriented; PO, provider oriented; HSO, health system oriented.

This study examined the quality of reporting of systematic reviews of interventions to improve vaccination coverage was assessed using the PRISMA statement. The mean percentage of applicable PRISMA items that were met across all studies was 66% (range 19–100%) and median compliance of 70%. 39 out of the 57 reviews were published after the release of the PRISMA statement in 2009. Highest compliance (100%) was observed in items related to the “description of rational”, “description of eligibility criteria”, “synthesis of results” and “provision of a general interpretation of the results” (items #3, #6, #14 and #26, respectively). Compliance was poorest in the items “describing summary of evidence” (item 24, 19%), “describing indication of review protocol and registration” (item 5, 26%) and “describing results risk of bias across studies (item 22, 33%) (Table 3).

Table 3.

Compliance with PRISMA checklist items (adapted from Moher et al.).

Section/Topic	No.	Brief description of the item	Compliance
TITLE
Title	1	Identification of the report	53 (93%)
ABSTRACT
Structured summary	2	Provide a structured summary	46 (81%)
INTRODUCTION
Rationale	3	Background rationale	57 (100%)
Objectives	4	Description of PICOS (Participants, Interventions, Comparisons, and Study design)	26 (46%)
METHODS
Protocol and registration	5	Indication of review protocol & registration information	15 (26%)
Eligibility criteria	6	Specification of study and review characteristics used as eligibility criteria	57 (100%)
Information sources	7	Describe all information sources and date last searched	55 (96%)
Search	8	Present repeatable full electronic search strategy for at least one database	44 (77%)
Study selection	9	State the process for selecting studies	39 (68%)
Data collection process	10	Describe the method of data extraction	39 (68%)
Data items	11	Report all variables and any assumptions and simplifications made	50 (88%)
Risk of bias in individual studies	12	Describe the methods used to assess the risk of bias of individual studies	30 (53%)
Summary measures	13	State the principal summary measures	50 (88%)
Synthesis of results	14	Describe the methods used to handle and analyse the data	57 (100%)
Risk of bias across studies	15	Specify any assessment of risk of bias that may affect the cumulative evidence	24 (42%)
Additional analyses	16	Describe methods of additional analyses	20 (35%)
RESULTS
Study selection	17	Give numbers of studies at each stage of the study	50 (88%)
Study characteristics	18	For each study, present characteristics for which data were extracted	43 (75%)
Risk of bias within studies	19	Present data on risk of bias for each study	34 (60%)
Results of individual studies	20	Report the summary of each data intervention group and estimates of confidence intervals	50 (88%)
Synthesis of results	21	Present the results of each meta-analysis	21 (37%)
Risk of bias across studies	22	Present the results of any assessment of risk of bias across studies	19 (33%)
Additional analysis	23	Give the results of additional analyses	20 (35%)
DISCUSSION
Summary of evidence	24	Summarize the main findings, including the strength of the evidence	11 (19%)
Limitations	25	Discuss limitations at the study, outcome & review levels	48 (84%)
Conclusions	26	Provide a general interpretation of the results	57 (100%)
FUNDING
Funding	27	Describe sources of funding & other support	42 (74%)
OVERALL ADHERENCE 18(67%)

In assessing the reporting quality of PRISMA items under the method section, our analysis shows that only three items (item 5, item 15 and item 16) had compliance less than 50% across all studies. These items were “protocol and registration” (15 reviews, 26%), “description of risk of bias across studies” (19 reviews, 33%), and “additional analysis” (20 reviews, 35%). This shows that most systematic reviews of intervention to improve immunization coverage have good reporting quality on methods. While in the results section, applicable PRISMA items that met across all studies with ≤50% compliance were four against three with compliance above 50% with the highest compliance observed in item 17 “describing the study selection criteria”, and in item 20 “describing results of individual studies” (50, 88% each), followed by item 18 “describing the study characteristic” (43, 75%). The summary of evidence item was the poorest PRISMA item that met across all studies with only 11 (19%) studies meeting this criterion. Overall, 18 out of 57 reviews had PRISMA items adherence of 50% and above (Table 3).

After selecting the three categories, i.e., Cochrane versus non-Cochrane, high impact versus low impact factor journals and reviews published before and after 2009, PRIMSA reporting quality was appraised. Amongst the Cochrane reviews, 10 reported that a protocol was available, two reported including papers published in English only, while nine considered all languages. In addition, nine made available the search strategy. Eight reviews reported that two authors screened the papers independently while all the 11 reported that data were extracted independently by two reviewers. Moreover, all these reviews assessed risk of bias in their included studies and five stated conducting meta-analysis while eight did subgroup analysis. Eight papers wrote the summary of findings; three wrote limitations in both the study and review level and 10 reported their sources of funding.

We also observed 17 reviews published in high impact factor journals, and these included the 11 Cochrane reviews. Among the six non-Cochrane reviews published in high impact factor journals, two reported using a protocol, one reported including papers published in English only and one provided a search strategy. In three of the non-Cochrane reviews, two authors screened papers and extracted data independently. Three assessed risk of bias in included studies, none conducted meta-analysis, additional analysis, and summary of evidence. Moreover, four wrote about limitations in both study and review level.

Regarding the 18 systematic reviews (which included two Cochrane reviews and two reviews published in high impact journals) published before PRISMA was released, four reported the use of a protocol, seven included papers published in English only and 15 provided their search strategy. Seven reviews reported having two authors to screen and extract data independently. Concerning analysis, 12 and two reported conducting meta-analysis and subgroup analysis, respectively. Only one of the studies included a summary of evidence, 10 mentioned limitations in both the study and review level and 14 reported their funding source.

An assessment of the descriptive characteristics of the search process in the included systematic reviews showed that most of the reviews searched mainly the MEDLINE/PubMed database (52, 91%), followed by EMBASE (34, 60%), Cochrane Library (32, 56%) and CINAHL (30, 53%) databases. It is important to note that in many reviews, authors also reported searching other databases (Table 4).

Table 4.

Descriptive characteristics of search process in systematic reviews.

Source	Characteristic	N (%)
Databases	Number of databases, median
	MEDLINE/PubMed	52 (91%)
	EMBASE	34 (60%)
	Cochrane Library	32 (56%)
	CINAHL	30 (53%)
	Scopus	8 (14%)
	Web of Science (ISI)	16 (28%)
	PsycINFO	15 (26%)
	Other database	50 (88%)
	No database reported	3 (5%)
Other sources	Reference lists reviewed	46 (81%)
	Hand searching journals	9 (16%)
	Experts or corresponding authors	20 (35%)
	Conference proceedings/abstracts	7 (12%)
	Other	3 (5%)

Discussion

PRISMA statement is a tool to enhance full disclosure of valuable steps in reporting systematic reviews and meta-analyses thereby reducing reporting bias and ultimately permitting more accurate critical appraisal and more balanced and discerning clinical decision-making and policy implementation. The number of systematic reviews and meta-analyses published in the field of vaccines uptake has dramatically increased over the last few years as seen in the number of systematic reviews of interventions included in this study. Even though large data sets could be a positive indication for recognition of increased high-quality articles published in this field, this could also translate to increased room for error. This will, therefore, require that increased importance be placed on accurate and detailed reporting. Clear and transparent information not only allows an adequate interpretation of results but also helps authors to correct errors during earlier stages of the study.^5,12

We conducted a study to assess the reporting quality of systematic reviews of interventions to improve vaccination coverage using the 27 items PRISMA checklist. Our review showed that the mean percentage of applicable PRISMA items that were met across all studies was 66% (range 19–100%) and median compliance of 70%. This compliance is relatively low in comparison with most previous studies that investigated the reporting quality of systematic reviews in different specialties^4,7–9,68 but higher than that observed in the field of plastic surgery, ophthalmology, and nursing.^10,69 All the Cochrane reviews met all 27 criteria of the PRISMA statement. Only a few of the non-Cochrane reviews reported working from a protocol provided a search strategy and assessed risk of bias in included studies. The reporting quality of these critical PRISMA checklist items among the non-Cochrane reviews was poor compared to Cochrane reviews thereby requiring improvement in reporting these intervention reviews in vaccination. Without a protocol that is publicly available, it is difficult to judge between proper and improper changes in the review¹. Among the 18 systematic reviews published before the release of the PRISMA checklist, we observed that only four reported working from a protocol, 15 provided a search strategy and 12 assessed risk of bias. We observed that six items with compliance less than or equal to 35% include items related to “describing the review protocol and registration”, “describing the risk of bias across studies”, “describing the methods of additional analyses”, “describing reporting additional analysis”, “reporting the risk of bias across studies” and “reporting summary of evidence” (items 5, 15, 16, 22, 23, and 24, respectively). Similar poor compliance with these items has also been observed in other previous studies that assessed the reporting quality of systematic reviews in different specialties.^2,6–10,69

It was also common to find that, irrespective of their overall compliance with the PRISMA checklist, previous studies in specialties including otorhinolaryngology,² orthodontics,⁶ and radiology⁷ all achieved compliance lower than 30% in items relating to describing the review protocol and registration and the risk of bias. A generally high median compliance above 80%, with the PRISMA checklist but poor compliance less than 20%, in describing the “protocol and registration” was observed in a study of journals in gastroenterology and hepatology.⁸ Similar studies that evaluated orthopedic journals,⁶ compliance of plastic surgery articles to PRISMA statement,¹⁰ and a study of nursing journals⁶⁹ produced overlapping results that showed there was low compliance regarding items related to the “protocol and registration”, “the description of bias”, and the “description of additional analyses”. The generally low compliance observed in this study is an indication that there is a lack of awareness regarding the importance of review protocols and registration information among authors. The use of protocol and review registration is important because a protocol allows more accurate comparisons to be made between authors and readers and supports better transparency during the review process.⁵ Surprisingly, most studies generally achieve a low disappointing level of compliance in items related to the assessment of risk of bias. The attempt to eliminate bias is a very fundamental component of randomized controlled trials, as is the assessment of bias in systematic reviews and meta-analysis (https://methods.cochrane.org/bias/assessing-risk-bias-included-studies, assessed 02 October 2018). We suspect that this is caused by a lack of awareness regarding the essential points of research reporting among authors.

Generally, the reviews included in this review had varied literature search sources with only three (5%) reviews that did not report the data sources. PRISMA checklist item 7 states that systematic reviews should “describe all information sources in the search”.⁵ Systematic reviews require a comprehensive search for published and unpublished studies to minimize bias. Failure to search multiple databases can increase the risk that relevant studies will be missed, which could bias the outcome of the review.^70,71 Interestingly, most of the reviews included in this study searched more than one database thus giving more confidence to the readers of reviews of interventions in vaccination.

While improved awareness among stakeholders (i.e., journal reviewers and editors, funders, institutions, and readers) is required, there remains the glaring need for journals to make the PRISMA checklist mandatory for the electronic submission of systematic reviews.¹⁰ The enforcement of these guidelines at the time of journal submission might improve the quality of reporting, and as their endorsement has been shown to improve adherence.^8,72

PRISMA was designed to cover a wide range of specialties which might not necessarily address specialty specifics. Also, the weighting of items does not take into consideration the weight it gives to the reporting quality of systematic reviews. In this study, we assigned equal weights to all items in the PRISMA checklist and used the same number of denominators in each article. However, this imposed a limitation on our study in that certain items on the checklist may have more impact than others on the reporting quality of systematic reviews. Failure to declare some of the PRISMA items does not necessarily mean they were undertaken but may be due to the publishing style of the journals and some reviews may have lost a mark due to this type of limitation. Another limitation of our study is that there were insufficient numbers of comparable Cochrane versus non-Cochrane reviews, reviews published before versus those published after the PRISMA statement was released and high impact factor versus low impact factor reviews. Therefore, caution should be taken as to the relevance of the quality assessment as accumulation of sufficient comparable numbers of reviews will likely require so much elapse time that may raise debate on the relevance of this assessment.

The overall reporting quality of systematic reviews of interventions to improve vaccination coverage requires significant improvement. The PRISMA tool is a very important tool in assessing the reporting quality of systematic reviews of interventions in vaccination. There remains a need for additional research targeted at addressing potential barriers to compliance and strategies to improve compliance with PRISMA guideline.

Methods

A comprehensive literature search was conducted in 2016 in the Cochrane Database of Systematic Reviews (CDSR), the Database of Abstracts of Reviews of Effects (DARE) and PubMed to identify peer-reviewed systematic reviews. The search strategy was built using a combination of keywords including vaccination, immunization, vaccine, uptake, coverage, and rate (Table 1).

Table 1.

PubMed, and cochrane library search strategies.

Search	Quarry
	PubMed
#1	Search immunization schedule[mh]
#2	Search (immunisation coverage[tiab] OR immunisation rate[tiab] OR immunisation uptake [tiab] OR immunization coverage[tiab OR immunization rate[tiab] OR immunization uptake [tiab] OR vaccination coverage[tiab] OR vaccination rate[tiab] OR vaccination uptake [tiab] OR vaccine coverage [tiab] OR vaccine uptake [tiab])
#3	Search (#1 OR #2)
#4	Search (meta-analysis[mh] OR meta analysis[pt] OR meta-review[ti] OR meta-analysis[tiab] OR meta-analysis [ti] OR metaanalysis [ti] OR meta-analyses [tiab])
#5	Search (review[tiab] OR systematic review[tiab]OR review[pt] OR systematic reviews[pt] OR literature review[tiab])
#6	Search (#4 OR #5)
#7	Search (#3 AND #6)
	Cochrane Library
#1	MeSH descriptor: [Immunization] explode all trees
#2	MeSH descriptor: [Immunization Schedule] explode all trees
#3	MeSH descriptor: [Immunization, Secondary] explode all trees
#4	MeSH descriptor: [Immunization Programs] explode all trees
#5	MeSH descriptor: [Vaccination] explode all trees
#6	immunisation coverage OR immunisation rate OR immunisation uptake OR immunization coverage OR immunization rate OR immunization uptake OR vaccination coverage OR vaccination rate OR vaccination uptake OR vaccine coverage OR vaccine uptake:ti,ab,kw
#7	#1 or #2 or #3 or #4 or #5 or #6

Two authors (VNN and AJ) independently screened titles and abstracts of systematic reviews for potential eligibility. Following this, full texts of potentially eligible reviews were retrieved. The authors (VNN and AJ) independently applied eligibility criteria to identify relevant reviews to be included in the study. We excluded reviews in which interventions to improve immunization coverage was not explicitly stated in the title or abstract. Where there were uncertainties, full-text articles were examined for further information. Disagreements regarding review selection were resolved through discussion and consensus, failing which a third author (CSW) arbitrated.

Two authors (VNN and AJ) independently extracted data such as participants, setting, interventions, outcomes, results, publications, and journal of publication. Also, data to assess the reporting quality of the eligible systematic reviews were independently extracted by two authors using the PRISMA checklist form. Discrepancies were discussed and resolved by agreement. Analysis of data was done using Microsoft excel. Continuous variables were presented as mean and range while categorical variables were presented as percentage values. PRISMA items adequately reported for all studies and their percentage compliant with each item of the PRISMA guidelines were analyzed.

Assessment of the reporting quality of the systematic reviews was done independently by two authors (VNN and AJ) for compliance against each item in the 27 items PRISMA checklist. Additionally, compliance was also assessed particularly for the data sources, journal type, impact factor, years since publication, country of corresponding author, number of authors, and type of immunization intervention. To assess the degree of compliance, every item was rated as “yes” for compliance, or ”no” for non-compliance. Differences in opinion were discussed until consensus was reached.

Three categories from the reviews were selected to assess their quality of reporting. These categories included Cochrane versus non-Cochrane reviews, journals of high (impact factor ≥ 5) versus low (impact factor <5) impact factor and reviews published before 2009 (i.e., reviews published before the PRISMA tool was released) versus post-2009 (i.e., reviews published after the PRISMA tool was released).

Funding Statement

The Dean of the Faculty of Medicine and Health Sciences strategic research Fund, Stellenbosch University, South Africa

Acknowledgments

We greatly appreciate the support given to us by Prof Taryn Young (Director of the Centre for Evidence Based Health Care) and the team at the Centre for Evidence Based Health Care, Faculty of Medicine and Health Sciences, Stellenbosch University, Cape Town South Africa.

Disclosure of potential conflicts of interest

No potential conflicts of interest were disclosed.

Authors contributions

VNN and CSW conceived and designed the study, VNN and AJ screened reviews, extracted data, and data analysis. VNN and AJ drafted the manuscript, and all authors contributed in the write-up and final approval of the manuscript for submission.