Assessing the methodological quality of systematic reviews of interventions aimed at improving vaccination coverage using AMSTAR and ROBIS checklists

ABSTRACT

Introduction: Systematic reviews (SRs) are the backbone of evidence-based health care, but no gold standard exists to assess their methodological quality. Although the AMSTAR tool is accepted for analyzing the quality of SRs, the ROBIS instrument was recently developed. This study compared the capacity of both instruments to capture the quality of SRs of interventions for improving vaccination coverage.

Methods: We conducted a comprehensive literature search in the Cochrane Library and PubMed. Two reviewers independently screened the search output, assessed study eligibility, and extracted data from eligible SRs; resolving differences through consensus. We conducted Principal Component Analysis (PCA) in Stata 14 to determine similarities and differences between AMSTAR and ROBIS.

Results: A total of 2322 records were identified through the search and 75 full-text publications were assessed for eligibility, of which 57 met inclusion criteria. Using AMSTAR, we found 32%, 60% and 9% of SRs to have high, moderate and low quality, respectively. With ROBIS, we judged 74%, 14% and 12% of SRs to have low, unclear and high risk of bias. PCA showed that SRs with low risk of bias in ROBIS clustered together with SRs having high-quality in AMSTAR, and SRs with high risk of bias in ROBIS clustered with low-quality SRs in AMSTAR.

Conclusions: Our findings suggest that there is an association between methodological quality and risk of bias in SRs of interventions focused on improving vaccination coverage. Therefore, either AMSTAR or ROBIS checklists can be used to evaluate methodological quality of SRs in vaccinology.

Background

Vaccination is a very powerful public health tool for improving human survival from infectious diseases^1-3 The use of vaccination as a public health strategy began when the World Health Organization (WHO) launched the Expanded Programme on Immunization (EPI) in 1974.⁴ When the EPI was launched, the WHO recommended a standard vaccination schedule covering six basic vaccines i.e., tuberculosis (Bacille Calmette-Guerin (BCG)), polio, diphtheria, tetanus, pertussis, and measles.⁵ Vaccination has the potential to increase uptake and coverage of newly available vaccines in EPI programs of low and middle-income countries.^1,6-8 However, the achievement so far is described as “fragile” as judged by outbreaks of some of these diseases in low, middle and high-income countries.⁷ The outbreaks reflect the existence of communities with partially vaccinated or unvaccinated children.⁷

A systematic review (SR), which is a type of study designed to synthesize available evidence can be used to answer questions about the effects of interventions used to improve the coverage rates of different vaccines.⁹ Therefore, the quality of SRs has to be assessed before they can be used to make any clinical recommendations. Various assessment tools have been developed to assess the methodological quality of SRs, with the most commonly used ones being the “Assessing the Methodological Quality of Systematic Reviews” (AMSTAR) and “Risk Of Bias In Systematic reviews” (ROBIS) checklists.^10,11 AMSTAR consists of 11 items (Appendix I, Table A1) used as a checklist added up into an overall score while ROBIS which is used to evaluate risk of bias in SRs is based on three phases.^10,11 Phase one is optional, phase two covers four domains (i.e., study eligibility criteria, identification and selection of studies, data collection and study appraisal and synthesis and findings) through which bias may be introduced in an SR and phase three, which has four domains (which summarizes the concerns identified during the phase two assessment), assesses the overall risk of bias (Appendix I, Table A2). AMSTAR reports the level of methodological quality as low, moderate and high while ROBIS reports SRs as having low or high risk of bias.¹⁰ Therefore, the aim of this study was to compare the methodological quality of systematic reviews of interventions for improving vaccination coverage, using AMSTAR and ROBIS tools.

Table A1.

AMSTAR checklist.

No	Review question	Y	N	Unc	NI	Page No.
	Y = Yes; N = No; Unc = Unclear; NI = No information NB: add comment below row when response is unclear
1	Was an ‘a priori’ design for eligible studies provided? The research question and inclusion criteria should be established before the conduct of the review.
2	Was there duplicate study selection and data extraction? There should be at least two independent data extractors and a consensus procedure for disagreements should be in place.
3	Was a comprehensive literature search performed? At least two electronic sources should be searched. The report must include years and databases used (e.g. Central, EMBASE, and MEDLINE). Key words and/or MESH terms must be stated and where feasible the search strategy should be provided. All searches should be supplemented by consulting current contents, reviews, textbooks, specialized registers, or experts in the particular field of study, and by reviewing the references in the studies found.
4	Was the status of publication (i.e. grey literature) used as an inclusion criterion? The authors should state that they searched for reports regardless of their publication type. The authors should state whether or not they excluded any reports (from the systematic review), based on their publication status, language etc.
5	Was a list of studies (included and excluded) provided? A list of included and excluded studies should be provided.
6	Were the characteristics of the included studies provided? In an aggregated form such as a table, data from the original studies should be provided on the participants, interventions and outcomes. The ranges of characteristics in all the studies analyzed e.g. age, race, sex, relevant socioeconomic data, disease status, duration, severity, or other diseases should be reported.
7	Was the scientific quality of the included studies assessed and documented? ‘A priori’ methods of assessment should be provided (e.g., for effectiveness studies if the author(s) chose to include only randomized, double-blind, placebo controlled studies, or allocation concealment as inclusion criteria); for other types of studies alternative items will be relevant.
8	Was the scientific quality of the included studies used appropriately in formulating conclusions? The results of the methodological rigor and scientific quality should be considered in the analysis and the conclusions of the review, and explicitly stated in formulating recommendations.
9	Were the methods used to combine the findings of studies appropriate? For the pooled results, a test should be done to ensure the studies were combinable, to assess their homogeneity (i.e. Chi-squared test for homogeneity, I-squared statistic). If heterogeneity exists a random effects model should be used and/or the clinical appropriateness of combining should be taken into consideration (i.e. is it sensible to combine?).
10	Was the likelihood of publication bias assessed? An assessment of publication bias should include a combination of graphical aids (e.g., funnel plot, other available tests) and/or statistical tests (e.g., Egger regression test).
11	Was the conflict of interest stated? Potential sources of support should be clearly acknowledged in both the systematic review and the included studies.
Total “Yes” score

Table A2.

Quality assessment using ROBIS tool R.

Phase 2: Identifying concerns with the review process
Domain 1: Study Eligibility Criteria Describe the study eligibility criteria, any restrictions on eligibility and whether there was evidence that objectives and eligibility criteria were pre-specified. Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
1.1 Did the review adhere to pre-defined objectives and eligibility criteria?
1.2 Were the eligibility criteria appropriate for the review question?
1.3 Were eligibility criteria unambiguous?
1.4 Were all restrictions in eligibility criteria based on study characteristics appropriate (e.g. date, sample size, study quality, outcomes measured)?
1.5 Were any restrictions in eligibility criteria based on sources of information appropriate (e.g. publication status or format, language, availability of data)?
Concerns regarding specification of study eligibility criteria		LOW
		HIGH
		UNCLEAR
Rationale for concern: study eligibility appropriate
Domain 2: Identification and selection of studies Describe methods of study identification and selection (e.g. number of reviewers involved). Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
2.1 Did the search include an appropriate range of databases/electronic sources for published and unpublished reports?
2.2 Were methods additional to database searching used to identify relevant reports?
2.3 Were the terms and structure of the search strategy likely to retrieve as many eligible studies as possible?
2.4 Were restrictions based on date, publication format, or language appropriate?
2.5 Were efforts made to minimize error in selection of studies?
Concerns regarding methods used to identify and/or select studies		LOW
		HIGH
		UNCLEAR
Rationale for concern: Identification and selection of studies appropriate
Domain 3: Data collection and study appraisal Describe methods of data collection, what data were extracted from studies or collected through other means, how risk of bias was assessed (e.g. number of reviewers involved) and the tool used to assess risk of bias. Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
3.1 Were efforts made to minimize error in data collection?
3.2 Were sufficient study characteristics available for both review authors and readers to be able to interpret the results?
3.3 Were all relevant study results collected for use in the synthesis?
3.4 Was risk of bias (or methodological quality) formally assessed using appropriate criteria?
3.5 Were efforts made to minimize error in risk of bias assessment?
Concerns regarding methods used to collect data and appraise studies		LOW
		HIGH
		UNCLEAR
Rationale for concern: Data collection and study appraisal good
Domain 4: Synthesis and findings. Describe synthesis methods Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
4.1 Did the synthesis include all studies that it should?
4.2 Were all pre-defined analyses reported or departures explained?
4.3 Was the synthesis appropriate given the nature and similarity in the research questions, study designs and outcomes across included studies?
4.4 Was between-study variation (heterogeneity) minimal or addressed in the synthesis?
4.5 Were the findings robust, e.g. as demonstrated through funnel plot or sensitivity analyses?
4.6 Were biases in primary studies minimal or addressed in the synthesis?
Concerns regarding the synthesis and findings		LOW
		HIGH
		UNCLEAR
Rationale for concern: Synthesis and findings good
Phase 3: Judging Risk of Bias
Summarize the concerns identified during the Phase 2 assessment:
Domain	Concern	Rationale for concern
1. Concerns regarding specification of study eligibility criteria
2. Concerns regarding methods used to identify and/or select studies
3. Concerns regarding used to collect data and appraise studies
4. Concerns regarding the synthesis and findings
RISK OF BIAS IN THE REVIEW
Describe whether conclusions were supported by the evidence Y = Yes, PY = Probably Yes, PN = Probably No, N = No, NI = No Information		Y	N	PY	PN	NI	Page No
A. Did the interpretation of findings address all of the concerns identified in Domains 1 to 4?
B. Was the relevance of identified studies to the review’s research question appropriately considered?
C. Did the reviewers avoid emphasizing results on the basis of their statistical significance?
Risk of bias in the review		LOW
		HIGH
		UNCLEAR

Results

Search results

The literature search yielded a total number of 2322 of records of which 75 full-text publications were retrieved and assessed for eligibility. After full-text assessment, 57 were considered eligible for inclusion. The search and selection of reviews are shown in Figure 1.

Figure 1.

PRISMA search flow diagram.

General characteristics

Among the 57 included reviews, 11(19.3%) were Cochrane reviews. The first authors of the SRs were from the following countries: UK (10 reviews),^12-21 Canada (9 reviews),^6,22-29 Norway (3 reviews),^30-32 Greece (1 review),³³ Australia (3 reviews),^34-36 Switzerland (2 reviews),^37,38 USA (9 reviews),^38-46 Thailand (1 review),⁴⁷ Hong Kong (1 review),⁴⁸ China (1 review)⁴⁹ and Nigeria (1 review).⁵⁰

Results using AMSTAR instrument

Using the AMSTAR tool, we judged 18(32%) reviews as being of high methodological quality, 34(60%) as moderate quality, and 5(9%) as low quality. Cochrane reviews were all high quality (total scores ranged from 9 to 11) while the quality of non-Cochrane reviews ranged from low to high quality (scores from 3 to 10).

Results using ROBIS instrument

Using the ROBIS tool, we classified 42(74%) reviews to have low risk of bias, 8(14%) as unclear risk and 7(12%) as having high risk of bias. None of the Cochrane reviews had a high risk of bias.

Principal component analysis

We used principal component analysis (PCA) to convert vectors of the 11 AMSTAR item scores and 21 ROBIS signaling questions to two principal components (PCs), respectively. Subsequently, we used PCA to determine the relationship between AMSTAR and ROBIS. PC1 and PC2 projections where SRs of similar methodological quality and risk of bias clustered together are shown in Figures 2 and 3. There was an overlap of SRs with low risk of bias in ROBIS and high quality in AMSTAR (robis = 1/amstar = 3), high risk of bias and low quality (robis = 3/amstar = 1), and unclear risk of bias and moderate quality (robis = 2/amstar = 2) (Figures 2 and 3).

Figure 2.

AMSTAR-based PCA showing the relationship between methodological quality and risk of bias of reviews.

Figure 3.

ROBIS-based PCA showing the relationship between methodological quality and risk of bias of reviews.

The AMSTAR-based PCA results showed that in the first component, the 5th AMSTAR item “Was a list of studies (included and excluded) provided?” explained most of the variability (40%) while the 8th AMSTAR item “Was the scientific quality of the included studies used appropriately in formulating conclusions?” explained most variability (17%) in the second component (Figure 4, Appendix II). Concerning the ROBIS-based PCA, data showed that in the first component, question 21 “Were biases in primary studies minimal or addressed in the synthesis?” contributed more than 48% variability while the second component showed that item 20 “Were the findings robust, e.g., as demonstrated through funnel plot or sensitivity analyses?” contributed 26% of variability in the data (Figure 5, Appendix I Appendix III).

Figure 4.

Scree plot of AMSTAR-based PCA showing total variance in the data as explained by each principal component.

Figure 5.

Scree plot of ROBIS-based PCA showing total variance in the data as explained by each principal component.

Discussion

The two validated tools were used for the assessment of methodological quality and risk of bias of SRs in this study. AMSTAR is a tool that consists of 11-items interpreted individually or as the sum of reported items, i.e., an overall score which assesses methodological quality while ROBIS is a domain-based instrument consisting of 21 signaling questions designed to evaluate the risk of bias. There are two primary differences between AMSTAR and ROBIS. AMSTAR does not evaluate how authors collected relevant data for the review and ROBIS does not assess anything about authors’ conflict of interests and funding sources.

Our results show that most systematic reviews that assess the effects of interventions aimed at improving vaccination coverage have been conducted in the United States of America. To our knowledge, this is the first study to compare the results of the AMSTAR and ROBIS tools for assessing risk of bias and methodological quality in SRs concerning interventions to improve vaccination coverage. Our findings report that AMSTAR judged most of the SRs as having high and moderate quality while it considered only 9% of the reviews as low methodological quality. In addition, we have observed that all Cochrane reviews were of high methodological quality while the non-Cochrane reviews were of low, moderate and high quality. The current results seem to be consistent with other studies whose objectives were also to assess methodological quality of SRs using the two tools, although focusing on different fields.^51,52

These previous studies also used the AMSTAR tool to evaluate the methodological quality of SRs of health-care interventions. Furthermore, they showed that AMSTAR rated Cochrane reviews as having high methodological quality compared to non-Cochrane reviews.^51,52 Taken together, these data suggest that AMSTAR is a useful tool for assessing the quality of Cochrane and non-Cochrane SRs.

Concerning bias, most of the reviews, including all Cochrane reviews, were judged to have low risk of bias (74%) while the rest had unclear or high risk of bias (Table 2). These results suggest that Cochrane reviews of interventions are of higher quality compared to non-Cochrane reviews. This study suggests that ROBIS has fair consistency and good legitimacy to appraise risk of bias in systematic reviews.

Table 2.

ROBIS assessment of systematic reviews of interventions to improve immunization coverage.

	Phase 2				Phase 3
Review	1: Study Eligibility Criteria	2: Identification and selection of studies	3: Data collection and study appraisal	4: Synthesis and findings	Judging risk of Bias
Adams 2015
Aigbogun 2015
Arditi
Batt 2004
Bassani 2013
Bordley 2000
Briss 2000
Chamla 2015
Corace 2016
Elkan 2000
Fu 2014
Gera 2016
Giuffrida 1999
Glenton 2011
Groom 2015
Gyorkos 1994
Harvey 2015
Hollmeyer 2012
Jacob 2016
Jacobson Van 2005
Johri 2015
Kauffman 2013
Kendrick 2000
Lagarde 2007
Lagarde 2009
Lassi 2015
Lau 2012
Lee 2016
Levin 2010
Lystra 2016
Maglione 2002
Mureed 2015
Ndiaye 2005
Niccolai 2015
Odone 2015
Oyo-ita 2016
Peacock 2013
Pegurri 2005
Poorman 2015
Rashid 2016
Rayman 2008
Saeterdal 2014
Schmidt 2013
Shea 1996
Shea 2009
Smulian 2016
Stone 2002
Szilagyi 2000
Thomas 2010
Thomas 2014
Voss 2016
Walling 2016
Wang 2016
Whittaker 2002
Wigham 2014
Williams 2011
Wong 2016

Low risk, High risk, Unclear risk

Regarding principal component analysis (PCA), the AMSTAR-based PCA data showed that SRs classified as having high methodological quality by AMSTAR clustered together with those considered to have low risk of bias by ROBIS. Furthermore, SRs judged as having low methodological quality clustered together with those having high risk of bias (Figure 2). The ROBIS-based PCA also showed similar findings (Figure 3). These results suggest that there is an association between SRs with high methodological quality and those with low risk of bias. Although these tools are designed to assess two different concepts, our data imply that low risk of bias translates to high methodological quality of SRs. Our findings are different from a study conducted by,⁵³ whose PCA data showed that reviews on psoriasis interventions, classified as having high and moderate quality were considered as having high risk of bias. This previous study thus suggested that it is possible for SRs with high methodological quality to have high risk of bias.⁵³

With regard to variability of data, the fifth item of AMSTAR explained the highest variability (40%) of data which implies that this item varies significantly between systematic reviews when it comes to methodological quality. Therefore, this also suggests that the fifth item is the best to distinguish between SRs of low, moderate and high quality. Similarly, contribution of 48% variability by signaling question 21 in ROBIS infers that this item varies significantly between SRs and is also the best to differentiate between SRs of low versus high risk of bias. Our results are different from those obtained by,⁵³ who reported that two of ROBIS signaling questions, i.e., the 14^th and 15^th items contributed most variability in their data. The findings by this previous study suggested that both these items were the best to differentiate between SRs of high and low risk of bias.⁵³

Other qualitative studies, e.g., survey methodology studies have used PCA to determine the different factors that contribute to variations in the survey population structures or to changes in public health or changes in behavior. These included research that set out to identify the contribution of different categories of mothers’ characteristics (e.g., birth order, mother’s education and household wealth index) to an increase in proportion of births attended by skilled personnel;⁵⁴ contribution of maternal variables (e.g., age, education level, Medicaid insurance status, infant birth order), the social environment and physical environment to non-Hispanic black and white individuals in preterm birth;⁵⁵ variations in the prevalence of missed opportunities for vaccination (MOVs) among children of poor and non-poor mothers in sub-Saharan African countries⁵⁶ and contribution of dietary patterns to the risk of developing colorectal cancer.⁵³

Similar to the present investigation, the above-mentioned studies successfully identified the factors that mainly contributed to changes in the number of births attended by skilled health-care workers, preterm births in non-Hispanic black and white individuals, prevalence of MOVs and risk of developing colorectal cancer, using PCA. Applying PCA, the current study has added to the body of knowledge regarding identifying the AMSTAR and ROBIS items that mostly contribute to variation in the quality of systematic reviews aimed at increasing vaccination coverage.

Practical implications of the study

ROBIS is a newly developed and approved instrument for appraising risk of bias in SRs and hence has been reported to have overcome limitations of prior instruments. Although many studies have assessed the methodological quality of SRs using AMSTAR in a variety of research fields, this study suggests that SRs use both AMSTAR and ROBIS to evaluate methodological quality of reviews. In addition, this paper recommends the use of PCA to identify the AMSTAR and ROBIS items that contribute to variance in the quality of SRs. The use of PCA also identifies the AMSTAR items and ROBIS signaling questions that contribute the most to the final judgment of methodological quality and risk of bias.

Limitations of the study

This study conducted a comprehensive search to procure a sample of published systematic reviews on interventions to increase vaccination coverage. Therefore, one of the limitations of this investigation includes the fact that we did not conduct a search in gray literature databases. The latter implies that we could not assess the methodological quality in the unpublished research. Additionally, the search was conducted in June 2016, suggesting that we may have missed recently published data on systematic reviews of interventions aimed at increasing vaccination coverage. With regards to the methodological quality assessment, one significant limitation is related to the indeterminate conversion of the AMSTAR total scores into categorical rankings, e.g., low, middle and high methodological quality. This would potentially make it difficult to measure the differences in methodological quality across systematic reviews.

Methods

This manuscript reports a pre-specified sub-study of an overview of systematic reviews of interventions aimed at improving vaccination coverage, whose protocol is registered in the International Prospective Register of Systematic Reviews (PROSPERO), with registration number CRD42018090342.⁵⁷ We included peer-reviewed systematic reviews which assessed the effects of interventions to improve vaccination coverage; irrespective of the type of study designs, participants, interventions, or vaccines concerned.

A comprehensive literature search was conducted in June 2016 in the Cochrane Library and PubMed using a combination of keywords, including vaccination, immunization, vaccine, uptake, and coverage (Table 1). Reference lists of relevant publications were also screened for potentially eligible reviews. Two authors, Anelisa Jaca (AJ) and Valantine Ngum Ndze (VNN), independently screened all search records for potentially eligible reviews. Full texts of articles judged to be potentially eligible by either author were retrieved and independently assessed by the two authors for inclusion. The two authors (AJ and VNN) independently extracted data on the citation details and AMSTAR and ROBIS quality items from eligible systematic reviews. In each case, the two authors resolved discrepancies by discussion and consensus.

Table 1.

PubMed, and Cochrane Library search strategies.

Search	Quarry
	PubMed
#1	Search immunization schedule[mh]
#2	Search (immunisation coverage[tiab] OR immunisation rate[tiab] OR immunisation uptake [tiab] OR immunization coverage[tiab OR immunization rate[tiab] OR immunization uptake [tiab] OR vaccination coverage[tiab] OR vaccination rate[tiab] OR vaccination uptake [tiab] OR vaccine coverage [tiab] OR vaccine uptake [tiab])
#3	Search (#1 OR #2)
#4	Search (meta-analysis[mh] OR meta analysis[pt] OR meta-review[ti] OR meta-analysis[tiab] OR meta-analysis [ti] OR metaanalysis [ti] OR meta-analyses [tiab])
#5	Search (review[tiab] OR systematic review[tiab]OR review[pt] OR systematic reviews[pt] OR literature review[tiab])
#6	Search (#4 OR #5)
#7	Search (#3 AND #6)
	Cochrane Library
#1	MeSH descriptor: [Immunization] explode all trees
#2	MeSH descriptor: [Immunization Schedule] explode all trees
#3	MeSH descriptor: [Immunization, Secondary] explode all trees
#4	MeSH descriptor: [Immunization Programs] explode all trees
#5	MeSH descriptor: [Vaccination] explode all trees
#6	immunisation coverage OR immunisation rate OR immunisation uptake OR immunization coverage OR immunization rate OR immunization uptake OR vaccination coverage OR vaccination rate OR vaccination uptake OR vaccine coverage OR vaccine uptake:ti,ab,kw
#7	#1 or #2 or #3 or #4 or #5 or #6

AMSTAR is an 11-item validated tool used to assess the methodological quality of systematic reviews (Appendix I, Table A1). Each item judged as “yes” was awarded 1 point and a total score was recorded. A review with a total score of 0 to 4 was indicated as low quality, 5 to 8 as moderate quality, and 9 to 11 as high quality.

The ROBIS tool consists of three phases: (1) assess relevance (optional), (2) identify concerns with the review process, and (3) judge risk of bias in the review. Signaling questions are included to help evaluate concerns about potential biases in the review. The ratings from these signaling questions help assessors to judge overall risk of bias. The signaling questions are answered as “Yes”, “Probably Yes”, “Probably No”, “No” and “No Information”, with “Yes” indicating low concerns of risk of bias. The level of concern about bias associated with each domain is then judged as “low,” “high,” or “unclear”⁵⁸ (Appendix I, Table A2).

Principal Component Analysis (PCA) is a statistical method which has been used by different studies to reduce a large set of variables to a small set of variables called principal components. Furthermore, PCA is used to identity differences and similarities in a dataset and identify the components of the dataset that contribute to variation.⁵⁹ PCA was performed in Stata 14 to determine similarities and differences between AMSTAR and ROBIS. The 11 AMSTAR item scores and 21 ROBIS signaling questions were, respectively, converted into two sets of uncorrelated values called principal components per review and each review was tagged based on ROBIS and AMSTAR final classification. For the purpose of PCA, SRs with low, moderate and high methodological quality were, respectively, coded as 1, 2 and 3. In addition, SRs with low, unclear and high risk of bias were coded as 1, 2 and 3, respectively. Subsequently, biplots were constructed to determine the relationship between methodological quality and risk of bias as per the final classification scores of the two tools (ROBIS and AMSTAR).

Conclusions

Our results indicate that there is an association between methodological quality (measured using AMSTAR) and risk of bias (measured using ROBIS) in SRs of interventions focused on improving vaccination coverage. Therefore, either AMSTAR or ROBIS checklists can be used to evaluate methodological quality of SRs in vaccinology.

Appendix II. The AMSTAR-based PCA data

Principal components/correlation        Number of obs = 57.

                          Number of comp = 13.

                          Trace = 13.

Rotation: (unrotated = principal).        Rho = 1.0000.

Principal components (eigenvectors).

Component	Eigenvalue	Difference	Proportion	Cumulative
Comp1	4.06112	2.60331	0.3124	0.3124
Comp2	1.45782	.149729	0.1121	0.4245
Comp3	1.30809	.373597	0.1006	0.5252
Comp4	1.27073	.311338	0.0977	0.6229
Comp5	.959391	.0672598	0.0738	0.6967
Comp6	.892131	.133032	0.0686	0.7653
Comp7	.759099	.116533	0.05884	0.8237
Comp8	.642566	.18544	0.0494	0.8731
Comp9	.457126	.0569896	0.0352	0.9083
Comp10	.400137	.0218823	0.0308	0.9391
Comp11	.378254	.0746761	0.0291	0.9682
Comp12	.303578	.193621	0.0234	0.9915
Comp13	.109958	.	0.0085	1.0000

Variable	Comp1	Comp2	Comp3	Comp4	Comp5	Comp6	Comp7	Comp8	Comp9	Comp10	Comp11
item1	0.1759	0.2721	0.1748	0.6267	0.1467	−0.0338	0.6104	−0.1295	−0.0985	0.14751	−0.1678
item2	0.3640	−0.2305	0.0955	−0.0933	0.0700	−0.5084	0.2085	0.5846	−0.1659	0.0218	0.3462
item3	0.2952	0.1327	−0.5285	0.0818	0.4330	0.0163	−0.0915	−0.3536	−0.0439	−0.1131	0.5222
item4	0.4008	−0.4187	−0.2304	0.0076	−0.0078	0.7178	0.2717	0.2693	−0.2289	−0.1446	−0.0657
item5	0.2820.	0.0451	0.2902	0.1593	−0.0811	0.0736	−0.5374	−0.1097	−0.6267	0.1083	−0.1023
Item6	0.2821	−0.4916	0.2660	−0.2089	−0.0844	0.0350	0.1604	−0.4984	0.2348	0.4538	0.1439
Item7	0.2729	0.3805	0.4697	−0.2653	−0.2317	0.0237	0.0473	0.1715	0.0199	0.5882	−0.2396
Item8	0.3327	0.1750	−0.0338	−0.0310	−0.7212	−00526	0.1649	−0.1951	0.0743	−0.4951	0.1334
Item9	0.3643	0.1005	0.1271	−0.4724	0.4295	−0.0857	0.1197	−0.1184	0.0534	−0.3597	−0.5152
Item10	0.1765	0.4423	0.4654	−0.1949	0.1173	0.4522	−0.0789	0.2581	0.3435	0.0528	0.3657
Item11	0.3575	−0.2347	−0.1355	0.4661	0.0126	−0.0555	−0.3704	0.1858	0.5778	−0.0562	−0.2637

Appendix III. The ROBIS-based PCA data

Principal components/correlation        Number of obs = 57.

                          Number of comp = 21.

                          Trace = 21.

Rotation: (unrotated = principal)       Rho = 1.0000.

Component	Eigenvalue	Difference	Proportion	Cumulative
Comp1	4.7725	2.13981	0.2273	0.2273
Comp2	2.63269	.306424	0.1254	0.3526
Comp3	2.32627	.647554	0.1108	0.4634
Comp4	1.67871	.0656661	0.0799	0.5433
Comp5	1.61305	.399984	0.0768	0.6202
Comp6	1.21306	.0793917	0.0578	0.6779
Comp7	1.13367	.270353	0.0540	0.7319
Comp8	.863317	.0641371	0.0411	0.7730
Comp9	.79918	.143115	0.0312	0.8111
Comp10	.656064	.016294	0.0312	0.8423
Comp11	.63977	.115759	0.0305	0.8728
Comp12	.524011	.0710757	0.0250	0.8977
Comp13	.452936	.0629754	0.0216	0.9193
Comp14	.38996	.0610955	0.0186	0.9379
Comp15	.328865	.0636681	0.0157	0.9535
Comp16	−265,197	0.0479669	0.0126	0.9662
Comp17	.21723	.0401903	0.0103	0.9765
Comp18	.17704	.0328356	0.0084	0.9849
Comp19	.144204	.0545696	0.0069	0.9918
Comp20	.0896344	.00699254	0.0043	0.9961
Comp21	0.0826419	.	0.0039	1.0000

Principal components (eigenvectors).

Variable	Comp1	Comp2	Comp3	Comp4	Comp5	Comp6	Comp7	Comp8	Comp9	Comp10	Comp11	Comp 12
Item1	0.3134	−0.2456	0.1615	0.1713	−0.2439	0.0485	0.0435	−0.0842	−0.0616	0.3157	0.0339
Item2	0.2607	−0.1960	−0.3837	−0.0382	−0.0371	0.0031	−0.087	−0.2552	−0.0108	0.2403	0.0355	0.2045
Item3	0.2777	−0.1566	−0.1519	0.1390	−0.0464	−0.1550	−0.4408	0.1197	−0.1121	−0.2963	0.1531	0.0045
Item4	0.2072	−0.2522	0.3909	−0.0453	0.1297	0.1480	0.1752	0.0803	0.1273	−0.1620	0.1454	−0.3470
Item5	0.0725	0.1960	−0.1914	0.4294	0.3695	−0.0261	0.2030	−0.1107	0.1662	−0.0626	0.0019	0.0227
Item6	0.1717	0.2657	0.2169	−0.0248	0.2592	−0.3665	−0.3040	0.0594	0.1030	0.2131	0.1329	−0.2445
Item7	0.2161	−0.0939	0.2986	−0.0867	0.2957	−0.0988	−0.2681	−0.1389	0.1896	0.3569	−0.1159	0.3656
Item8	0.2145	−0.2248	0.1490	−0.2998	0.1580	0.0862	0.0688	0.5448	−0.2158	0.0657	0.0415	0.1252
Item9	0.0284	−0.0271	0.0404	0.5547	0.3098	0.1406	0.2236	0.2337	−0.0257	0.1859	0.3204	0.1111
Item10	0.2193	0.1741	−0.2663	−0.1830	0.1075	−0.3695	0.0917	0.3168	0.1616	−0.1187	−0.0694	0.2344
Item11	0.1507	0.1434	−0.3582	−0.1316	0.3022	−0.1621	0.2816	−0.0388	−0.1241	−0.0377	−0.2227	−0.3196
Item12	0.2260	−0.1882	−0.1083	−0.3820	0.1703	0.0011	0.2662	−0.2691	−0.0813	−0.1158	0.3931	−0.1317
Item13	0.2751	0.0291	−0.0010	0.1504	−0.4395	−0.0624	0.2752	0.2592	−0.0158	0.3339	−0.0329	−0.1938
Item14	0.2368	0.2627	0.3467	0.1192	−0.0303	0.0032	0.0544	−0.1087	−0.1944	−0.1771	−0.1317	−0.0380
Item15	0.2749	0.1794	0.1107	0.0376	0.0824	0.2223	0.0921	−0.1196	−0.5408	−0.1820	−0.2543	0.3696
Item16	0.1476	0.1156	0.2264	0.0184	−0.3665	−0.4254	0.3149	−0.0315	0.2219	−0.1759	−0.0056	0.2338
Item17	0.2427	0.0493	−0.2105	0.1241	−0.0715	0.3029	−0.2995	0.3305	0.2298	−0.3210	−0.1484	−0.1012
Item18	0.2113	−0.3457	0.0922	0.0909	0.0549	0.1080	0.1172	−0.2335	0.4714	−0.2703	−0.2206	0.0793
Item19	0.1350	0.3270	−0.0211	−0.2555	0.0236	0.4466	0.0643	0.0314	0.3031	0.2991	−0.2797	−0.0912
Item20	0.0573	0.4057	−0.0520	−0.1657	−0.1196	0.2804	0.0232	−0.0155	0.1974	−0.1030	0.5976	0.2956
Item21	0.3311	0.2209	0.0588	0.0778	−0.1342	0.0328	−0.2287	−0.2948	0.1101	0.0309	0.1360	−0.3126

Funding Statement

This work was supported by the South African Medical Research Council and the National Research Foundation of South Africa (Grant number 106035).

Acknowledgments

The authors would like to acknowledge Yusentha Balakrishna, the Biostatistician who assisted with conducting the Principle Component Analysis.

Disclosure of potential conflicts of interest

No potential conflicts of interest were disclosed.

Declarations

Ethics approval and consent to participate

This study was based on an analysis of existing survey data with all identifier information removed; therefore, no ethical clearance was required.

Consent for publication

Not applicable.