Effects of MVA85A vaccine on tuberculosis challenge in animals: systematic review

Abstract

Background: The existing Bacillus Calmette–Guérin (BCG) vaccination provides partial protection against tuberculosis (TB). The modified vaccinia ankara virus-expressing antigen 85A (MVA85A) aims to boost BCG immunity. We evaluated the animal evidence supporting the testing of MVA85A in humans.

Methods: Our protocol included in vivo preclinical studies of the MVA85A booster with BCG compared with BCG alone, followed by a TB challenge. We used standard methods for systematic review of animal studies, and summarized mortality, measures of pathology and lung bacterial load. The comprehensive literature search was to September 2014. Two independent investigators assessed eligibility and performed data extraction. We assessed study quality and pooled bacteria load using random effect meta-analysis.

Findings: We included eight studies in 192 animals. Three experiments were in mice, two in guinea pigs, two in macaques and one in calves. Overall, study quality was low with no randomization, baseline comparability not described and blinding not reported. For animal death (including euthanasia due to severe morbidity), studies were underpowered, and overall no benefit demonstrated. No difference was shown for lung pathology measured on an ordinal scale or bacterial load. The largest mortality trial carried out in macaques had more deaths in the MVA85A vaccine group, and was published after a trial in South Africa had started recruiting children.

Conclusions: This independent assessment of the animal data does not provide evidence to support efficacy of MVA85A as a BCG booster. More rigorous conduct and reporting of preclinical research are warranted, and we believe the results of studies should be publicly available before embarking on trials in humans, irrespective of the findings.

Key Messages.

• In this systematic review of animal studies evaluating MVA85A vaccine to boost BCG immunity to tuberculosis, we found eight studies in 192 animals in all. Studies were underpowered and of poor quality. No effect was demonstrated on animal death, lung pathology or bacterial load.

• The largest mortality trial in macaques had more deaths in the experimental group, and was published after a trial of the vaccine in children had started recruitment.

• Whereas it is recognized that there are problems with the predictive value of some animal models, this review indicates that animal researchers need to be more rigorous in their methods, in their reporting and in prompt publishing of the results, irrespective of the study findings.

Introduction

Modified vaccinia ankara virus-expressing antigen 85A (MVA85A) is a new tuberculosis (TB) vaccine currently undergoing clinical trials in children as a boost to Bacillus Calmette–Guérin (BCG), as BCG protection is at best partial and variable.^1,2 As we set up to carry out an independent systematic review of MVA85A vaccine trials in children, there was considerable debate in the literature that highlighted the potential gains from optimizing the design, conduct and analysis of biomedical research to avoid misleading findings and wasting resources.^3,4 We therefore decided to explore these issues in animal studies, using systematic review and meta-analysis as methods to carry out syntheses in animal studies.^5,6 These systematic reviews are important in animal research as they ask both a scientific and moral question, because studies that are poorly designed, conducted or reported are unlikely to be reliable and the ‘animals in effect [have] been wasted’.^6,7 We sought to assess the experimental design and study quality and summarize the results of studies evaluating MVA85A combined with BCG compared with BCG alone in in vivo animals challenged with TB. This would allow us not only to independently evaluate the strength of the pre-clinical evidence, but also to assess the rigour of the design and reporting against standards in emerging animal research quality criteria in the field of vaccine development.⁸

Methods

Our methods were pre-specified in a study protocol [http://www.dcn.ed.ac.uk/camarades//research.html],⁹ included in Supplement 1 (available as Supplementary data at IJE online). We included in vivo controlled studies of any animal with a TB challenge, where animals were allocated to an intervention group and a control group. We defined control groups as those treated with BCG alone, and the intervention group as those treated with MVA85A vaccine given after BCG vaccination. Studies of MVA85A combined with other antigens were also included. We included studies that measured at least one of the following outcomes: (i) death, including severe morbidity that required euthanasia (termed ‘humane endpoint’); (ii) measures of lung pathology; and (iii) lung bacterial loads. We excluded parameters such as spleen bacterial loads or immunological measures as these are not considered to directly relate to functional protection against TB. These selected endpoints are defined as indicators of protection by specialists in this field in a recent review.¹⁰

Search strategy

We searched the following databases from inception up to 8 September 2014: MEDLINE (Pubmed); EMBASE (OVID); Science Citation Index-expanded and Science Conference Proceedings (Web of Science); and Biosis previews (Web of Science), using the following search terms in title, abstract and keywords: ‘MVA85A’ OR ‘modified vaccinia virus Ankara’ OR ‘Ag85A’ OR ‘Antigen 85A’ AND ‘tuberculosis’ OR ‘TB’ OR ‘BCG’. We did not apply any language restrictions to the searches. We also contacted experts in the field, individual animal researchers and vaccine trial groups for unpublished data. We also checked the reference lists of relevant studies.

Selection and description of studies

Two investigators independently applied the predefined inclusion criteria (R.K. and T.Y.), and extracted data from relevant studies (R.K. and E.S.). Discrepancies were discussed by the team and agreement reached with P.G. We extracted details of the vaccines used, the route of vaccine administration, the type of TB challenge strain, and the route of TB administration. We also extracted the duration between the initial BCG vaccination and MVA85A booster (BCG/MVA85A interval), the duration between the MVA85A boost and the TB challenge (MVA85A/challenge interval), and the duration between the challenge and outcome assessment.

We used aspects of the Animal Research Reporting in vivo Experiments (ARRIVE) guidelines and the survey of the quality of experimental design, statistical analysis and reporting of research using animals to assess the design quality, reporting and risk of bias of included studies.^6,8,11 We assessed whether study objectives were stated, whether sample size calculations were reported, whether the number of animals included were clearly described and whether there was a competing interest statement. We evaluated if animals were randomized to treatment allocation and assessed for baseline comparability and whether assessors were blind to the allocated group for: (i) humane endpoint; (ii) pathology; and (iii) bacteriology assessment.^8,11

Outcome data

The numbers of animals that reached a humane endpoint in each group were recorded for each study where this was reported. We summarized our assessment of humane endpoints in a table. We also calculated the risk ratio of death in studies that reported outcome data of for macaques that reached humane endpoint, and pooled these using random effects meta-analysis. Pathological data were summarized in a table. For scores, we reported median and range values that were derived manually.

For bacterial load data, we identified individual comparisons where outcome was measured in animals receiving intervention compared with control group animals. Two authors (R.K. and E.S.) independently extracted means and corresponding variances for each experimental arm using digital ruler software from graphs. On comparison of results, in cases where there was more than a 10% difference, both authors re-extracted the data until an agreement was reached. A mean of the values extracted by both authors were used for meta-analysis. Where aggregate data were not reported but individual animal data were provided, we used Microsoft Excel to calculate the means and standard deviations. For studies that gave a standard error (SE) or confidence interval (CI) of the bacterial loads, the figures were changed to standard deviations using Review Manager 5 software.¹² We calculated a normalized mean difference effect size for each comparison.⁶ The data were pooled using DerSimonian and Laird random effect meta-analysis.

Results

We included seven studies from 421 records identified from our search, after considering duplication of studies (Figure 1). We also found an additional study through contacting experts in the field.¹³ This additional study was the largest preclinical trial of MVA85A, carried out in monkeys, but was not identified by our electronic search or by checking references; it was not indexed under MVA85A and had a title without relevant keywords. This gives a total of eight included studies.

Figure 1.

PRISMA flow diagram.

One publication described two similar experiments that were carried out in two different laboratories (Oxford and Berlin) with different challenge strains; we treated these as the same study but stratified the results by city.¹⁴ Williams et al. in the journal Tuberculosis describes four experiments, two of which met the inclusion criteria of our review. In a second publication, in the journal Infection and immunity, the authors appeared to report on one of these experiments again. We therefore included the third experiment’s data under Williams (a).¹⁵ We report the fourth experiment from the Tuberculosis paper, also reported in Infection and immunity, as a single experiment labelled Williams (b).¹⁶ Excluded studies are detailed in Figure 1 and listed in the Supplement (available as Supplementary data at IJE online).

Description of included studies

Of the eight included studies, two evaluated BCG followed by MVA85A followed by a recombinant fowlpox-expressing antigen 85A (FPAg85A) acting as a second viral vector and booster. Six studies evaluated BCG followed by MVA85A alone (Table 1). Three studies used mice, two used macaques, two used guinea pigs and one used calves. Sample sizes ranged from 4 to 14 per group and study duration ranged from 26 to 73 weeks after TB challenge. One study reported a range of animals from 9 to 12 in each group, and we used the minimum of 9 for analysis.¹⁷

Table 1.

Characteristics of included experiments

Intervention	Study	Duration (weeks)	Animal	MVA85A routes	BCG route	Animals in each group x groupsa	Challenge strain	Challenge	BCG/MVA85A interval	Boost to challenge (weeks)	Challenge to autopsy (weeks)	Funding	Reference list
								Route	(weeks)
BCG+	Williams 2005 (a) Expt. 3	29	Guinea pigs	IDa	SC	6 × 6	H37Rv	Aerosol	4	8b	17	EU; DOH UK	15
MVA85a + FP9Ag85a	Williams 2005 (b)	40	Guinea pigs	ID	SC	6 × 3	H37Rv	Aerosol	4	6b	26	EU; Wellcome Trust; DOH UK	16
BCG + MVA85A	Goonetilleke 2003	32	Mice	IM/NAc	NA	9–12 × 7	H37Rv	‘In lungs’	22	4	6	Wellcome Trust	17
	Romano 2006	Not stated	Mice	IV	IV	11–14 × 3	H37Rv	Intravenous	34.8	14	Not mentioned	FWO – Vlaanderen, Brussels capital region, Damiaanktie Belgiium, European Economic community.	18
	Tchilian 2008:Oxford	26	Mice	ID	SC	5–6 × 3	Erdman	Aerosol	10	4	12	European FP6 integrated project, NIHR Oxford Biomedical Research Centre programme	14
	Tchilian 2008: Berlin	26	Mice	ID	SC	4–5 × 3	H37Rv	Aerosol	10	4	12		14
													27
	Verrick 2009	34/35	Macaques	ID	ID	6 × 4	Erdman	Tracheal	9	9	16/ 17	Wellcome Trust; EU	19
	Vordermeier 2009	28	Calves	ID	SC	10 × 4	M.bovis	Tracheal	8	6	14	DOE UK; Wellcome Trust	1
	Sharpe 2010	73	Macaques	ID	ID	4–6 × 3	Erdman	Aerosol	12	9	52	DOH UK	13

EU, European Union; DOH UK, Department of Health UK; DOE UK, Department of Environment UK; expt, experiment; IM, intramuscular; NA, nasal; ID, intradermal; SC, subcutaneous.

^aSource of vaccine similar to that used in Williams et al. 2005 (b).

^bThis boost challenge interval shows weeks after FP-Ag85A, the second booster that was given 4 weeks after MVA85A booster compared with other studies where the TB challenge follows MVA85A, the only booster used.

^cTwo intervention arms: one with MVA85A given nasally, and one arm with MVA85A given by injection.

Intradermal administration was used in six studies and intravenous in one study. In the remaining one study, two comparisons were identified where MVA85A was administered nasally in one group of mice and parenterally in another. Three different challenge strains were used: H37Rv (five experiments), Erdman (three experiments) and M. bovis (one experiment). The BCG/MVA85A interval ranged from 4 to 22 weeks (median 9.5 weeks); the MVA85A/challenge interval ranged from 4 to 9 weeks (median 6 weeks). The challenge to outcome assessment interval ranged from 6 to 52 weeks(median 13 weeks). One study had no reported endpoint but survival data presented did not go beyond 40 weeks.¹⁸

Five studies reported on death or animals reaching a humane endpoint; four reported on pathology; and six reported on bacterial load.

Reported study quality

Table 2 summarizes the reported study quality. None of the studies reported a sample size or power calculation. Of the eight studies, six studies described precisely the number of animals used; one reported ‘five animals in each group’ but the results showed groups of four to six in each group;¹⁴ and one reported a range of between nine and 12 animals.¹⁷

Table 2.

Reported study quality

Grey shading - outcome not evaluated in study.

COI, conflicts of interest

Two studies reported potential conflict of interest as some authors were ‘co-inventors of MVA85A and shareholders in the joint venture developing the vaccine’. The other six studies had no statements regarding conflict of interest although the same authors who had declared a potential conflict of interest were also co-authors of five of the studies.

One study reported random allocation of animals into treatment groups. Only one study had a baseline comparability table and reported that there was no significant difference in the body weight and age of animals between groups at the start of the experiment.¹⁹ Of the others, three reported age, sex and species; one, weight, sex and species; two, age and species; and one, species only.

We assessed the blinded assessment of outcome for each of the three endpoints separately. None of the studies reporting humane endpoints or bacteriology reported blinding their assessment. For pathology, two of the four studies reported that the assessors were blind to treatment allocation.^1,19

Mortality, including euthanasia for severe morbidity

Five studies with a total of 107 animals in the relevant arms assessed mortality (Table 3). Two used macaques, two used guinea pigs and one used mice. Two of the five studies that assessed mortality/euthanasia did not provide data on the number of animals euthanized. Williams et al. (a) reported that the results ‘did not allow any distinction to be made between MVA85A boosting and BCG alone’, but no data were reported.¹⁵ Williams et al. (b) reported a ‘statistically different increase in survival (P = 0.018)’.¹⁶

Table 3.

Experiments MVA85A comparative vaccines trials in animals: death, including euthanasia for severe morbidity^a

Intervention	Study I.D/reference	Model	Boost-challenge interval (weeks)	Criteria for humane endpoint	Follow-up (weeks)	MVA85A boost	BCG alone	No vaccine	Statistical methods for significance reported	Authors conclusions BCG/MVA85A vs BCG alone
BCG + MVA85A + FP9Ag85a	Williams 2005 (a) Experiment 315	Guinea pigs	8	Specified	17	No report /6	No report /6	No report /6	Yes	‘Unable to discriminate between MVA85A boosting and BCG’
	Williams 2005 (b)16	Guinea pigs	6	Specified	26	0/6	4 /6	6/6	No	‘Statistically different survival, P = 0.018’
BCG + MVA85A	Romano 20061 8	Mice	14.3	Not specified	ND	a	a	a	Yes	No statistical difference
	Verrick 20091 9	Macaques	18	Specified	35	0/6	1/6	2/6	Yes	No statistical comparison made
	Sharpe 20101 3	Macaques	9	Specified	52	5/6	2/6	4/4	Yes	‘No statistical differences in survival time’

ND, no data reported.

^aStudy did not report number of deaths but that median survival time of animals in the MVA85A group (18.5 weeks) was not statistically different from the BCG group (19 weeks).

In the remaining three studies, results were variable but the numbers are small (Figure 2). In one study with the longest follow-up, five out of the six macaques died in the MVA85A group, compared with two out of six in the BCG group.¹³ The other two studies reported no deaths in the MVA85A groups, compared with four and one in the BCG groups, respectively.^16,19 Another study reported median survival rather than the number of events at a single time, so we were unable to include these data in the meta-analysis. In this study, median survival time of animals in the MVA85A group (18.5 weeks) was reported as not statistically different from the BCG group (19 weeks).¹⁸

Figure 2.

MVA85A combined with BCG compared with BCG alone. Death, including euthanasia for severe morbidity.

Lung pathology

Four studies with a total of 100 animals in the comparison arms reported pathological changes after TB challenges were given, reported on ordinal scales. The text in the papers made inferences that were not evident from the data; for example, one study implied benefit by comparing the MVA85A group with controls rather than with BCG,¹ but overall there was no effect obvious in comparisons between MVA85A with BCG comparedwitho BCG alone (Table 4).

Table 4.

Experiments MVA85A boosting vs BCG alone or naive, outcome 2: pathology changes

Study	Model	Outcome	Measured	Summary value	MVA85A	BCG alone	No vaccine	MVA85A vs BCG alone
								Author conclusions	Our interpretation
Williams 2005 (b)15	Guinea pigs	Lung consolidation	%		NDa,e	NDa,e	Unclear	‘Not able to discriminate…’	No data provided
Williams 2005 (b)15	Guinea pigs	Lung consolidation	%		NDa,e	NDa,e	Unclear	Not significantly better than BCG	No data provided
Verrick 2009c 19	Macaques	Lung lesions	No .of animals	n/N	3/6	5/6	6/6	BCG/MVA85A less severe disease and fewer lesions than BCG only	Scores in both BCG alone and MVA85A boost much lower than controls
		Hilar involvement	No. of animals	n/N	4/6	5/6	6/6
			Hilar LN score	Median, range	1.5, 0–3	2, 0–3	2.78, 2–3
		Disseminated pathology	Animals	n/N	1/6	1/6	4/6
		Lung pathologyd	Score	Median, range	2.2, 0–14	6, 0–25	18, 9.6–25
		Total pathology	Score		39	63.5	142
Vordermeier 20091	Calves	Lung TB lesions	No. of animals	n/N	5/10b	7/10	10/10	‘Significantly reduced after MVA85A vaccination compared to control’f	No direct statistical comparison between MVA85A and BCG
		Lymph node involvement	No. of animals	n/N	6/10	7/10	9/10
			LN score	Median, range	3.5, 0–11	2.5, 0–9	7.5, 0–14
		Lung pathology	Score	Median, range	2.5, 0–9a	5, 0–10	10.5, 5–35
		Total pathology	Score	Median, range	7.5, 0–16a	6, 0–18	19.5, 5–35
Sharpe 20101 3	Macaques	Lesions in lungsd	No. of lesions/animal	Median, range	153.13, 60 to 360	211.97, 160 to 280b	98.44, 80 to 200	No statistical difference between MVA85A-vaccinated and BCG-vaccinated animals for lesions in lung, lung pathology and gross pathology scoring	No statistical difference but more lesions in BCG and MVA85A groups. Other parameters show no trends
		Extra-pulmonary granulomas	No. of animals	n/N	4/6	3/6	4/4
		Lung pathology scored	Score	Median, range	20.625, 10.25 to 23.5	19, 5.75 to 21.75	26, 18.5 to 25
		Total pathology scored	Score	Median, range	25.3, 14.8–37.4	22.9, 18–26a	30.96, 25.6–37.4

^aVaccinated significantly less pathology than naïve group.

^bVaccinated significantly more pathology than naïve group

^cNo difference in all other pathology parameters used including lung congestion, consolidation, fibrous pleuritis, lung oedema and emphysema.

^dResults estimated from extrapolation on graphs.

^eNo data given.

^f‘BCG vaccination reduced the number of infected lobes significantly (P < 0.05) and MVA85A reduced these even further which is reflected in the higher level of statistical significance observed (P < 0.001).’

Bacterial load

Six studies with a total of 137 animals measured bacterial loads in animals after a TB challenge (Table 5). Of these six studies, we excluded the study by Williams et al. from the meta-analysis as they did not report the data required to perform meta-analysis. The results in this study were reported as being significantly or not significantly better when compared with the BCG alone group, without outcome data.¹⁵ From the five remaining studies, we extracted seven comparisons of MVA85A boosting vs BCG alone that we included in meta-analysis. Overall, MVA85A boosting showed no reduction in bacterial loads (3.28%, 95% CI 3.5 to 9.8, P = 0.267) (Figure 3).

Table 5.

Experiments in MVA85A-boosted animals vs BCG-vaccinated or naïve animals: bacterial loads, reported as log₁₀ CFU

Intervention	Study ID	Model	MVA85A booster			BCG alone			No vaccine			Difference detected? MVA85A/BCG vs BCG
			Mean	SE	n	Mean	SE	n	Mean	SE	n
BCG/MVA85A/ FP9.Ag85A	Williams 2005 (a) experiment 315	Guinea pigs	ND	ND	6	ND	ND	6	ND	ND	6	No
	Williams 2005 (a) experiment 415	Guinea pigs	ND	ND	6	ND	ND	6	ND	ND	6	No
BCG/MVA85A	aGoonetilleke 20031 7	Mice	4.5	0.2	9 to 12	4.6	0.8	9 to 12	5.5	0.4	9 to 12	MVA85A: parenteral, no MVA85A intranasal, yes
	bGoonetilleke 20031 7	Mice	3	0.2	9 to 12	4.6	0.8	9 to 12	5.5	0.4	9 to 12
	Tchilian 2008 (Berlin)14	Mice	3.8	0.09	5	3.8	0.1	4	4.75	0.18	5	No
	Tchilian 2008 (Oxford)14	Mice	5.5	0.13	6	5.5	0.18	5	6.2	0.03	5
	Verrick 20091 9	Macaques	3.15	0.39	6	3.72	0.39	6	4.15	0.62	6	No
	Vordermeier 20091	Calves	2.313	0.5	10	2.930	0.34	10	3.487	0.4	10	No
	Sharpe 20101 3	Macaques	6.5	0.7	6	6	0.52	6	7.5	0.33	4	No

ND, no data.

^aIntranasal administration.

^bParenteral administration.

Figure 3.

Meta-analysis using normalized mean difference for MVA85A vs BCG alone: bacterial loads after TB challenge given to animals.

Discussion

We confined this review to functional parameters of protection in animal models. Indeed, the vaccine scientists state the decisions to move to trials in humans are defined by animal studies with ‘statistically significant improvements in disease compared to control groups as measured by bacterial load, severity of pathology, and time to death’.¹⁰ We used standard methods for systematic reviews and meta-analysis in animal studies.^5,6

Our meta-analysis suggests an apparent lack of evidence of efficacy in animals in data collected before the start of the recent phase 11b trial in South African children that enrolled children between 15 July 2009 and 4 May 2011.²⁰ We acknowledge that the decision to progress to clinical trial is not solely based on evidence derived from preclinical efficacy studies, but these studies are an important component of the TB vaccine development paradigm.¹⁰ Selection of adequately powered endpoints in preclinical studies, such as the 60% improvement in clinical efficacy that is required for licensing a vaccine, may result in similar magnitude of improvement in animals and thus be more predictive of human trial efficacy.¹⁰ Indeed, the recent clinical trial testing MVA85A in 2797 South African infants did not demonstrate a beneficial effect of MVA85A.²⁰ In subsequent papers, the authors of this trial have explained that this trial did not have an effect due to species differences, clinical trial settings, M. tuberculosis strain and exposure, magnitude of efficacy, definition of protection and whether to use reduction of disease or prevention of disease as an endpoint.¹⁰ The principal investigator of a number of the animal studies was also the senior principal investigator in the South Africa infant trial.

We believe that, as with any research endeavour, validity depends on experiments conducted and reported with rigour. None of the studies in our review report a sample size or power calculation; few studies were either blinded or randomized, casting doubt on the internal validity of the studies. There were too few studies to statistically assess for selective reporting of outcomes and a priori protocols of the analysis were not available. We know the shortcomings in the conduct and reporting of animal research may lead to over- or under-estimations of treatment effects,^21,22,25 and this indicates critical, systematic and independent appraisal of animals studies is a potentially important component of translational research. The limitations of the animal model in tuberculosis also causes the researchers themselves to be concerned about the predictive abilities.²³

We appreciate that the effects of a boost vaccine to BCG may be more modest, with requirements for larger samples of animals.²³ Indeed, research that is inadequately powered could be regarded as unethical, as the design is insufficient to answer the question the research is trying to address. It is important that the ethics in animal research and power are more fully debated. In addition, it is important that if the decision by researchers to proceed from animal studies to children is not based on ‘statistically significant improvements in disease compared to control groups’, but on other evidence, this evidence needs to be systematically summarized, appraised, checked for completeness and documented to allow transparency in the translational process.¹⁰ There are gating criteria for TB vaccines published, that include safety, immunogenicity and animal efficacy.²⁴

Adherence to reporting guidelines will allow consumers of animal research to draw informed conclusions from results presented. Journal editors, peers and granting bodies should drive this improvement. In addition, a priori protocols, where investigators provide details of appropriate experimental design and statistical analysis, are important to this process. ^25,26 Finally, it may be that publication bias further confounds our conclusion, but we identified too few studies to allow us to assess for this using standard techniques.

However, there did seem to be some evidence of a delay of 2 years in publication of one study, which concerned us. The longest follow-up study in our dataset, containing almost half (16/34) of the data testing this vaccine in monkeys, was published in June 2010, almost a year after the South African trial had started testing the vaccine in South African children and 2 years after this trial in monkeys had been completed. This trial reported that five out of the six monkeys given the experimental vaccine with BCG died or were so ill they needed to be euthanized, compared with only two out of six in the BCG control group. In addition, even when published, Sharpe 2010^{1 3} is not detected on a standard MEDLINE search using MVA85A because of MVA85A is not mentioned in the title or abstract.

Our review presents a useful summary of the preclinical data on MVA85A but there are limitations to our approach. We are confident that our search strategy was robust, but there were only eight studies that met our inclusion criteria. Indeed, two of the eight studies included FPag85A in addition to MVA85A; these were included as overall data were limited, and if effects were seen this could be discussed in terms of confounding. The data in these two studies were similar to the others. Few studies using different experimental designs and species may be considered akin to lumping oranges with apples, which may mask subtle but relevant differences in efficacies. However, we used a normalized mean difference effect size in our pooling of bacteriology; this is presented as a percentage improvement relative to the magnitude of effect in normal healthy animals.⁶

We acknowledge that our findings are at variance with a narrative review by the academic groups responsible for MVA85A. In this review, they discuss six of the eight studies included in this review and concluded from individual study data from three of the studies that MVA85A had a ‘variable and modest level of efficacy in animals that failed to predict efficacy in BCG-vaccinated infants to a level required for progression of the vaccine development’.¹⁰

Conclusions

Our review raises a question about the robustness of claims that MVA85A animal studies provide evidence of protection against TB challenge. We also found that there was a need to attend to methodological standards in the design, execution and reporting of pre-clinical animal studies. Many were inadequately powered and little attention was given to potential sources of biases. We would echo a recent publication that stated research needs to improve ‘reproducibility practices, more appropriate (usually more stringent) statistical thresholds; and implement in study design standards’;³ and researchers of animal studies should publish their results promptly, irrespective of the findings.

Supplementary Data

Supplementary data are available at IJE online.

Funding

This project has been supported by the US President’s Emergency Plan for AIDS relief (PEPFAR) through HRSA under the terms of T84HA21652 and via a bursary from the Stellenbosch University Rural Medical Education Partnership Initiative (SURMEPI). This paper is a research activity within the Cochrane Infectious Diseases Group funded by DfID for the benefit of developing countries. This study was funded by the Department for International Development UK for research synthesis aimed to benefit the poor in developing countries. The contribution of E.S. was funded by the UK National Centre for the Replacement, Refinement and Reduction of Animals in Research (NC3Rs) infrastructure award: ivSyRMAF–the CAMARADES–NC3Rs in vivo systematic review and meta-analysis facility. The study sponsors had no role in design, collection, analysis or interpretation of data, writing of the report or decision to submit the manuscript for publication. DfID funded Open Access through a grant to CIDG. R.K., the corresponding author, had full access to the data and made the final decision to publish.