An evaluation of a test-negative design for EV-71 vaccine from a randomized controlled trial

ABSTRACT

Background: The test-negative design has been used widely in evaluation of various vaccines’ effectiveness, such as influenza, rotavirus, and so on. Recently, there have been some studies about EV-71 vaccine effectiveness by using test-negative design(TND). However, the validity of the TND application in EV-71 vaccines has not been evaluated.

Methods: This study is set upon prior methods to evaluate the validity of TND for influenza vaccine by using a randomized controlled clinical trial database. Vaccine effectiveness estimated by TND (VE-TND) in modified intention-to-treat population (mITT) and per-protocol-set population(PPS) was derived from a large randomized placebo-controlled clinical trial (RCT) of inactivated monovalent EV-71 vaccine in China. Derived VE-TND estimates were compared to the original vaccine efficacy results in RCT (VE-RCT).

Results: We totally enrolled 7325 participants who seeked medical care for suspected EV-71 infected diseases during the surveillance. There are no significant differences between cases(test-positive) and controls(test-negative) on sex, age, height, and weight. TND vaccine effectiveness estimates were similar to original RCT vaccine efficacy estimates, both in modified intention-to-treat population and per-protocol populations.

Conclusions: This study supports that TND, as an appropriate observational study design is valid to measure EV-71 vaccine effectiveness.

1. Introduction

EV-71 infection could cause various diseases including hand, foot, and mouth disease (HFMD), herpangina, neurological signs and nonspecific illnesses, affecting mainly children younger than 5 years-old.¹ EV-71 has emerged as a major neurotropic virus responsible for severe neurological complications and has become a serious public health threat to infants and young children across the Asia-Pacific region. Since 1990s, EV-71 infections have become the primary cause of HFMDs. In China, more than 10 million cases of HFMD with more than 3000 fatalities were reported between 2008 and 2012.^2,3 Currently, there is no approved antiviral drug against EV-71 infection.⁴ An efficacious prophylactic vaccine is the best choice to prevent EV-A71 infection and disease. Until 2016, three inactivated monovalent EV-71 vaccines (developed by Chinese Academy of Medical Sciences (CAMS), Sinovac Biotech, and Beijing Vigoo Biological) were licensed to prevent EV-71 infection in China.⁵ These EV-71 vaccines showed high efficacy against EV71-associated diseases in phase III randomized controlled trials.^6–8 The vaccines’ efficacy rates were 97.4%(CAMS), 94.8%(Sinovac) and 90 · 0%(Beijing Vigoo), separately.

However, the studies about evaluating the effectiveness of EV-71 vaccine after licensure are only few. So far, three studies have been published about evaluating the effectiveness of EV-71 vaccine in the real-world.^5,9,10 The EV71 vaccine effectiveness rates were 85 · 4%(Henan), 88.3%(Guangxi) and 83.7%(Beijing), separately. They all used “test-negative design”(TND) to estimate the effectiveness of EV-71 vaccine. They did a pioneering work and gave a template for evaluating the effectiveness of EV-71 vaccines after licensure by TND.¹¹

TND is considered as a variant of the traditional case-control study and has become popular for post-licensure observational studies of the effectiveness of vaccines, especially for influenza and rotavirus vaccines.^12,13 In this design, patients seeking medical care for a predefined clinical condition are tested for a specific viral infection by using a highly sensitive and specific laboratory test. Those tested positive are cases and controls are tested negative for infection but meeting the same enrollment criteria. Vaccine status is compared between test-positive cases versus test-negative controls.¹⁴

The real-world EV-71 vaccine effectiveness estimates showed that the effectiveness of EV-71 vaccines are a little lower than the efficacy in phase III randomized controlled trials.^5,9,10 Maybe it reflected the difference of vaccine’s effectiveness between the real world and clinical-trial situation. Or possibly, the accuracy and precision of TND for EV-71 vaccines evaluation is not steady enough. Therefore, we would like to examine the validity about the TND application in EV-71 vaccines. De Serres, G. etc used databases from four influenza vaccine randomized controlled trials(RCT) to verify the validity about the TND application in influenza vaccines.¹⁵ The RCTs represent the ideal background to assess the validity of the TND,¹⁵ providing the comparability of vaccinated and unvaccinated participants with follow-up and disease ascertainment which could minimize the bias and confounding. So here we did a post-hoc analysis and try to confirm the validity of TND for EV-71 vaccine by using the database from a randomized controlled trial of EV-71 vaccine.

2. Method

2.1. Data source

The database is from a multi-center, randomized, double-blind, placebo-controlled, phase 3 trial, which conducted in China(NCT01508247).⁸ The first EV71 vaccine phase 3 trial performed by Vigoo had high levels of quality control and data. It could provide the ideal setting for us to simulate the TND approach The inactivated alum-adjuvant EV-71 vaccine was developed by Beijing Vigoo Biological, which children aged 6–35 months received at 0 and 28 days. The trial enrolled 10245 children and infants, 5120 participants in vaccine group and 5125 in placebo group. Details can be found in the related publication.⁸ In original phase 3 trial, we did a high-quality combination of active and passive monitoring to capture any suspected EV71 cases. Thus, we concluded that we had no omission of potential cases and this was why phase 3 trial could provide an ideal setting for us to simulate TND approach. Participants were actively followed after being administered two doses of vaccine or placebo. Guardians were encouraged to take their children to treatment in clinics or hospitals for any illness. The surveillance was from 56 days after first dose to 14 months. We collected and sort out information from those participants who had suspected illness associated EV71 infection and did a test-negative design to evaluate the EV-71 vaccine effectiveness for Enterovirus 71 (EV71)-associated diseases, EV71-associated HFMD, and EV71-associated other cases.

2.2. Case definitions and laboratory testing

EV71-associated disease is defined as clinical symptoms including HFMD, herpangina, neurological signs (aseptic meningitis or encephalitis) with or without serious sequelae, and nonspecific illnesses – e.g., febrile illness, viral exanthema, respiratory infection.¹⁶ HFMD-like illness is characterized by febrile illness with papulovesicular rash (can occasionally be maculopapular without vesicular lesion) on palms and soles, with or without vesicles or ulcers in the mouth, buttocks, knees, or elbows. EV71-associated other cases are EV71-associated disease except HFMD.⁸

All potential cases were reported by the clinics or hospitals. Throat or rectal swabs or both were taken for pathogen detection within 24 h. These samples were tested for EV71 by real-time PCR (fluorescence assay) with a viral RNA diagnostic kit. A participant was defined as a case if at least one positive test result during follow-up. A participant was defined as a control if at least one negative test results during follow-up and the participant had no positive results.

2.3. Statistical analysis

The analysis was based on the mITT population and PPS populations so that we could compare the results with RCT’s. The modified intention-to-treat population included those participants who received at least one dose. The per-protocol population included those participants who received two doses. TND vaccine effectiveness (VE-TND) was defined as (1- Odds Ratio) *100. Logistic regression was used to estimate the odds ratio and associated 95% confidence intervals (CI) of test-positive participants compared to test-negative participants by vaccine and placebo status. We used the χ2 test or Fisher’s exact test to analyze categorical data, ANOVA to analyze continuous data. Statistical analyses were done with SAS software (version 9.4). All statistical tests were two-sided, and statistical significance was defined as P < .05.

3. Results

3.1. Participant characteristics

During the surveillance, we totally enrolled 7325 participants who seeked medical care for suspected EV-71-infected diseases, 52 tested positive as cases and 7273 tested negative as controls in modified intention-to-treat population, 49 tested positive as cases and 7088 tested negative as controls in per-protocol population. Table 1 shows the characteristics between cases and controls. There are no significant statistical differences between cases and controls on sex, age, height, and weight. Baseline characteristics of groups were balanced with so that further adjustment for residual confounding was not required. Figure 3 shows the major EV-71 infection epidemic curves. EV71 infection major occurred from April to May and from November to December. Our surveillance embraced two epidemic period about EV71.

Table 1.

Comparison of demographic characteristics between cases and controls in mITT and PPS

Modified intention-to-treat population	cases(52)	controls(7276)	P
sex(male,%)	30(57.7%)	4117(56.6%)	0.8749
age(month)	19.5 ± 8.3	18.3 ± 8.0	0.2957
height(cm)	82.8 ± 9.0	81.4 ± 8.1	0.1997
weight(kg)	12.9 ± 2.7	12.4 ± 2.2	0.2523
Per-protocol population	cases(49)	controls(7091)
sex(male,%)	29(59.2%)	400(56.4%)	0.6988
age(month)	19.8 ± 8.3	18.4 ± 8.0	0.2084
height(cm)	83.2 ± 9.0	81.4 ± 8.1	0.1203
weight(kg)	13.0 ± 2.7	12.5 ± 2.2	0.1560

Sex: percent of male, P value was examined by Chi2 Test; Age: mean±SD, P value was examined by Anova; Height: mean±SD, P value was examined by Anova; Weight: mean±SD, P value was examined by Anova; Cases were detected EV71 positive while controls were detected EV71 negative.

Figure 3.

Suspected EV71 infected cases epidemic curves in mITT and PPS

3.2. Vaccine effectiveness

There were 1453 participants had HFMD-like illness, 36 tested positive while 1417 tested negative. The rest were 5872 EV71-associated other cases, 16 tested positive and 5856 tested negative. Table 2 shows the test-negative vaccine effectiveness and the original RCT vaccine efficacy. The VE-TND against EV71-associated disease, EV71-associated HFMD and EV71-associated other cases are 81.7%(95%CI: 61.6,91.4), 91.2%(95%CI: 71.2,97.3), and 53.9%(95%CI: −32.9,84.0), separately.

Table 2.

VE-TND and VE-RCT results

Modified intention-to-treat population	Case#	Controls#	VE-TND%(95%CI)	VE-RCT%(95%CI)
EV71-associated disease	8/44	3628/3645	81.7(61.1,91.4)	81.9(61.5,91.5)
EV71-associated HFMD	3/33	720/697	91.2(71.2,97.3)	90.9(70.4,97.2)
EV71-associated other cases	5/11	2908/2948	53.9(−32.9,84.0)	54.6(–30.6,84.2)
Per-protocol population
EV71-associated disease	8/41	3522/3566	80.2(57.8,90.8)	80.4(58.2,90.8)
EV71-associated HFMD	3/30	705/686	90.3(68.0,97.0)	90.0(67.1,96.9)
EV71-associated other cases	5/11	2817/2880	53.5(−33.9,83.9)	54.3(–31.4,84.1)

#:Vaccination/Placebo; Data about VE-RCT was from the original phase 3 trial. VE-TND was calculated by Logistic regression. Cases were detected EV71 positive while controls were detected EV71 negative.

In the per-protocol population, 1424 participants seeked medical care for HFMD-like illness and 33 of them tested positive while 1391 tested negative. There were 5713 EV71-associated other cases, 16 of them tested positive and 5697 tested negative. The VE-TND against EV71-associated disease, EV71-associated HFMD and EV71-associated other cases are 80.2(95%CI: 57.8,90.8), 90.3(95%CI: 68.0,97.0), and 53.5(95%CI: −33.9,83.9), separately. Figure 1,2 summarizes the comparison of estimates and 95% CIs between VE-TND and VE-RCT.

Figure 1.

VE-TND and VE-RCT and 95% confidence intervals in mITT

Figure 2.

VE-TND and VE-RCT and 95% confidence intervals in PPS

4. Discussion

Overall, the results from the TND analysis for EV-71 vaccine effectiveness were similar to the original RCT efficacy results, both in modified intention-to-treat population and per-protocol population. Our research result is similar to the previous study in influenza and rotavirus vaccine.^15,17 G De Serres etc and Lauren M. Schwartz etc both used per-protocol population data to calculate the VE-TND and compared them with the results of corresponding RCTs. Their estimates of TND were not meaningfully different from efficacy results. Not only we received the same result in per-protocol population database, but also in modified intention-to-treat population. Thus, the TND method could be applied widely in evaluating effectiveness of EV-71 vaccine furthermore.

We reviewed three-phase 3 trials and found that the definitions about EV71 disease were similar because they all made a comprehensive definition about EV71 disease and HFMD. But in three TND studies, the enrolled criteria they used were different. In Henan’s study, they enrolled severe HFMD inpatients aged 6–71 months with comprehensive definition about EV71 disease. In Beijing’s study, they enrolled all suspected HFMD outpatients aged 6–60 months while in Guangxi’s study severe HFMD cases under 12 years old were enrolled. We suppose comprehensive definition about EV71 disease and HFMD should be more suitable because EV71 could cause various diseases more than HFMD. And, although most HFMD cases may have typical clinical symptoms, there are still some special and untypical cases so detailed and completed definition is needed. As Sullivan, S. G.’s reviews mentioned,¹⁸ in influenza TND research, definition of disease may affect positive and negative ratio of test results and then affect the estimated VE. Then biases could persist under the comprehensive definition.¹⁹ Thus, in EV71 TND studies, definition of disease needs to be further considered. The importance of clear and specific case definitions should not be overlooked.

Except the definition of disease, the severity of disease and study subjects(such as inpatient or outpatient) are also potential bias to VE.¹⁹ Disease outcomes may consist of a broader spectrum of diseases from mild to severe, and patients may be recruited from both inpatient or outpatient settings. Biases could persist under the test-negative design if broad variation in disease manifestation gives rise to differential healthcare-seeking behavior in terms of vaccination and care seeking for symptoms, and affect the probability of being tested.²⁰ Nevertheless, the choice of clinical symptoms for enrolled patients should ensure similarity between cases and controls and not be dependent on the vaccination or important confounders of the vaccination–disease relationship. Test-negative studies have typically, but not always, included persons seeking medical care during disease epidemic periods because calendar time is thus correlated with both vaccine uptake and with incidence of disease, analyses of test-negative data must control for calendar time.¹⁴ Another source of bias is misclassification of disease status due to the imperfect sensitivity and specificity of test method3. And imperfect specificity of influenza testing caused greater bias in the test-negative design than did imperfect sensitivity.²¹ In terms of distinguishing cases from controls, most TND studies defined vaccine-type positive as cases, where vaccines being evaluated were believed to have protective effect against the strains included in the vaccine. However, in the scenario where vaccination is thought to provide cross-protection against nonvaccine-types of the same pathogen, cases also included nonvaccine-types or all patients testing positive for the pathogen regardless of type. There is a lack of consensus on the choices of controls for each vaccine. A basic assumption of the test-negative design is that the risk of infections by nonvaccine-targeted causative pathogens resulting in similar clinical disease does not vary by vaccination status. This assumption may be violated if there is cross-protection in nonvaccine-type pathogens. Therefore, the choice to include vaccine-matched pathogens or not should depend on the public health question to be answered: are we interested in knowing by how much this vaccine will reduce the burden of the clinical disease of interest? Or are we interested in estimating the effectiveness of the vaccine only with respect to the pathogens it specifically targets? From a programmatic, policy, and public health perspective, the former argument is perhaps more relevant.¹⁹In general case–control studies, controls must be representative of the source population. As such, in test-negative designs controls should belong to the same source population from which cases were identified, and they should be individuals who probably would have been identified as cases got the targeted pathogen of interest. Test-negative controls are chosen from patients seeking medical care with similar clinical illness and tested negative from laboratory diagnosis. They provide reassurance that they emerge from the same source population as cases, and would have been suspected cases with the targeted pathogen. The main advantage of the TND is its convenience of access to get controls representative of the source population. However, TND has its assumptions. The core assumption of the TND is that vaccination does not affect the probability of other pathogen infection. The controls with EV-71 negative but other enterovirus positive may affect the validity and accuracy of TND.²² One of our limitations is that we did not demonstrate that EV-71 vaccine had no effect on EV-71-negative but other Enterovirus-positive disease with sensitivity analysis, although there had been studies indicating that EV-71 vaccine did not have cross-protection on other enterovirus.^6,7 In influenza vaccine and rotavirus vaccine TND studies, most of them have sensitivity analysis with different control groups.^23,24 Therefore, in EV-71 vaccine TND studies, sensitivity analysis is needed for further application.

Another limitation of our study is that we only have one RCT database. This RCT only contains participants aged from 6 to 35 months. In China, there are three kinds of EV-71 vaccine, two of which for 6–35 months and one of which for 6–71 months.^6–8 In the EV-71 vaccine effectiveness studies by TND,^5,9,10 between cases and controls, age level had significant statistical difference and was adjusted by multiple regression. They did not analyze the influence may caused by vaccine types although there has been a research shows that children aged 3–5 years old had non-inferior immunogenicity to that in infants aged 6–35 months.²⁵ In the RCT, characteristics between vaccine group and placebo group were balanced because of randomization. In this study, characteristics between cases and controls have no statistical differences and we didn’t adjust potential variates that may cause bias. Possibility it was from RCT database so cases and controls were balanced at the baseline. However, in general TND studies, cases, and controls usually have different baselines like gender composition, age level, or others.^26,27 Thence, multivariate logistic regression is usually used to adjust covariates like sex, age, education level, calendar time, and so on.

In conclusion, we have validated the TND approach for EV-71 vaccine by using RCT database. TND, as an observational design, appears valid in EV-71 vaccine effectiveness assessment but requires further consideration in confounding and bias.

Funding Statement

This work was supported by the Vaccine Clinical Evaluation Platform Construction Project under China National Science and Technology Major Projects [2018ZX09734004].

Authors’ contributions

Fengcai Zhu and Jingxin Li contributed to conception and design of the study; Li Zhang and Mingwei Wei contributed to collecting and analyzing the data; Li Zhang draft the manuscript; Jingxin Li, Mingwei Wei and Pengfei Jin revised the manuscript; All authors read and approved the final manuscript.

Disclosure of potential conflicts of interest

No potential conflicts of interest were disclosed.