Abstract
Objective
To actively solicit adverse events experienced in the days following immunisation with quadrivalent inactivated influenza vaccine using Australia’s near real-time, participant-based vaccine safety surveillance system, AusVaxSafety.
Design and setting
Observational cohort study conducted in 194 sentinel surveillance immunisation sites (primary care, hospital and community-based clinics) across Australia.
Participants
Individuals aged ≥6 months who received a routine seasonal influenza vaccine at a participating site (n=102 911) and responded to a survey (via short message service or email) sent 3 days after vaccination about adverse events experienced (n=73 892; 71.8%).
Main outcome measure
Near real-time and cumulative participant-reported rates of any adverse event, fever or medical attendance experienced within 3 days after vaccination overall, by brand, age, pregnancy status and concomitant vaccine receipt.
Results
Participant median age was 57 years (range: 6 months to 102 years); 58.1% (n=42 869) were female and 2.7% (n=2018) were pregnant. Near real-time fast initial response cumulative summation and Bayesian analyses of weekly event rates did not demonstrate a safety signal. Children aged 6 months to 4 years had higher event rates (522/6180; 8.4%) compared with older ages; participants aged ≥65 years reported fewer events (1695/28 154; 6.0%). There were no clinically significant differences in safety between brands, by age group or overall. Cumulative data analysis demonstrated that concomitant vaccination was associated with increased rates of fever (2.1% vs 0.8%) and medical attendance (0.8% vs 0.4%), although all rates were low and did not exceed expected levels.
Conclusions
Novel, postmarketing AusVaxSafety surveillance demonstrated comparable and expected safety outcomes for the 2017 quadrivalent inactivated influenza vaccine brands used in Australia. These near real-time, participant-reported data are expected to encourage confidence in vaccine safety and promote uptake.
Keywords: epidemiology, vaccine safety, active surveillance
Strengths and limitations of this study.
A large number of vaccinated individuals of all ages across Australia participated, leading to a greater ability to detect serious adverse events.
Comprehensive data enabled analysis of adverse events with respect to age, pregnancy, vaccine brand and concomitant vaccination with a wide variety of vaccines.
Safety signal detection was conducted in near real time using multiple statistical methods, with results reported to the public each week.
Individuals participating in active surveillance may be less inclined to report common and expected reactions, limiting the ability to compare reported adverse event rates with those from clinical trials.
Some outcomes of vaccine safety, such as participant-reported fever, are subjective and have not been verified.
Introduction
Influenza vaccines are given to hundreds of millions of people within short, fixed periods of time worldwide each year.1 This widespread use, coupled with the high degree of influenza vaccine variability, including multiple vaccine types (live, inactivated, subunit or adjuvanted), manufacturing processes (in eggs, cell lines or with recombinant techniques) and strain compositions (trivalent or quadrivalent, with the potential for vaccine viruses to change twice yearly across the Southern and Northern Hemisphere seasons), underscores the need for timely postmarketing vaccine safety surveillance. The European Medicines Agency (EMA) now requires manufacturers to address the paucity of clinical trial safety data available for vaccine changes by conducting enhanced postmarketing safety surveillance for seasonal influenza vaccines.2
AusVaxSafety, an automated, active vaccine safety surveillance system, reports near real-time, brand-specific data independent of manufacturers using participant-reported outcomes. AusVaxSafety was established to improve vaccine safety monitoring following recommendations of an independent inquiry into the unprecedented increase in febrile seizures observed in young Australian children in 2010, ultimately determined to be associated with one influenza vaccine brand (Fluvax/Afluria; bioCSL).3 This incident, which led to temporary nationwide suspension of paediatric influenza immunisation, resulted in a loss of confidence in influenza vaccines among consumers and immunisation providers and decreased influenza vaccine uptake.4 5
From 2014 to 2016 AusVaxSafety conducted influenza vaccine safety surveillance in 8184 children aged 6 months to 4 years.6 7 A retrospective analysis comparing safety profiles of trivalent inactivated influenza vaccine (TIIV) and quadrivalent inactivated influenza vaccine (QIIV) brands in 2015 and 2016 demonstrated that concomitant vaccine administration in young children was associated with increased fever and medical attendance (MA) rates after vaccination, although rates were low and within expected ranges.7 Importantly, detailed follow-up data on the small number of children who sought medical attention showed no serious or unexpected vaccine-associated adverse events following immunisation (AEFI).
In 2017, AusVaxSafety surveillance expanded to include influenza vaccine recipients of all ages. Here we provide an overview of AusVaxSafety’s weekly surveillance and a detailed analysis of cumulative (end of vaccine season) safety data by QIIV brand, age, pregnancy status and concomitant vaccine receipt.
Methods
AusVaxSafety active vaccine safety surveillance
Surveillance included individuals aged ≥6 months who received a 2017 seasonal influenza vaccine between 1 April and 31 August 2017 at one of 194 participating immunisation providers across Australia, including general practices, hospitals, community-based clinics and Aboriginal Medical Services. Annual influenza vaccination is recommended for all individuals aged 6 months and older who wish to protect themselves from influenza, but it is funded (available for free) under the Australian National Immunisation Program (NIP) for groups at increased risk of complications from influenza. These include individuals aged 65 years and older; Aboriginal and Torres Strait Islander people aged 6 months to 4 years and 15 years and older; pregnant women; and anyone 6 months and older who has a medical condition (including heart or lung disease, asthma, chronic neurological conditions, immune compromising conditions or other chronic illnesses such as diabetes).8 In 2017, one state (Western Australia) also funded influenza vaccine for all children aged 6 months to 4 years.
Most individuals were enrolled using the opt-out, computer-based monitoring platform SmartVax, which integrates with immunisation provider management software to issue automated surveys to vaccine recipients or their caregivers via short message service (SMS), as previously described.9 A minority of AusVaxSafety sites (n=30) used one of two alternative computer-based monitoring platforms—Vaxtracker10 (recipients aged 6 months to 4 years only) or Stimulated Telephone-Assisted Rapid Safety Surveillance (STARSS)11—to solicit influenza vaccine adverse events following opt-in enrolment.
Vaccinated individuals/caregivers received an SMS from their medical provider 3 days after vaccination inquiring about AEFI (‘We would like to know if there were any reactions to the vax. Please reply with JUST a Y or N.’). Those who responded ‘Y’ or ‘N’ were classified as participants, and those who responded ‘Y’ were then asked whether or not the event was medically attended. ‘Yes’ responders were asked to detail the adverse event(s) and/or medical attention in a short online survey, which listed a range of symptoms and asked participants to tick all symptoms experienced. As children aged 6 months to 8 years and immunocompromised individuals of any age are recommended to receive two vaccine doses at least 4 weeks apart when first immunised, some may have been represented by more than one record.
Primary outcomes surveyed were reports of any event (yes or no), fever (solicited in the online survey) and MA (yes or no). Secondary outcomes (solicited in the online survey) were injection site (IS) pain, swelling and/or redness; tiredness/fatigue; headache; sleep pattern change; irritability; rash; vomiting; diarrhoea; rigours; non-responsiveness/loss of consciousness; and convulsions/seizures. Unsolicited symptoms were detailed by participants in free text.
Detailed clinical data from MAs were sought using additional information from participants’ immunisation providers and/or by a public health authority, who attempted to contact participants/caregivers to ascertain whether or not MAs were serious (as defined by the Australian Therapeutic Goods Administration).12
Patient involvement
The AusVaxSafety surveillance system does not specifically recruit patients but does rely on community participation. The majority of participants are included in the surveillance system because their primary care provider or immunisation clinic has installed the SmartVax data monitoring platform, which functions in conjunction with the clinic software. Where installed, SmartVax automatically sends text messages to all patients who receive any vaccine to seek information regarding any AEFI as a routine part of patient management and aftercare. In this study, we report only on patient responses regarding influenza vaccine. A small proportion of participant data are provided to AusVaxSafety via the Vaxtracker or STARSS data monitoring platforms which similarly survey individuals who have received an influenza vaccination from a participating provider or clinic. The data monitoring platforms were piloted and developed with feedback from users. The AusVaxSafety surveillance system Advisory Committee includes a consumer/patient representative. Surveillance results are uploaded to the AusVaxSafety website (www.ausvaxsafety.org.au) weekly and available to the public.
Near real-time reporting and analysis
Deidentified records (including demographic, immunisation visit and SMS/survey response data) were uploaded to the computer-based monitoring systems and exported weekly to the AusVaxSafety coordinating centre for aggregation and analysis. MA reports triggered clinical follow-up by designated public health authorities each weekday. Weekly analysis of cumulative data (received up to 5 days prior) for age and pregnancy-specific AEFI rates and participant demographic characteristics were reported in detail to the Australian Department of Health and summary results were published online each Friday (www.ausvaxsafety.org.au) from week 3 of surveillance for the duration of the surveillance period.
Weekly signal detection
Participant-reported rates of fever (for those aged 6 months to 4 years) and MA (for all participants, grouped by age: 6 months to 4 years, 5–64 years and ≥65 years; and pregnant participants) as a surrogate for serious adverse events (SAE)7 were considered the most objective outcome measures of vaccine safety and were monitored weekly using signal detection methods.
Fast initial response cumulative summation (FIR CUSUM) control charts monitored log-likelihood ratios of each event rate being at a maximum acceptable level versus expected level.13 Expected and maximum acceptable rates were set based on syntheses of clinical trial data and previous surveillance results.6 7 10 14–16 The expected MA rate was set at 1%, and the expected fever rate at 3%. Maximum acceptable rates were set at 3% and 10% for MA and fever, respectively. A safety signal is generated if the log-likelihood ratio (a measure of the degree to which the data are more consistent with an event rate equal to the maximum acceptable rate vs the expected rate) rises above a predetermined threshold. The threshold log-likelihood ratio was selected such that across 10 000 simulated vaccination seasons there would be ≥80% probability of signal generation within 3 weeks of commencement if the event rate is at the maximum acceptable level, and ≤2% probability of (false) signal generation over the entire season when the event rate is at the expected level.
Bayesian analysis was also performed weekly for robust, optimal estimation of the 95% credibility interval (CI) for true cumulative event rates. Beta distributions with means derived from 2016 surveillance data and literature review (MA: 1% for participants aged 6 months to 4 years; 0.3% for participants aged 5–64 years and ≥65 years; 1% for pregnant participants; and fever: 3% for participants aged 6 months to 4 years)7 were used as priors at the start of the 2017 season. Priors were updated with each week’s observed data and CIs from the posterior beta distribution were reported weekly.
End-of-surveillance cumulative analysis
Cumulative data were reported by epidemiological week and demographic information including age (6 months to 4 years, 5–14 years, 15–39 years, 40–64 years and ≥65 years), sex, pregnancy status (available for SmartVax participants only), Aboriginal and/or Torres Strait Islander (hereafter referred to as Indigenous) status and concomitant vaccine administration (defined as any additional vaccine(s) received at the same visit as influenza vaccine).
For any adverse event, fever and MA, rates were calculated for each age group and pairwise proportion tests with Holm adjustment for multiple comparisons were performed to compare AEFI rates between pairs of age groups using R V.3.4.2 (R Foundation for Statistical Computing, Vienna, Austria). AEFI rates in pregnant women were compared with those of non-pregnant female SmartVax participants of the same age range (15–49 years) using Pearson’s Χ2 test in Stata V.14.2 (StataCorp, College Station, TX, USA). Rates of primary and secondary outcomes were calculated by brand, and secondary outcomes were calculated for each age group and pregnant women.
Primary outcome AEFI rates were also calculated for age groups and pregnant women by vaccine brand and concomitant vaccine receipt (yes or no). The relative risk of each adverse event was compared for those receiving influenza vaccine plus any concomitant vaccine(s) versus influenza vaccine alone, and for those receiving FluQuadri versus Fluarix Tetra, using a generalised linear model with a log link and binomial distribution in Stata V.14.2.
Results
Weekly signal detection throughout 2017
No safety signals were detected by the FIR CUSUM method (online supplementary efigure 1). Weekly and cumulative Bayesian rates of fever and MA remained well below their respective maximum acceptable rates over the surveillance period: the cumulative (end-of-season) fever rate in children aged 6 months to 4 years was 2.3% (95% posterior CI 2.0 to 2.7), while cumulative MA rates were 1.0% (95% CI 0.73 to 1.21) in children aged 6 months to 4 years, 0.5% (95% CI 0.41 to 0.55) in participants aged 5–64 years, 0.3% (95% CI 0.22 to 0.34) in participants aged ≥65 years and 0.5% (95% CI 0.26 to 0.87) in pregnant women. For those MAs that were followed up, none of these events were categorised as serious.
bmjopen-2018-023263supp001.pdf (453.6KB, pdf)
End-of-surveillance analysis
Over the surveillance period, 73 892 of 102 911 enrollees (71.8%) responded to the postvaccination SMS; over 95% of participants with response time available (n=71 093) responded on the same day of SMS receipt. Participants received one of four available QIIVs: Fluarix Tetra (GlaxoSmithKline; 45.3%), FluQuadri (Sanofi-Aventis; 42.3%), FluQuadri Junior (Sanofi-Aventis; 5.6%) or Afluria Quad (Seqirus; 6.8%); less than 1.0% received a vaccine whose brand could not be determined. Half of all vaccines were administered within 5 weeks of starting surveillance, with older participants (≥65 years) receiving vaccines earlier compared with young children (6 months to 4 years old) and pregnant women (figure 1).
Among all participants, 58.1% were female and the median age was 57 years (range: 6 months to 102 years). Two per cent (1156/58 145 with data available) were Indigenous, which is representative of the Australian national Aboriginal and/or Torres Strait Islander population (2.8%) (table 1). Among female participants aged 15–49 years for whom pregnancy status was available (98.6%), 15.2% (2018/13 242) were pregnant. Individuals aged ≥65 years represented the largest proportion of participants (38.1%; 33.6% aged 65–79 years and 4.5% aged ≥80 years). Approximately 14% of participants (10 428/73 892) received a concomitant vaccine, of which 86.6% received only one. The most commonly received concomitant vaccines are listed in table 1.
Table 1.
Variable | Description | n (%) |
Sex* | Male | 30 968 (41.9) |
Female | 42 869 (58.1) | |
Indigenous status† | Aboriginal | 1000 (1.7) |
Torres Strait Islander | 32 (0.1) | |
Both | 124 (0.2) | |
Total | 1156 (2.0) | |
Pregnant‡ | 2018 (2.8) | |
Age median (IQR; range) | 57 years (31–69 years; 6 months to 102 years) | |
Age group | 6 months to 4 years | 6180 (8.4) |
5–14 years | 4415 (6.0) | |
15–39 years | 13 434 (18.2) | |
40–64 years | 21 709 (29.4) | |
≥65 years | 28 154 (38.1) | |
Number of participants receiving concomitant vaccine(s) | 10 428 (14.1) | |
Most common concomitant vaccines by group§¶ | ||
Overall (n=10 428) |
23vPPV | 2756 (26.4) |
dTpa/dTpa-IPV | 2504 (24.0) | |
Zoster | 1708 (16.4) | |
6 months to 4 years (n=1295) |
DTPa-IPV | 268 (20.7) |
HibMenCCV+MMR | 235 (18.1) | |
MenBV | 206 (15.9) | |
DTPa+MMRV | 205 (15.8) | |
5–14 years (n=295) |
HPV | 46 (15.6) |
Typhoid+hepatitis A | 43 (14.6) | |
Hepatitis A | 39 (13.2) | |
MenBV | 34 (11.5) | |
15–39 years (n=2612) |
dTpa/dTpa-IPV | 1743 (66.7) |
Typhoid-hepatitis A | 94 (3.6) | |
Hepatitis A | 85 (3.3) | |
40–64 years (n=1516) |
dTpa/dTpa-IPV | 534 (35.2) |
23vPPV | 311 (20.5) | |
Typhoid | 87 (5.7) | |
≥65 years (n=4710) |
23vPPV | 2403 (51.0) |
Zoster | 1699 (36.1) | |
dTpa/dTpa-IPV | 220 (4.7) | |
Pregnant** (n=634) |
dTpa/dTpa-IPV | 633 (99.8) |
dTpa/dTpa-IPV+hepatitis B | 1 (0.2) |
*Sex available for n=73 837 participants.
†Indigenous status available for n=58 145 participants.
‡Pregnancy status available for n=72 951 participants (SmartVax only).
§The percentages listed under ‘concomitant vaccines’ are the percentage of all concomitant vaccine(s) administered per group.
¶+ indicates two separate vaccines administered concomitantly.
**Pregnant participants are also included in their respective age categories (age range: 15–49 years).
23vPPV, 23-valent pneumococcal polysaccharide vaccine; DTPa, Diphtheria tetanus acellular pertussis (for children aged <10 years); DTPa-IPV, DTPa-inactivated polio vaccine (for children aged <10 years); dTpa, diphtheria tetanus acellular pertussis (for individuals aged ≥10 years); dTpa-IPV, dTpa-inactivated polio vaccine (for individuals aged ≥10 years); HibMenCCV, Haemophilus influenzae type B meningococcal C conjugate vaccine; HPV, human papillomavirus; MenBV, meningococcal B vaccine; MMR, measles, mumps and rubella; MMRV, measles, mumps, rubella and varicella.
Compared with other age groups, children aged 6 months to 4 years were reported as having significantly higher rates of any adverse event, while participants aged ≥65 years reported events less often (table 2). Pregnant women reported significantly lower rates of any adverse event compared with non-pregnant women of the same age range (15–49 years; p=0.019, data not shown). Rates of more subjective secondary outcomes surveyed showed similar trends across age groups and by pregnancy status (online supplementary etable 1).
Table 2.
Type of adverse event* | Group (n/N (%)) | ||||||
6 months to 4 years | 5–14 years | 15–39 years | 40–64 years | ≥65 years | Pregnant† | Total | |
Any adverse event‡ | 522/6180 (8.4) |
295/4415 (6.7) |
836/13 434 (6.2) |
1533/21 709 (7.1) |
1695/28 154 (6.0) |
118/2018 (5.8) |
4881/73 892 (6.6) |
Fever‡ | 140/5979 (2.3) |
54/4266 (1.3) |
121/13 020 (0.9) |
190/20 953 (0.9) |
209/27 222 (0.8) |
20/1963 (1.0) |
714/71 440 (1.0) |
Medical attendance‡ | 59/6180 (1.0) |
21/4415 (0.5) |
75/13 434 (0.6) |
94/21 709 (0.4) |
77/28 154 (0.3) |
10/2018 (0.5) |
326/73 892 (0.4) |
*Denominators differ between any adverse event/medical attendance and fever because reports of fever are solicited in an online survey following the initial short message service (SMS) regarding an adverse event following immunisation (AEFI), and not all participants complete the survey.
†Pregnant participants are also included in their respective age categories (age range: 15–49 years). They are not compared with another group in this table.
‡P<0.001 for participants aged 6 months to 4 years compared with all other age groups.
bmjopen-2018-023263supp002.pdf (380.1KB, pdf)
Participants who received concomitant vaccine(s) had an elevated risk of reporting any adverse event and fever compared with participants who received influenza vaccine alone (table 3). This pattern was seen for all age groups, with the exception of fever in participants aged 15–39 years and pregnant women. Participants aged ≥40 years who received concomitant vaccine(s) reported MA at a significantly higher rate than those who received only an influenza vaccine.
Table 3.
Group | Type of adverse event* | ||||||||
Any adverse event (n/N (%)) | Fever (n/N (%)) | Medical attendance (n/N (%)) | |||||||
Influenza vaccine+concomitant vaccine(s) | Influenza vaccine alone | Relative risk (95% CI)† |
Influenza vaccine+concomitant vaccine(s) | Influenza vaccine alone | Relative risk (95% CI)† |
Influenza vaccine+concomitant vaccine(s) | Influenza vaccine alone | Relative risk (95% CI)† |
|
6 months to 4 years | 189/1295 (14.6) |
333/4885 (6.8) |
2.1 (1.8 to 2.5) |
58/1211 (4.8) |
82/4768 (1.7) |
2. 8 (2.0 to 3.9) |
15/1295 (1.2) |
44/4885 (0.9) |
1.3 (0.7 to 2.3) |
5–14 years | 33/295 (11.2) |
262/4120 (6.4) |
1.8 (1.3 to 2.5) |
9/281 (3.2) |
45/3985 (1.1) |
2.8 (1.4 to 5.7) |
2/295 (0.7) |
19/4120 (0.5) |
1.5 (0.3 to 6.3) |
15–39 years | 205/2612 (7.8) |
631/10 822 (5.8) |
1.4 (1.2 to 1.6) |
27/2491 (1.1) |
94/10 529 (0.9) |
1.2 (0.8 to 1.9) |
18/2612 (0.7) |
57/10 822 (0.5) |
1.3 (0.8 to 2.2) |
40–64 years | 138/1516 (9.1) |
1395/20 193 (6.9) |
1.3 (1.1 to 1.6) |
27/1456 (1.9) |
163/19 497 (0.8) |
2.2 (1.5 to 3.3) |
19/1516 (1.3) |
75/20 193 (0.4) |
3.4 (2.1 to 5.6) |
≥65 years | 568/4710 (12.1) |
1127/23 444 (4.8) |
2.5 (2.3 to 2.8) |
89/4439 (2.0) |
120/22 783 (0.5) |
3.8 (2.9 to 5.0) |
33/4710 (0.7) |
44/23 444 (0.2) |
3.7 (2.4 to 5.9) |
Pregnant‡ | 57/634 (9.0) |
61/1384 (4.4) |
2.0 (1.4 to 2.9) |
7/602 (1.2) |
13/1361 (1.0) |
1.2 (0.5 to 3.0) |
2/634 (0.3) |
8/1384 (0.6) |
0.6 (0.1 to 2.6) |
Total | 1133/10 428 (10.9) |
3748/63 464 (5.9) |
1.8 (1.7 to 2.0) |
210/9878 (2.1) |
504/61 562 (0.8) |
2.6 (2.2 to 3.1) |
87/10 428 (0.8) |
239/63 464 (0.4) |
2.2 (1.7 to 2.8) |
*Denominators differ between any adverse event/medical attendance and fever because reports of fever are solicited in an online survey following the initial short message service (SMS) regarding an adverse event following immunisation (AEFI), and not all participants complete the survey.
†Relative risk of any adverse event, fever or medical attendance for influenza vaccine administered with any concomitant vaccine(s) as compared with influenza vaccine administered alone.
‡Pregnant participants are also included in their respective age categories (age range: 15–49 years).
Brand-specific AEFI rates were similar, particularly for FluQuadri and Fluarix Tetra, the brands administered to the majority of participants (table 4, online supplementary etable 2).
Table 4.
Group | Type of adverse event* | ||||||||
Any adverse event (n/N (%)) | Fever (n/N (%)) | Medical attendance (n/N (%)) | |||||||
FluQuadri | Fluarix Tetra | Relative risk†
(95% CI) |
FluQuadri | Fluarix Tetra | Relative risk†
(95% CI) |
FluQuadri | Fluarix Tetra | Relative risk†
(95% CI) |
|
3–14 years‡ | 317/4076 (7.8) |
150/2203 (6.8) |
1.1 (1.0 to 1.4) |
68/3933 (1.7) |
29/2133 (1.4) |
1.3 (0.8 to 2.0) |
14/4076 (0.3) | 20/2203 (0.9) |
0.4 (0.2 to 0.8) |
15–39 years | 479/7484 (6.4) |
295/5059 (5.8) |
1.1 (1.0 to 1.3) |
65/7254 (0.9) |
46/4908 (0.9) |
1.0 (0.7 to 1.4) |
40/7484 (0.5) |
33/5059 (0.7) |
0.8 (0.5 to 1.3) |
40–64 years | 805/10 620 (7.6) |
607/9252 (6.6) |
1.16 (1.0 to 1.3) |
101/10 237 (1.0) |
76/8938 (0.9) |
1.2 (0.9 to 1.6) |
45/10 620 (0.4) |
42/9252 (0.5) |
0.9 (0.6 to 1.4) |
≥65 years | 638/8916 (7.2) |
912/16 938 (5.4) |
1.3 (1.2 to 1.5) |
81/8587 (0.9) |
117/16 426 (0.7) |
1.3 (1.0 to 1.8) |
22/8916 (0.2) |
43/16 938 (0.3) |
1.0 (0.6 to 1.6) |
Pregnant§ | 60/963 (6.2) |
43/901 (4.8) |
1.3 (0. 9 to 1.9) |
7/932 (0.8) |
10/885 (1.1) |
0.7 (0.3 to 1.7) |
4/963 (0.4) |
6/901 (0.7) |
0.6 (0.2 to 2.2) |
Total | 2239/31 096 (7.2) |
1964/33 452 (5.9) |
1.2 (1.2 to 1.3) |
317/30 136 (1.1) |
268/32 420 (0.8) |
1.3 (1.1 to 1.5) |
121/31 096 (0.4) |
138/33 452 (0.4) |
0.9 (0.7 to 1.2) |
*Denominators differ between any adverse event/medical attendance and fever because reports of fever are solicited in an online survey following the initial short message service (SMS) regarding an adverse event following immunisation (AEFI), and not all participants complete the survey.
†Relative risk of any adverse event or medical attendance for FluQuadri compared with Fluarix Tetra.
‡Fluarix Tetra (GlaxoSmithKline) and FluQuadri (Sanofi-Aventis) are each licensed for use in individuals aged ≥3 years. We excluded n=15 individuals aged 6 months to 2 years who were reported to have received Fluarix Tetra, and n=129 individuals aged 6 months to 2 years who were reported to have received FluQuadri, from this analysis.
§Pregnant participants are also included in their respective age categories (age range: 15–49 years).
bmjopen-2018-023263supp003.pdf (293.6KB, pdf)
Discussion
AusVaxSafety surveillance used almost 74 000 actively solicited participant-reported outcomes to demonstrate that the four brands of QIIV used in Australia in 2017 were safe and had low and comparable adverse event rates within expected ranges for all age groups and pregnant women.10 14–16 This novel system provided reassuring, locally derived feedback on vaccine safety in near real time to the public and immunisation providers as influenza vaccination was rolled out across Australia.17
Consistent with data published from vaccine clinical trials, the most common participant-reported event following influenza immunisation was IS pain (1.7% overall). IS pain was also commonly reported in clinical trials, but at higher rates than those demonstrated in this postmarketing surveillance. Clinical trials in children reported IS pain in approximately two-thirds of those aged 3–17 years18 with similarly high rates (up to 72.4%) in adults aged 18–60 years.19 20 This difference is likely due to more active solicitation of AEFI in clinical trials via daily diary cards, resulting in more complete reporting. Also, as AusVaxSafety participants may be informed of expected common vaccine reactions by their clinicians, these symptoms may be less likely to be reported. By comparison, data from both this postmarketing surveillance and clinical trials confirmed low rates of SAEs (0.4% for AusVaxSafety compared with 0.0%–2.3% for the clinical trials), despite differences in SAE definitions. Equally reassuring, both IS pain rates and SAEs among pregnant women in our surveillance were low and consistent with rates reported among participants of all ages.
Adverse event rates were similar for Fluarix Tetra (GlaxoSmithKline) and FluQuadri (Sanofi-Aventis), the two most used QIIVs in Australia in 2017. Though small and variable differences in AEFI rates between brands were reported, this is likely attributable to factors such as age and uncontrolled confounding, and is not of clinical significance. Ongoing brand-specific surveillance will provide valuable safety data in future years, especially as two new, more immunogenic vaccine types—the high dose TIIV (Fluzone High-Dose, Sanofi-Aventis) and the MF-59 adjuvanted influenza vaccine (Fluad, Seqirus)—are being included on the Australian NIP for adults aged ≥65 years from 2018.21
We previously observed that AusVaxSafety participants aged 6 months to 4 years who received influenza vaccine and another vaccine concomitantly (in particular diphtheria-tetanus-acellular pertussis (DTPa) inactivated poliovirus or meningococcal B vaccines) had significantly increased AEFI rates (especially fever) compared with those receiving influenza vaccine alone.6 7 The present analysis showed that AEFI were more common with concomitant vaccination among participants of all ages, including increased fever rates in both children and older adults and an increased risk of MA among those aged ≥40 years. The most commonly received concomitant vaccines were 23-valent pneumococcal vaccine, reduced antigen pertussis-containing vaccine (dTpa) and live attenuated zoster vaccine, which are reactogenic when administered individually.22–28 It has been shown that concomitant receipt of influenza and 13-valent pneumococcal vaccines results in increased local and systemic events, including fever among children,29–31 while such differences in AEFI rates were not observed with concomitant receipt of influenza and pertussis or zoster vaccines.32–35 Importantly, the increased risks of AEFI occurring with concomitant vaccination reported by AusVaxSafety—including those requiring MA—were low and likely not of clinical importance. This information may help providers to reassure patients who are receiving more than one vaccine at the same time that although they may have a slightly higher rate of side effects, the absolute rate is low overall. As more vaccines become available, assessment of adverse events associated with concomitant vaccination using surveillance like AusVaxSafety has the potential to contribute valuable detail to postmarketing pharmacovigilance.
To the best of our knowledge, AusVaxSafety is a unique postmarketing vaccine safety surveillance system in its high level of automation, patient and provider engagement and ability to provide data on vaccine brand-specific AEFI rates in near real time. However, since the EMA recommendation to provide annual brand-specific safety data, there has been an increase in pilot and feasibility studies of influenza vaccine safety surveillance methods and systems.36–40 Several are enhanced passive surveillance systems relying on patients returning adverse event reports via cards or telephone.36 38 Such systems are limited by potential under-reporting of events and are likely slower and more resource intensive as staff must enter AEFI details or conduct interviews. The Canadian National Vaccine Safety Network has conducted a small pilot of a mobile phone app for reporting adverse events.40 Eighty-six per cent of those replying to questions about the usability of an app for reporting AEFI said they would prefer an app to visiting a website. Nevertheless, investigators acknowledged that the app was limited by download requirements and low survey completion rates. The US Centers for Disease Control and Prevention’s Vaccine Safety Datalink (VSD), which uses large linked databases from healthcare organisations, conducts Rapid Cycle Analysis (RCA) to report AEFI rates in near real time but may be limited by delays between AEFI occurrence and electronic reporting to administrative data sets. VSD’s surveillance compares outcomes of interest in those who received the vaccine against the same outcomes experienced by a group of individuals who did not receive the vaccine (or in a control period for the vaccine recipient for self-controlled case series).41 42 While AusVaxSafety does not currently monitor some of the more severe adverse events that the VSD’s RCA may detect (particularly those occurring more than 3 days following vaccination), AusVaxSafety’s strength comes from its ability to quickly estimate the number of vaccine recipients who have (or have not) experienced an AEFI without relying on complex analytical methods.
There are several limitations of AusVaxSafety surveillance and the analysis in this report. First, self-report or parent/carer reports of outcomes gathered through participant-based feedback may be less accurate for common and expected reactions than those solicited from clinical trial participants or those detected by systems like the VSD. Second, though we have attempted to adjust for potential biases by reporting the more objective outcomes of MA and fever, it should be noted that participant-reported fever is subjective and has not been confirmed. Also, should a very serious event, such as death, occur after immunisation, an individual may not be capable of participating in AusVaxSafety surveillance; the system may therefore not identify the most SAEs. Third, not all adverse events are vaccine attributable, and AEFI rates may be affected by other illnesses with similar outcomes, for example, fever from intercurrent viral illness. Finally, in this report, data did not allow for comparisons of the reactogenicity of each non-influenza vaccine administered alone, and therefore conclusions made about increased adverse event rates associated with concomitant vaccination must be tempered. As AusVaxSafety expands to include safety surveillance for more vaccines, the system’s capacity to make such comparisons and provide data on the reactogenicity of more and varied vaccines will be enhanced.
In its requirement that annual enhanced postauthorisation influenza vaccine safety monitoring occur for all seasonal influenza vaccines, the EMA stated a preference for active surveillance.2 Data in this report and for other vaccines in the AusVaxSafety system (including pertussis, human papillomavirus and herpes zoster vaccines17) from hundreds of thousands of vaccinated participants since 2014 demonstrate the value of active vaccine safety surveillance systems. Age and brand-specific AEFI rates are available within weeks of the commencement of each year’s seasonal influenza immunisation programme, which ensure early detection of potential safety signals. This includes 2018 Southern Hemisphere seasonal influenza vaccines, for which data from more than 140 000 influenza vaccine recipients vaccinated between April and June 2018 demonstrate no safety concerns (data not shown, but available in summary form at www.ausvaxsafety.org.au).
Australia also has a comprehensive national passive vaccine safety surveillance system.43 However, all passive or spontaneous reporting systems have inherent limitations, including incomplete and under-reporting, stimulated reporting and limited data on vaccine brands. Importantly, with passive systems, it is often difficult to determine AEFI rates due to lack of denominator data on vaccines administered. In Australia, these limitations have especially affected passive influenza vaccine safety surveillance, and have led to previous difficulty in interpreting early or potential vaccine safety signals.44 In this context, AusVaxSafety provides important data to ensure confidence in the safety of vaccines in use in large populations in near real time.
Conclusions
Approximately 74 000 influenza vaccine recipients reported low adverse event rates following immunisation with the four brands of QIIV used in Australia in 2017. Concomitant vaccination was associated with an increased AEFI risk, but rates were still low and within expected ranges. Our novel participant-based postmarketing vaccine safety surveillance system is a valuable tool for monitoring immunisation, especially for annually changing influenza vaccines.
Supplementary Material
Acknowledgments
We thank the AusVaxSafety Expert Leadership Group Members for their contribution to oversight of the 2017 surveillance efforts. We also express our gratitude to the staff at participating hospitals, clinics, general practices and jurisdictional health departments, as well as the vaccine recipients who participated in this surveillance. AusVaxSafety thanks the three contributing data monitoring platforms: Vaxtracker, STARSS and, in particular, SmartVax, which provided the majority of the 2017 influenza vaccine surveillance data, and particularly acknowledges the contribution of and technological expertise of Ian Peters, codeveloper of SmartVax. Finally, we acknowledge Chloe Damon for her excellent work coordinating the AusVaxSafety active surveillance system.
Footnotes
Contributors: AJP, CG, HEQ and KKM made substantial contributions to the conception or design of the manuscript. AJP and CG were responsible for drafting the manuscript and conducting all data analyses, with the exception of signal detection analyses, which were performed by PF. TS, PJ and PF were responsible for the conceptualisation and execution of the weekly safety signal detection analyses. AL, as codeveloper of the SmartVax system, served as the system operator and advisor regarding SmartVax data. PC served as the system operator and advisor regarding Vaxtracker data. PC, AL, CCB, MSG, TS and KKM were integral to the design and development of the AusVaxSafety vaccine safety surveillance system and served as key vaccine safety experts. All authors made substantial contributions to the analysis and interpretation of data for the work and revised the manuscript critically for important intellectual content. All authors had final approval of the version to be published and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. This manuscript was written on behalf of the AusVaxSafety Expert Leadership Group: Jim Buttery, Nigel Crawford, David Durrheim, Paul Effler and Nicholas Wood.
Funding: AusVaxSafety surveillance was funded under a contract with the Australian Government Department of Health.
Competing interests: All authors are either located at organisations that hold the AusVaxSafety contract from the Australian Government Department of Health or are subcontract holders.
Patient consent: Not required.
Ethics approval: The AusVaxSafety surveillance system and its data monitoring platforms operate nationally under human research ethical approval obtained from the Sydney Children’s Hospital Network (HREC/16/SCHN/19) and the Royal Australian College of General Practitioners National Research and Evaluation Ethics Committee (NREEC15-007).
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: AusVaxSafety compiles ongoing, deidentified surveillance data of patient-reported adverse events for specific vaccines as contracted by the Australian Government Department of Health. Summarised results are publicly available on the AusVaxSafety website (www.ausvaxsafety.org.au) but AusVaxSafety data sets are not publicly available.
References
- 1. Palache A, Abelin A, Hollingsworth R, et al. . Survey of distribution of seasonal influenza vaccine doses in 201 countries (2004-2015): The 2003 World Health Assembly resolution on seasonal influenza vaccination coverage and the 2009 influenza pandemic have had very little impact on improving influenza control and pandemic preparedness. Vaccine 2017;35:4681–6. 10.1016/j.vaccine.2017.07.053 [DOI] [PubMed] [Google Scholar]
- 2. Agency EM. Guideline on influenza vaccines non-clinical and clinical module. London, UK: European Medicines Agency, 2016. [Google Scholar]
- 3. Armstrong PK, Dowse GK, Effler PV, et al. . Epidemiological study of severe febrile reactions in young children in Western Australia caused by a 2010 trivalent inactivated influenza vaccine. BMJ Open 2011;1:e000016 10.1136/bmjopen-2010-000016 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4. Blyth CC, Richmond PC, Jacoby P, et al. . The impact of pandemic A(H1N1)pdm09 influenza and vaccine-associated adverse events on parental attitudes and influenza vaccine uptake in young children. Vaccine 2014;32:4075–81. 10.1016/j.vaccine.2014.05.055 [DOI] [PubMed] [Google Scholar]
- 5. The Royal Children’s Hospital Melbourne. Vaccination: perspectives of Australian parents. The Royal Children’s Hospital National Child Health Poll, 2017. [Google Scholar]
- 6. Pillsbury A, Cashman P, Leeb A, et al. . Real-time safety surveillance of seasonal influenza vaccines in children, Australia, 2015. Euro Surveill 2015;20 10.2807/1560-7917.ES.2015.20.43.30050 [DOI] [PubMed] [Google Scholar]
- 7. Pillsbury A, Quinn H, Cashman P, et al. . Active SMS-based influenza vaccine safety surveillance in Australian children. Vaccine 2017;35:7101–6. 10.1016/j.vaccine.2017.10.091 [DOI] [PubMed] [Google Scholar]
- 8. Australian Technical Advisory Group on Immunisation (ATAGI). The Australian Immunisation Handbook. 10th ed Canberra: ACT: The Australian Government Department of Health and Ageing, 2017. [Google Scholar]
- 9. Leeb A, Regan AK, Peters IJ, et al. . Using automated text messages to monitor adverse events following immunisation in general practice. Med J Aust 2014;200:416–8. 10.5694/mja13.11166 [DOI] [PubMed] [Google Scholar]
- 10. Cashman P, Moberley S, Dalton C, et al. . Vaxtracker: Active on-line surveillance for adverse events following inactivated influenza vaccine in children. Vaccine 2014;32:5503–8. 10.1016/j.vaccine.2014.07.061 [DOI] [PubMed] [Google Scholar]
- 11. The University of Adelaide, 2015. About STARSS Adelaide http://www.adelaide.edu.au/trials/starss/about-starss (accessed 9 May 2017).
- 12. Therapeutic Goods Administration (TGA). Pharmacovigilance responsibilities of medicine sponsors: Australian recommendations and requirements: Australian Government Department of Health; https://www.tga.gov.au/publication/pharmacovigilance-responsibilities-medicine-sponsors (accessed 27 Jun 2018). [Google Scholar]
- 13. Grigg OA, Farewell VT, Spiegelhalter DJ. Use of risk-adjusted CUSUM and RSPRT charts for monitoring in medical contexts. Stat Methods Med Res 2003;12:147–70. 10.1177/096228020301200205 [DOI] [PubMed] [Google Scholar]
- 14. Blyth CC, Markus TY, Effler PV, et al. . Ensuring safety of the 2011 trivalent influenza vaccine in young children. Med J Aust 2011;195:52. [DOI] [PubMed] [Google Scholar]
- 15. Wood NJ, Blyth CC, Willis GA, et al. . The safety of seasonal influenza vaccines in Australian children in 2013. Med J Aust 2014;201:596–600. 10.5694/mja13.00097 [DOI] [PubMed] [Google Scholar]
- 16. Li-Kim-Moy J, Yin JK, Rashid H, et al. . Systematic review of fever, febrile convulsions and serious adverse events following administration of inactivated trivalent influenza vaccines in children. Euro Surveill 2015;20:21159 10.2807/1560-7917.ES2015.20.24.21159 [DOI] [PubMed] [Google Scholar]
- 17. National Centre for Immunisation Research and Surveillance (NCIRS). AusVaxSafety. Sydne: NCIRS, 2017. http://ausvaxsafety.org.au (accessed 16 Jan 2018). [Google Scholar]
- 18. Langley JM, Wang L, Aggarwal N, et al. . Immunogenicity and reactogenicity of an inactivated quadrivalent influenza vaccine administered intramuscularly to children 6 to 35 months of age in 2012-2013: a randomized, double-blind, controlled, multicenter, multicountry, clinical trial. J Pediatric Infect Dis Soc 2015;4:242–51. 10.1093/jpids/piu098 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19. Beran J, Peeters M, Dewé W, et al. . Immunogenicity and safety of quadrivalent versus trivalent inactivated influenza vaccine: a randomized, controlled trial in adults. BMC Infect Dis 2013;13:224 10.1186/1471-2334-13-224 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20. Pépin S, Donazzolo Y, Jambrecina A, et al. . Safety and immunogenicity of a quadrivalent inactivated influenza vaccine in adults. Vaccine 2013;31:5572–8. 10.1016/j.vaccine.2013.08.069 [DOI] [PubMed] [Google Scholar]
- 21. Australian Government Department of Health and Ageing. Ground-breaking flu vaccines to protect millions of Aussies. Canberra: Australian Government Department of Health and Ageing, 2018. [Google Scholar]
- 22. Hammitt LL, Bulkow LR, Singleton RJ, et al. . Repeat revaccination with 23-valent pneumococcal polysaccharide vaccine among adults aged 55-74 years living in Alaska: no evidence of hyporesponsiveness. Vaccine 2011;29:2287–95. 10.1016/j.vaccine.2011.01.029 [DOI] [PubMed] [Google Scholar]
- 23. Musher DM, Manoff SB, McFetridge RD, et al. . Antibody persistence ten years after first and second doses of 23-valent pneumococcal polysaccharide vaccine, and immunogenicity and safety of second and third doses in older adults. Hum Vaccin 2011;7:919–28. 10.4161/hv.7.9.15996 [DOI] [PubMed] [Google Scholar]
- 24. Booy R, Van der Meeren O, Ng SP, Sp N, et al. . A decennial booster dose of reduced antigen content diphtheria, tetanus, acellular pertussis vaccine (Boostrix™) is immunogenic and well tolerated in adults. Vaccine 2010;29:45–50. 10.1016/j.vaccine.2010.10.025 [DOI] [PubMed] [Google Scholar]
- 25. Halperin SA, McNeil S, Langley J, et al. . Tolerability and antibody response in adolescents and adults revaccinated with tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis vaccine adsorbed (Tdap) 4-5 years after a previous dose. Vaccine 2011;29:8459–65. 10.1016/j.vaccine.2011.07.068 [DOI] [PubMed] [Google Scholar]
- 26. Halperin SA, Sweet L, Baxendale D, et al. . How soon after a prior tetanus-diphtheria vaccination can one give adult formulation tetanus-diphtheria-acellular pertussis vaccine? Pediatr Infect Dis J 2006;25:195–200. 10.1097/01.inf.0000202082.56403.c4 [DOI] [PubMed] [Google Scholar]
- 27. Talbot EA, Brown KH, Kirkland KB, et al. . The safety of immunizing with tetanus-diphtheria-acellular pertussis vaccine (Tdap) less than 2 years following previous tetanus vaccination: Experience during a mass vaccination campaign of healthcare personnel during a respiratory illness outbreak. Vaccine 2010;28:8001–7. 10.1016/j.vaccine.2010.09.034 [DOI] [PubMed] [Google Scholar]
- 28. Oxman MN, Levin MJ, Johnson GR, et al. . A vaccine to prevent herpes zoster and postherpetic neuralgia in older adults. N Engl J Med 2005;352:2271–84. 10.1056/NEJMoa051016 [DOI] [PubMed] [Google Scholar]
- 29. Frenck RW, Gurtman A, Rubino J, et al. . Randomized, controlled trial of a 13-valent pneumococcal conjugate vaccine administered concomitantly with an influenza vaccine in healthy adults. Clin Vaccine Immunol 2012;19:1296–303. 10.1128/CVI.00176-12 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30. Schwarz TF, Flamaing J, Rümke HC, et al. . A randomized, double-blind trial to evaluate immunogenicity and safety of 13-valent pneumococcal conjugate vaccine given concomitantly with trivalent influenza vaccine in adults aged ≥65 years. Vaccine 2011;29:5195–202. 10.1016/j.vaccine.2011.05.031 [DOI] [PubMed] [Google Scholar]
- 31. Van Buynder PG, Frosst G, Van Buynder JL, et al. . Increased reactions to pediatric influenza vaccination following concomitant pneumococcal vaccination. Influenza Other Respir Viruses 2013;7:184–90. 10.1111/j.1750-2659.2012.00364.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Sukumaran L, McCarthy NL, Kharbanda EO, et al. . Safety of tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis and influenza vaccinations in pregnancy. Obstet Gynecol 2015;126:1069–74. 10.1097/AOG.0000000000001066 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. Levin MJ, Buchwald UK, Gardner J, et al. . Immunogenicity and safety of zoster vaccine live administered with quadrivalent influenza virus vaccine. Vaccine 2018;36:179–85. 10.1016/j.vaccine.2017.08.029 [DOI] [PubMed] [Google Scholar]
- 34. Kerzner B, Murray AV, Cheng E, et al. . Safety and immunogenicity profile of the concomitant administration of ZOSTAVAX and inactivated influenza vaccine in adults aged 50 and older. J Am Geriatr Soc 2007;55:1499–507. 10.1111/j.1532-5415.2007.01397.x [DOI] [PubMed] [Google Scholar]
- 35. Weston WM, Chandrashekar V, Friedland LR, et al. . Safety and immunogenicity of a tetanus toxoid, reduced diphtheria toxoid, and acellular pertussis vaccine when co-administered with influenza vaccine in adults. Hum Vaccin 2009;5:858–66. 10.4161/hv.9961 [DOI] [PubMed] [Google Scholar]
- 36. de Lusignan S, Dos Santos G, Correa A, et al. . Post-authorisation passive enhanced safety surveillance of seasonal influenza vaccines: protocol of a pilot study in England. BMJ Open 2017;7:e015469 10.1136/bmjopen-2016-015469 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37. Stockwell MS, Marchant CD, Wodi AP, et al. . A multi-site feasibility study to assess fever and wheezing in children after influenza vaccines using text messaging. Vaccine 2017;35:6941–8. 10.1016/j.vaccine.2017.10.073 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 38. Bricout H, Chabanon AL, Souverain A, et al. . Passive enhanced safety surveillance for Vaxigrip and Intanza 15 µg in the United Kingdom and Finland during the northern hemisphere influenza season 2015/16. Euro Surveill 2017;22 10.2807/1560-7917.ES.2017.22.18.30527 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 39. Stuurman AL, Verstraeten T, De Schryver A. Rapid assessment of the reactogenicity of a 2016-2017 seasonal influenza vaccine: results from a feasibility study. Expert Rev Vaccines 2017;16:187–91. 10.1080/14760584.2017.1264272 [DOI] [PubMed] [Google Scholar]
- 40. Wilson K, Atkinson KM, Westeinde J, et al. . An evaluation of the feasibility and usability of a proof of concept mobile app for adverse event reporting post influenza vaccination. Hum Vaccin Immunother 2016;12:1738–48. 10.1080/21645515.2016.1152434 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41. McNeil MM, Gee J, Weintraub ES, et al. . The Vaccine Safety Datalink: successes and challenges monitoring vaccine safety. Vaccine 2014;32:5390–8. 10.1016/j.vaccine.2014.07.073 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Greene SK, Kulldorff M, Lewis EM, et al. . Near real-time surveillance for influenza vaccine safety: proof-of-concept in the vaccine safety datalink project. Am J Epidemiol 2010;171:177–88. 10.1093/aje/kwp345 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43. Dey A, Wang H, Quinn H, et al. . Annual report: surveillance of adverse events following immunisation in Australia, 2015. Commun Dis Intell 2017;41:E264–E278. [PubMed] [Google Scholar]
- 44. Clothier HJ, Crawford N, Russell MA, et al. . Allergic adverse events following 2015 seasonal influenza vaccine, Victoria, Australia. Euro Surveill 2017;22 10.2807/1560-7917.ES.2017.22.20.30535 [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
bmjopen-2018-023263supp001.pdf (453.6KB, pdf)
bmjopen-2018-023263supp002.pdf (380.1KB, pdf)
bmjopen-2018-023263supp003.pdf (293.6KB, pdf)