Abstract
Background
Invasive pneumococcal disease (IPD) in people ≥60 years old is on the rise in Germany. There has been a recommendation for pneumococcal vaccination in this age group since 1998.
Methods
We determined the vaccination status of people ≥60 years old with IPD in Germany. We assessed vaccine effectiveness (VE) of the recommended 23-valent polysaccharide vaccine (PPV23) against IPD using the indirect cohort method.
Results
The rate of pneumococcal vaccination in older adults with IPD is low, 26%, with only 16% of people receiving a pneumococcal vaccine within five years of the IPD episode. Age- and gender- adjusted vaccine effectiveness (VE) for PPV23 was 37% (95% confidence interval 12% - 55%). For people vaccinated with PPV23 less than two years prior to IPD, VE was -20% (-131% - 34%), between two and four years prior to IPD, VE was 56% (20% - 76%), and 47% (17% - 63%) for those vaccinated ≥5 five years ago. Excluding serotype 3, overall VE for the remaining serotypes in PPV23 was 63% (49% - 74%). For people receiving PPV23 within the past two years, VE against all serotypes except 3 was 49% (12% - 71%); for people vaccinated between two and four years prior to IPD 66% (37% - 82%); for those vaccinated ≥five years ago, 69% (50% - 81%). VE of PPV23 against serotype 3 IPD only was -110% (-198% - -47%).
Conclusions
To reduce IPD in older adults in Germany, we must increase the rate of pneumococcal vaccine uptake. For 22/23 serotypes, PPV23 was effective. Serotype 3 remains a major problem.
Funding
This work was supported by an investigator-initiated research grant from Pfizer.
Keywords: Pneumococcal disease, Pneumococcal vaccination, PPV23, PCV13
Research in context.
Evidence before this study
23-valent pneumococcal polysaccharide vaccines (PPV23) have been in use for decades, and in Germany, they were first recommended in 1998 for adults 60 years of age and older. Despite this longstanding recommendation, the incidence of invasive pneumococcal disease in older adults has increased, which is particularly noteworthy given the aging population in Germany. The German National Reference Center for Streptococci receives pneumococcal isolates from invasive pneumococcal disease cases throughout Germany, but the vaccination status of the adults with invasive pneumococcal disease is not known.
Added value of this study
We determined the vaccination status for invasive pneumococcal disease cases occurring in adults 60 years of age and older in Germany, received between January 1, 2018 and December 31, 2019. For people for whom we could determine a definite pneumococcal vaccination status (n = 928), we found that 26% had ever received a pneumococcal vaccine, with only 16% receiving a pneumococcal vaccine within the past 5 years. We then estimated vaccine effectiveness of PPV23 using the indirect cohort method. Excluding serotype 3, for which there was no effectiveness, PPV23 provided fair protection against vaccine-serotype invasive pneumococcal disease, with 63% effectiveness.
Implications of all the available evidence
We confirm that, among older adults with invasive pneumococcal disease, the rate of pneumococcal vaccine uptake is low. The available pneumococcal vaccines provide protection against invasive pneumococcal disease, which has a high (30–66%) mortality rate in older adults. Additional efforts to improve vaccine uptake in older adults are needed to reduce the burden of disease in this growing, vulnerable population.
Alt-text: Unlabelled box
1. Introduction
The worldwide disease burden of invasive pneumococcal disease (IPD) among the elderly remains high. The most common presentation of pneumococcal disease in this age group is community acquired pneumonia (CAP). And although the larger part of CAP is non-bacteremic, an estimated 3–6% [1,2] of CAP cases progress to invasive disease, with pneumococci entering the blood stream (bacteremic pneumonia). An estimated 50% of pneumonia in the elderly is caused by pneumococci [3], [4], [5], with serotype 3 as the most prevalent serotype [6], [7], [8]. Mortality associated with IPD and pneumococcal CAP both increase in older age groups, with case fatality estimates for bacteremic pneumonia ranging from 11.9% to 20% in people ≥65 years of age and from 37.7% to 66.7% in people ≥80 years of age [9], [10], [11], [12]. Pneumococcal polysaccharide vaccines were introduced as early as 1977 (a 14-valent vaccine, PPV14) and 1983 (a 23-valent vaccine, PPV23) [13,14], but incidence and mortality of pneumococcal disease in older adults have remained largely unchanged. Apart from the broadly discussed effectiveness of PPV23, which is currently recommended for the elderly in many countries, there are reports of low uptake of adult pneumococcal vaccination [15,16].
To address this important cause of morbidity and mortality in older adults in Germany, the National Immunization Technical Advisory Group (STIKO) of the Robert Koch Institute first recommended that all adults 60 years and older receive a dose of PPV23 in 1998. The recommendation has been reassessed periodically, and currently calls for vaccination with PPV23 for all adults 60 years and older, with possible repeat doses at ≥6-year intervals according to perceived necessity (Supplemental Table 1). Individuals with congenital immune defects or immune suppression, as well as individuals with increased risk for pneumococcal meningitis, are recommended to receive sequential vaccination with PCV13 followed by PPV23 6–12 months later. Individuals with other chronic diseases are recommended to receive one dose of PPV23 [17].
There is not a single-product national immunization plan for pneumococcal vaccines in Germany: individual providers (in consultation with their patients) are allowed to choose which vaccine is used (within the recommendations) after which reimbursement by health insurance companies is guaranteed. STIKO issues general recommendations for all German federal states. However, some federal states also have their own vaccine committees, which can issue different recommendations. One federal state out of 16 (Saxony), representing 5% of the German population, has recommended sequential vaccination with the 13-valent pneumococcal conjugate vaccine (PCV13) followed by PPV23 for all adults 60 years and older with and without underlying health conditions, though PCV13 costs are not reimbursed [18].
Pneumococcal conjugate vaccines have been recommended for all children under the age of two in Germany since July 2006. Vaccine-serotype IPD has decreased across all age groups since the onset of the infant vaccination program, though the effects have not been as pronounced in older age groups [19].
Considering the forthcoming higher-valent pneumococcal conjugate vaccines [20,21] and the increasing burden of IPD in older adults [19], the recommendation may be re-evaluated soon. In this study, we wished to determine the rate of pneumococcal vaccine uptake among older adults with IPD, and assess vaccine effectiveness (VE) against vaccine type IPD in this population.
2. Materials and methods
2.1. Surveillance methods
We performed a two-year prospective survey of pneumococcal and influenza vaccination status for adults ≥ 60 years old with IPD for whom a pneumococcal isolate was sent to the German National Reference Center for Streptococci (GNRCS) between January 1, 2018 and December 31, 2019. Isolates were sent as part of a nationwide, voluntary surveillance of IPD, which is estimated to receive around half of all invasive pneumococcal isolates in Germany [22]. Although it is requested on the case report form, the GNRCS IPD questionnaires that correspond to the isolates rarely contain information about vaccination status. The vaccination status of the patient is not known by the diagnostic laboratory that sends the isolate, because Germany lacks a central patient registry. For all pneumococcal isolates originating from IPD cases in people 60 years and older, we contacted clinical and diagnostic laboratories, hospital admissions departments, and primary care providers by telephone call to ascertain the vaccination status of patients with IPD. We received approval to conduct this study from the ethics committee of the University Hospital RWTH Aachen, EK 035/18.
This work was supported by an investigator-initiated research grant from Pfizer. SP was supported by a NIH T32 fellowship, #2 T32AI7210–36A1. The funders had no role in the design and conduct of the study, and they had no role in the retrospective analyses.
2.2. Analytical methods
Pneumococcal isolates were identified and serotyped as previously described [19] using Neufeld's Quellung method with sera from the Statens Serum Institute (SSI Diagnostica A/S, Denmark).
Records of IPD cases were grouped by vaccination status (unknown, not vaccinated, vaccinated with PCV13, vaccinated with PPV23, vaccinated with both PCV13 and PPV23). VE was calculated by comparing the vaccination status of people with vaccine-serotype (VT) IPD to the vaccination status of people with non-vaccine serotype (NVT) IPD [23]. Patients who received both PPV23 and PCV13 were excluded from VE calculations of both vaccines. VE was adjusted for age and gender with Firth's bias-reduced logistic regression [24,25], where VT IPD was the dependent variable and the patient's vaccination status, age in years, and gender were independent variables.
We also stratified patients by the time between vaccination and IPD: less than two years prior to onset of IPD, between two and four years prior to IPD, and those vaccinated ≥ five years prior to IPD, and repeated the analyses. We also stratified patients by age, from 60 to 69, 70–79, and ≥80, and repeated the analyses. VT IPD was defined for PPV23 as serotypes (1, 2, 3, 4, 5, 6B, 7F, 8, 9N, 9V, 10A, 11A, 12F, 14, 15B, 17F, 18C, 19F, 19A, 20, 22F, 23F, and 33F) and for PCV13 as serotypes (1, 3, 4, 5, 6A, 6B, 7F, 9V, 14, 18C, 19A, 19F, and 23F). Because of previous work [3,26,27] indicating poor VE against serotype 3, we also calculated VE for all serotypes except 3, and for serotype 3 by itself. We also calculated VE for all serotypes except 3 and 8, and serotype 8 separately, due to the increasing prevalence of this serotype in Europe and in older adults [28]. We assessed correlation between pneumococcal vaccination status and influenza vaccination status with Spearman correlations. All analyses were performed in R (The R Foundation for Statistical Computing, Version 3.6.1) using the package logistf [25].
3. Results
From January 1, 2018 to December 31, 2019 the GNRCS received 6310 IPD isolates from individuals 16 years and older. Of these, 4642 (74%) were samples from IPD in people 60 years and older. We were able to determine the vaccination status for 928 people (20%), 243 (26%) of whom had ever received a pneumococcal vaccine and 146 (16%) of whom had received a pneumococcal vaccine within the five years prior to onset of IPD (Fig. 1). 67 people who were not vaccinated at the time of their IPD episode (therefore counted as unvaccinated) were then vaccinated after recovery. 11 people were vaccinated with both PPV23 and PCV13 and were excluded from the primary analyses of VE.
Fig. 1.
Study population of invasive pneumococcal disease cases from people ≥60 years old in Germany, 2018–2019. We used 928 cases in the full analysis for vaccine effectiveness calculations.
We compared the serotype distribution of isolates for which we could not determine a vaccination status with isolates with a known vaccination status, and for isolates from labs which declined to participate in the study. There appeared to be no major differences in the serotype prevalence between these groups (Fig. 2). Vaccine-type and individual serotypes were stratified by vaccination status, age group, and time since vaccination (Tables 1, 2, 3). Serotype distributions by vaccination status (Table 1) and for PPV23-vaccinated people by age group (Table 2) show that serotype 3 remains a dominant cause of IPD for both vaccinated and unvaccinated people ≥60 and older.
Fig. 2.
Percentage of isolates (by serotype) in the GNRCS collection received from January 1 2018, to December 31 2019, from adults ≥60 with invasive pneumococcal disease, stratified by study participation.
Table 1.
Isolates from invasive pneumococcal disease cases in people ≥60 in Germany, by serotype and vaccination status.
| Serotype | Unvaccinated |
PPV23 |
PCV13 |
PPV23 + PCV13 |
Total |
|||||
|---|---|---|---|---|---|---|---|---|---|---|
| n | % | n | % | n | % | n | % | n | % | |
| 3 | 131 | 19 | 66 | 33 | 5 | 15 | 3 | 27 | 205 | 22 |
| 4 | 5 | 1 | 0 | 0 | 1 | 3 | 0 | 0 | 6 | 1 |
| 6A | 6 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 7 | 1 |
| 6B | 2 | 0 | 0 | 0 | 0 | 0 | 1 | 9 | 3 | 0 |
| 6C | 10 | 1 | 5 | 3 | 0 | 0 | 0 | 0 | 15 | 2 |
| 6D | 2 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 2 | 0 |
| 7B | 6 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 7 | 1 |
| 7C | 2 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 3 | 0 |
| 7F | 2 | 0 | 0 | 0 | 1 | 3 | 0 | 0 | 3 | 0 |
| 8 | 73 | 11 | 11 | 6 | 4 | 12 | 0 | 0 | 88 | 9 |
| 9N | 51 | 7 | 8 | 4 | 4 | 12 | 1 | 9 | 64 | 7 |
| 9V | 3 | 0 | 2 | 1 | 0 | 0 | 0 | 0 | 5 | 1 |
| 10A | 31 | 5 | 7 | 4 | 1 | 3 | 1 | 9 | 40 | 4 |
| 10B | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 |
| 11A | 15 | 2 | 4 | 2 | 0 | 0 | 0 | 0 | 19 | 2 |
| 12A | 1 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 2 | 0 |
| 12F | 39 | 6 | 1 | 1 | 2 | 6 | 0 | 0 | 42 | 5 |
| 13 | 1 | 0 | 0 | 0 | 1 | 3 | 0 | 0 | 2 | 0 |
| 14 | 5 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 6 | 1 |
| 15A | 22 | 3 | 7 | 4 | 0 | 0 | 0 | 0 | 29 | 3 |
| 15B | 16 | 2 | 3 | 2 | 0 | 0 | 0 | 0 | 19 | 2 |
| 15C | 7 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 7 | 1 |
| 16F | 15 | 2 | 4 | 2 | 0 | 0 | 0 | 0 | 19 | 2 |
| 17F | 4 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 4 | 0 |
| 18A | 3 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 3 | 0 |
| 18C | 4 | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 5 | 1 |
| 19A | 33 | 5 | 6 | 3 | 1 | 3 | 0 | 0 | 40 | 4 |
| 19F | 12 | 2 | 1 | 1 | 0 | 0 | 0 | 0 | 13 | 1 |
| 20 | 12 | 2 | 2 | 1 | 0 | 0 | 0 | 0 | 14 | 2 |
| 21 | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 22F | 57 | 8 | 11 | 6 | 2 | 6 | 0 | 0 | 70 | 8 |
| 23A | 9 | 1 | 14 | 7 | 2 | 6 | 1 | 9 | 26 | 3 |
| 23B | 17 | 2 | 8 | 4 | 2 | 6 | 1 | 9 | 28 | 3 |
| 23F | 3 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 4 | 0 |
| 24F | 14 | 2 | 6 | 3 | 0 | 0 | 1 | 9 | 21 | 2 |
| 28A | 2 | 0 | 2 | 1 | 0 | 0 | 0 | 0 | 4 | 0 |
| 28F | 1 | 0 | 1 | 1 | 0 | 0 | 1 | 9 | 3 | 0 |
| 31 | 10 | 1 | 4 | 2 | 0 | 0 | 0 | 0 | 14 | 2 |
| 33F | 12 | 2 | 2 | 1 | 1 | 3 | 1 | 9 | 16 | 2 |
| 34 | 1 | 0 | 3 | 2 | 1 | 3 | 0 | 0 | 5 | 1 |
| 35B | 13 | 2 | 2 | 1 | 1 | 3 | 0 | 0 | 16 | 2 |
| 35C | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 35D | 1 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 35F | 18 | 3 | 5 | 3 | 4 | 12 | 0 | 0 | 27 | 3 |
| 38 | 10 | 1 | 5 | 3 | 0 | 0 | 0 | 0 | 15 | 2 |
| NT | 0 | 0 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| Total | 685 | 100 | 199 | 100 | 33 | 100 | 11 | 100 | 928 | 100 |
Table 2.
Isolates from invasive pneumococcal disease cases in people ≥60 in Germany, by serotype and age group.
| Serotype | Age 60–69 |
Age 70–79 |
Age ≥80 |
Total |
||||
|---|---|---|---|---|---|---|---|---|
| n | % | n | % | n | % | N | % | |
| 3 | 63 | 22 | 71 | 21 | 71 | 23 | 205 | 22 |
| 4 | 5 | 2 | 0 | 0 | 1 | 0 | 6 | 1 |
| 6A | 1 | 0 | 2 | 1 | 4 | 1 | 7 | 1 |
| 6B | 0 | 0 | 2 | 1 | 1 | 0 | 3 | 0 |
| 6C | 3 | 1 | 7 | 2 | 5 | 2 | 15 | 2 |
| 6D | 0 | 0 | 1 | 0 | 1 | 0 | 2 | 0 |
| 7B | 1 | 0 | 3 | 1 | 3 | 1 | 7 | 1 |
| 7C | 1 | 0 | 0 | 0 | 2 | 1 | 3 | 0 |
| 7F | 0 | 0 | 2 | 1 | 1 | 0 | 3 | 0 |
| 8 | 26 | 9 | 43 | 13 | 19 | 6 | 88 | 9 |
| 9N | 24 | 9 | 19 | 6 | 21 | 7 | 64 | 7 |
| 9V | 3 | 1 | 2 | 1 | 0 | 0 | 5 | 1 |
| 10A | 10 | 4 | 16 | 5 | 14 | 5 | 40 | 4 |
| 10B | 3 | 1 | 0 | 0 | 0 | 0 | 3 | 0 |
| 11A | 7 | 2 | 6 | 2 | 6 | 2 | 19 | 2 |
| 12A | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 0 |
| 12F | 13 | 5 | 17 | 5 | 12 | 4 | 42 | 5 |
| 13 | 1 | 0 | 1 | 0 | 0 | 0 | 2 | 0 |
| 14 | 2 | 1 | 2 | 1 | 2 | 1 | 6 | 1 |
| 15A | 8 | 3 | 9 | 3 | 12 | 4 | 29 | 3 |
| 15B | 2 | 1 | 7 | 2 | 10 | 3 | 19 | 2 |
| 15C | 1 | 0 | 1 | 0 | 5 | 2 | 7 | 1 |
| 16F | 3 | 1 | 5 | 1 | 11 | 4 | 19 | 2 |
| 17F | 1 | 0 | 3 | 1 | 0 | 0 | 4 | 0 |
| 18A | 1 | 0 | 0 | 0 | 2 | 1 | 3 | 0 |
| 18C | 1 | 0 | 2 | 1 | 2 | 1 | 5 | 1 |
| 19A | 14 | 5 | 12 | 4 | 14 | 5 | 40 | 4 |
| 19F | 2 | 1 | 4 | 1 | 7 | 2 | 13 | 1 |
| 20 | 4 | 1 | 5 | 1 | 5 | 2 | 14 | 2 |
| 21 | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| 22F | 24 | 9 | 21 | 6 | 25 | 8 | 70 | 8 |
| 23A | 10 | 4 | 6 | 2 | 10 | 3 | 26 | 3 |
| 23B | 11 | 4 | 14 | 4 | 3 | 1 | 28 | 3 |
| 23F | 1 | 0 | 2 | 1 | 1 | 0 | 4 | 0 |
| 24F | 3 | 1 | 11 | 3 | 7 | 2 | 21 | 2 |
| 28A | 0 | 0 | 3 | 1 | 1 | 0 | 4 | 0 |
| 28F | 1 | 0 | 1 | 0 | 1 | 0 | 3 | 0 |
| 31 | 4 | 1 | 6 | 2 | 4 | 1 | 14 | 2 |
| 33F | 7 | 2 | 7 | 2 | 2 | 1 | 16 | 2 |
| 34 | 2 | 1 | 1 | 0 | 2 | 1 | 5 | 1 |
| 35B | 5 | 2 | 6 | 2 | 5 | 2 | 16 | 2 |
| 35C | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
| 35D | 0 | 0 | 1 | 0 | 0 | 0 | 1 | 0 |
| 35F | 7 | 2 | 9 | 3 | 11 | 4 | 27 | 3 |
| 38 | 3 | 1 | 6 | 2 | 6 | 2 | 15 | 2 |
| NT | 1 | 0 | 0 | 0 | 0 | 0 | 1 | 0 |
| Total | 281 | 100 | 338 | 100 | 309 | 100 | 928 | 100 |
Table 3.
Isolates from invasive pneumococcal disease cases in people ≥60 in Germany, by serotype and time since vaccination.
| Serotype | Vaccinated less than 2 years prior to IPD |
Vaccinated between 2 and 4 years prior to IPD |
Vaccinated 5 or more years prior to IPD |
Total |
||||
|---|---|---|---|---|---|---|---|---|
| n | % | n | % | n | % | N | % | |
| 3 | 28 | 36 | 16 | 25 | 31 | 32 | 75 | 31 |
| 4 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 6A | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 6B | 0 | 0 | 1 | 2 | 0 | 0 | 1 | 0 |
| 6C | 1 | 1 | 2 | 3 | 2 | 2 | 5 | 2 |
| 7B | 0 | 0 | 1 | 2 | 0 | 0 | 1 | 0 |
| 7C | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 7F | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 8 | 8 | 10 | 1 | 2 | 6 | 6 | 15 | 6 |
| 9N | 3 | 4 | 6 | 9 | 4 | 4 | 13 | 5 |
| 9V | 0 | 0 | 1 | 2 | 1 | 1 | 2 | 1 |
| 10A | 4 | 5 | 2 | 3 | 3 | 3 | 9 | 4 |
| 11A | 2 | 3 | 0 | 0 | 2 | 2 | 4 | 2 |
| 12A | 0 | 0 | 1 | 2 | 0 | 0 | 1 | 0 |
| 12F | 1 | 1 | 2 | 3 | 1 | 1 | 4 | 2 |
| 13 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 14 | 0 | 0 | 1 | 2 | 0 | 0 | 1 | 0 |
| 15A | 1 | 1 | 2 | 3 | 4 | 4 | 7 | 3 |
| 15B | 0 | 0 | 2 | 3 | 1 | 1 | 3 | 1 |
| 16F | 1 | 1 | 1 | 2 | 2 | 2 | 4 | 2 |
| 18C | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| 19A | 1 | 1 | 1 | 2 | 4 | 4 | 6 | 3 |
| 19F | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 20 | 1 | 1 | 0 | 0 | 0 | 0 | 1 | 0 |
| 22F | 4 | 5 | 2 | 3 | 6 | 6 | 12 | 5 |
| 23A | 1 | 1 | 5 | 8 | 11 | 11 | 17 | 7 |
| 23B | 3 | 4 | 4 | 6 | 4 | 4 | 11 | 5 |
| 23F | 0 | 0 | 1 | 2 | 0 | 0 | 1 | 0 |
| 24F | 2 | 3 | 0 | 0 | 4 | 4 | 6 | 3 |
| 28A | 0 | 0 | 1 | 2 | 1 | 1 | 2 | 1 |
| 28F | 1 | 1 | 0 | 0 | 1 | 1 | 2 | 1 |
| 31 | 0 | 0 | 2 | 3 | 2 | 2 | 4 | 2 |
| 33F | 2 | 3 | 2 | 3 | 0 | 0 | 4 | 2 |
| 34 | 3 | 4 | 1 | 2 | 0 | 0 | 4 | 2 |
| 35B | 1 | 1 | 0 | 0 | 2 | 2 | 3 | 1 |
| 35C | 0 | 0 | 0 | 0 | 1 | 1 | 1 | 0 |
| 35F | 4 | 5 | 2 | 3 | 2 | 2 | 8 | 3 |
| 38 | 0 | 0 | 3 | 5 | 2 | 2 | 5 | 2 |
| NT | 0 | 0 | 1 | 2 | 0 | 0 | 1 | 0 |
| Total | 78 | 100 | 64 | 100 | 97 | 100 | 239 | 100 |
We calculated age- and sex- adjusted VE for VT IPD for all PPV23 serotypes, for serotype 3 individually, for PPV23 serotypes excluding serotype 3, and for PPV23 serotypes excluding 3 and 8 (Table 4) and repeated the process for PCV13 (Supplemental Table 2). PPV23 had an overall VE of 37% (95% confidence interval (CI) 12% - 55%, a VE of −110% (CI: −198% - −47%) against serotype 3 IPD, a VE of 63% (CI: 49% - 74%) against all PPV23 serotypes except 3, and a VE of 58% (CI: 40 – 70) for PPV23 serotypes except 3 and 8. Calculations of PPV23 VE against individual serotypes (except for serotype 3) were limited by small sample sizes (Supplemental Table 3), but VE against serotype 12F IPD was 87% (CI: 52–99%) and VE was 46% (CI: 1–73%) against serotype 8 IPD.
Table 4.
Age- and gender- adjusted vaccine effectiveness of 23-valent pneumococcal polysaccharide vaccine (PPV23) against invasive pneumococcal disease in people ≥60 in Germany. 95% CI shows the lower and upper bounds of the 95% confidence interval of the vaccine effectiveness estimate.
| All | vaccinated <2 years prior to IPD |
vaccinated between 2 and 4 years prior to IPD |
vaccinated ≥5 years prior to IPD |
Age 60–69 |
Age 70–79 |
Age ≥80 |
||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| VE (%) | 95% CI (%) | VE (%) | 95% CI (%) | VE (%) | 95% CI (%) | VE (%) | 95% CI (%) | VE (%) | 95% CI (%) | VE (%) | 95% CI (%) | VE (%) | 95% CI (%) | |
| PPV23 serotypes | 37 | 12,55 | −29 | −159,31 | 56 | 17,76 | 40 | 4,63 | 43 | −820,100 | −5 | −538,83 | −17 | −80,77 |
| PPV23 serotypes except 3 | 63 | 49,74 | 45 | 4,69 | 69 | 41,85 | 66 | 44,79 | −252 | −520,77 | 8 | −373,37 | −3 | −335,48 |
| PPV23 serotypes except 3 and 8 | 58 | 40,70 | 51 | 12,74 | 52 | 9,76 | 62 | 36,78 | 62 | 21,83 | 59 | 28,78 | 51 | 16,72 |
| PPV23-nonPCV13 serotypes | 59 | 42,72 | 35 | −14,65 | 68 | 36,85 | 61 | 36,78 | 78 | 50,92 | 54 | 21,74 | 50 | 12,72 |
| Serotype 3 | −110 | −198,−47 | −178 | −393,−54 | −80 | −243,10 | −116 | −250,31 | 88 | −90,100 | 2 | −40,84 | 6 | −133,57 |
Stratifying by age group, VE for PPV23 in people age 60–69 was 71% (CI: 42–86%); VE for PPV23 serotypes except serotype 3 was 67% (CI: 33–85%); VE for PPV23 serotypes except 3 and 8 was 62% (21–83%). For people aged 70–79, VE for PPV23 serotypes except 3 was 63% (CI: 38–79) while VE for PPV23 serotypes except 3 and 8 was 59% (CI: 28–78%). For people ≥80, VE for PPV23 serotypes except 3 was 60% (CI: 32–77%) and VE for PPV23 serotypes except 3 and 8 was 51% (CI: 16–72%).
Stratifying by time since vaccination revealed that only 30% of vaccinated people ≥60 with IPD had been vaccinated within the last two years before their IPD episode. VE for PPV23 serotypes except 3 for people vaccinated less than two years prior to IPD was 49% (CI: 12–71); VE for PPV23 serotypes except 3 and 8 was 52%, but with a very wide CI (−256–97). VE for PPV23 serotypes except 3 for people vaccinated between two and four years prior to IPD was 66% (CI: 37–82%) and 48% (CI: 3–73%) for PPV23 serotypes except 3 and 8. VE for PPV23 serotypes except 3 for people vaccinated more than five years ago was 69% (CI: 50–81%) and 65% (CI: 42–79%) for serotypes except 3 and 8.
Stratifying by both age at time of IPD episode and time since vaccination showed that 23% of people 60–69 with IPD had been vaccinated ≥6 years ago, 45% of people 70–79 had been vaccinated ≥6 years ago, and 44% of people 80 and over had been vaccinated ≥6 years ago.
We included the 11 people vaccinated with both PPV23 and PCV13 as PPV23-vaccinated and as PCV13-vaccinated to assess the sensitivity of our VE estimates; there was no effect on VE estimates for either PPV23 or PCV13 when including these cases (Supplemental Table 4).
Receiving a pneumococcal vaccination, either with PPV23 or with PCV13, correlated strongly with receiving an influenza vaccination (Fig. 3). Of the 928 people for whom we could determine a vaccination status, 255 were vaccinated against influenza within five years of their IPD episode. Of those vaccinated against influenza, 91 were also vaccinated with PPV23, while 21 were also vaccinated with PCV13. 108 people who received PPV23 did not receive an influenza vaccine, and 12 people who received PCV13 did not receive an influenza vaccine.
Fig. 3.
Concurrence between pneumococcal vaccination and influenza vaccination in adults ≥60 with IPD in Germany. Subjects who received an influenza vaccination (n = 255) are in blue; subjects who received PPV23 (n = 199) are in pink; subjects who received PCV13 are in yellow (n = 33).
4. Discussion
4.1. Rate of pneumococcal vaccination
Recent estimates of adult vaccination in Germany indicate that pneumococcal vaccination rates are low, and our study falls in line with these findings. Rates of pneumococcal vaccination in the general population of adults aged ≥60 were between 4.2% and 28.4% in 2015–2019. Our study found that 26% of people with IPD aged ≥ 60 had received a pneumococcal vaccine, which indicates that our study population had similar vaccination behavior to the general population of older adults in Germany [29,30]. Even among vaccinated people, adherence to schedule recommendations were poor: nearly half of people ≥70 years old had not been vaccinated within six years of their IPD episode, and over a third of people had not been vaccinated within the past decade. The rate of pneumococcal vaccination in the general public seems to have improved slightly in 2020 (24.2% of people age 60–72 had been vaccinated; compared to 18.7% of people 60–72 in our study), concurrent with public vaccination campaigns related to the SARS-CoV-2 pandemic [31]. We are hopeful that recent efforts in increasing vaccination in older adults will result in sustained improvements in pneumococcal vaccine uptake in this vulnerable population.
Work is underway to develop effective methods of communicating with this age group and their primary care providers about vaccination [32]. A recent survey of German family practice physicians indicated that physicians’ own vaccination behaviors influenced their likelihood of recommending vaccines to their adult patients, and their likelihood of remembering to check patients’ vaccination status. Encouraging or incentivizing physicians to get vaccinated themselves may be an avenue to improve the low adult vaccination rates in Germany [33]. The strong correlation found between pneumococcal vaccination and influenza vaccination may provide an opportunity to increase pneumococcal vaccination rates, and the potential of bundling influenza and pneumococcal vaccines should be further investigated [32]. Before we can realistically expect further population-level reductions in VT IPD in older adults, we must engage with this age group and their care providers to increase the rate of vaccination.
4.2. Vaccine effectiveness
The overall VE of PPV23 established in this study was substantially higher than that found in the UK [26] and South Korea [34], but similar to VE described in Japan [35], and lower than the results in two meta-analyses [13,36]. The VE of PPV23 on serotypes excluded in PCV13 (serotypes 2, 8, 9 N, 10A, 11A, 12F, 15B, 17F, 20, 22F, and 33F) was 59%, indicating a decent level of protection against IPD from serotypes not included in the current PCVs. The VE of PPV23 excluding serotype 3 was high (63%), and in concordance with values published earlier. The VE of PPV23 against all serotypes except 3 and 8 was 58%, indicating that the moderate protection provided against serotype 8 does contribute to the overall effectiveness of PPV23 in this population.
Our study corroborates the total lack of protection of PPV23 against serotype 3 [26]. PPV23 had a strongly negative VE for serotype 3, though the confidence interval was wide without crossing zero possibly indicating an association between PPV23-vaccinated individuals and serotype 3 IPD. This finding is of particular importance, as serotype 3 VE is a main driver in the cost-effectiveness model for the German vaccine recommendation [13]. VE for serotype 8 was 46%. Considering that 32% of all IPD cases in older adults in Germany are caused by serotypes 3 and 8, our results highlight the need for a vaccine that targets the dominant serotypes causing disease in this population. In a clinical trial for a 20-valent pneumococcal conjugate vaccine, serotype 8 missed the non-inferiority lower bound criterion of >0.5 by a small margin indicating a lower immune response for this serotype [37].
For PPV23 serotypes except serotype 3, our VE estimate was lower (49%, CI: 12–71) in people vaccinated within two years of their IPD episode, than in people vaccinated within two to four years of their IPD episode (66%, CI: 37–82%), which is unexpected, given waning protection over time. This may result may be driven by differences in age (the group vaccinated within two years of their IPD episode had a median age of 78.5, versus a median of 72.5 years of age for the group vaccinated between two and four years of IPD), or by serotype 8, which was completely absent in the group vaccinated between two and four years of IPD, and comprised around 1/4 of vaccine-type cases in the group vaccinated within two years of IPD.
In our study, PCV13 was only used in 33 patients with IPD (3.5%), and low VT sample sizes resulted in unreliable estimates with very wide confidence intervals, so we are unable to make definitive conclusions about PCV13 VE in the older adult population in Germany. Determining the VE of PCV13 against serotype 3 IPD would have been of particular interest, as the existing body of evidence remains varied, with wide-ranging estimates appearing in different populations and study methods [3,27].
4.3. Policy considerations
Combined with the reported lack of herd protection of the childhood PCV13 vaccination on serotype 3 IPD among adults in Germany, and the fact that currently 20% of IPD in adults 60 years and older in Germany is caused by serotype 3 [19], the negative VE found for PPV23 against serotype 3 IPD indicates that the current pneumococcal vaccination strategy for older adults does not offer sufficient protection against one of the major causes of both IPD and non-bacteremic pneumonia in this age group. The primary problem with the current pneumococcal vaccination recommendation is low uptake, but if increasing uptake does not address the burden of disease caused by serotype 3, the current recommendation will not adequately address the needs of this growing population group in Germany.
Around half of the isolates received by the GNRCS as part of the IPD surveillance program were sent by clinical laboratories which refused to participate in this additional study to ascertain vaccination status, many of them citing the recently enacted general data protection regulation (GDPR) issued by the European Union. Despite the law having specific provisions for the sharing of sensitive information for public health surveillance purposes [38], our experience provides evidence that the interpretation of these new requirements for epidemiological surveillance studies is inconsistent. Additional official guidance is needed to inform clinical laboratories how to safely provide data in order to continue infectious disease surveillance activities so that valuable data are not needlessly lost.
4.4. Limitations
The GNRCS surveillance system only collects cases of IPD, so we do not know definitively if vaccination behaviors in the general population of adults ≥60 are necessarily similar to the low uptake rates described in adults ≥60 with IPD. Microbiological laboratories do not know the serotype prior to sending the isolate to the GNRCS as they do not perform serotyping, and the serotype distributions and age distributions of IPD cases from included and excluded laboratories were similar, which reduces the likelihood that the high number of laboratories who refused to participate in the study has biased the results.
Conclusions
PPV23 offers moderate protection overall, but solid protection against PPV23 serotypes except serotype 3. PCV13 is rarely used in adults with IPD in Germany. In order to accurately assess pneumococcal vaccine impact in older adults in Germany, pneumococcal vaccine uptake must be increased.
Declaration of Interests
Dr. van der Linden reports grants from Pfizer Deutschland GmbH, during the conduct of the study; personal fees from Pfizer, personal fees from Pfizer, grants from Pfizer, personal fees from Merck, personal fees from Merck, grants from Merck, outside the submitted work. Dr. Perniciaro reports grants from Pfizer Deutschland GmbH, during the conduct of the study; personal fees from Pfizer Deutschland GmbH, outside the submitted work.
Acknowledgments
Acknowledgements
We thank all of the participating laboratories, hospitals and primary care providers, the Center for Translational & Clinical Research (CTC-A) at the University Hospital RWTH Aachen, and Jutta Krasenbrink.
Contributor
SP performed data entry, auditing, and statistical analyses. ML contacted laboratories, managed personnel at the CTC-A, and provided resources for the conduct of the study.
Data Sharing
A deidentified participant dataset will be made available with publication as part of the supplementary material of this article.
Footnotes
Main Point of Article: Pneumococcal vaccination in older adults with invasive pneumococcal disease in Germany is rare. Increased uptake would reduce the burden of disease, however, vaccine effectiveness against the most common cause of pneumococcal disease in this age group is poor.
Supplementary material associated with this article can be found, in the online version, at https://doi.org/10.1016/j.lanepe.2021.100126.
Appendix. Supplementary materials
References
- 1.Menéndez R., Torres A., Zalacaín R. Risk factors of treatment failure in community acquired pneumonia: implications for disease outcome. Thorax. 2004;59(11):960–965. doi: 10.1136/thx.2003.017756. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Rosón B., Carratalà J., Fernández-Sabé N., Tubau F., Manresa F., Gudiol F. Causes and factors associated with early failure in hospitalized patients with community-acquired pneumonia. Arch Intern Med. 2004;164(5):502–508. doi: 10.1001/archinte.164.5.502. [DOI] [PubMed] [Google Scholar]
- 3.Bonten M.J.M., Huijts S.M., Bolkenbaas M. Polysaccharide conjugate vaccine against pneumococcal pneumonia in adults. N Engl J Med. 2015;372(12):1114–1125. doi: 10.1056/NEJMoa1408544. [DOI] [PubMed] [Google Scholar]
- 4.Welte T., Torres A., Nathwani D. Clinical and economic burden of community-acquired pneumonia among adults in Europe. Thorax. 2012;67(1):71–79. doi: 10.1136/thx.2009.129502. [DOI] [PubMed] [Google Scholar]
- 5.Ewig S., Birkner N., Strauss R. New perspectives on community-acquired pneumonia in 388 406 patients. Results from a nationwide mandatory performance measurement programme in healthcare quality. Thorax. 2009;64(12):1062–1069. doi: 10.1136/thx.2008.109785. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Horácio A.N., Silva-Costa C., Lopes J.P. Serotype 3 remains the leading cause of invasive pneumococcal disease in adults in Portugal (2012–2014) despite continued reductions in other 13-valent conjugate vaccine serotypes. Front Microbiol. 2016;7 doi: 10.3389/fmicb.2016.01616. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.LeBlanc J., ElSherif M., Ye L. Streptococcus pneumoniae serotype 3 is masking PCV13-mediated herd immunity in Canadian adults hospitalized with community acquired pneumonia: a study from the Serious Outcomes Surveillance (SOS) Network of the Canadian immunization research Network (CIRN) Vaccine. 2019;37(36):5466–5473. doi: 10.1016/j.vaccine.2019.05.003. [DOI] [PubMed] [Google Scholar]
- 8.Forstner C., Kolditz M., Kesselmeier M. Pneumococcal conjugate serotype distribution and predominating role of serotype 3 in German adults with community-acquired pneumonia. Vaccine. 2020;38(5):1129–1136. doi: 10.1016/j.vaccine.2019.11.026. [DOI] [PubMed] [Google Scholar]
- 9.Heo J.Y., Seo Y.B., Choi W.S. Incidence and case fatality rates of community-acquired pneumonia and pneumococcal diseases among Korean adults: catchment population-based analysis. PLoS ONE. 2018;13(3) doi: 10.1371/journal.pone.0194598. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Mufson M.A., Stanek R.J. Bacteremic pneumococcal pneumonia in one American city: a 20-year longitudinal study, 1978–1997. Am J Med. 1999;107(1):34–43. doi: 10.1016/S0002-9343(99)00098-4. [DOI] [PubMed] [Google Scholar]
- 11.Jansen A.G.S.C., Rodenburg G.D., de Greeff S.C. Invasive pneumococcal disease in the Netherlands: syndromes, outcome and potential vaccine benefits. Vaccine. 2009;27(17):2394–2401. doi: 10.1016/j.vaccine.2009.01.127. [DOI] [PubMed] [Google Scholar]
- 12.Naucler P., Darenberg J., Morfeldt E., Örtqvist Å., Normark B.H. Contribution of host, bacterial factors and antibiotic treatment to mortality in adult patients with bacteraemic pneumococcal pneumonia. Thorax. 2013;68(6):571–579. doi: 10.1136/thoraxjnl-2012-203106. [DOI] [PubMed] [Google Scholar]
- 13.Falkenhorst G., Remschmidt C., Harder T., Hummers-Pradier E., Wichmann O., Bogdan C. Effectiveness of the 23-valent pneumococcal polysaccharide vaccine (PPV23) against pneumococcal disease in the elderly: systematic review and meta-analysis. PLoS ONE. 2017;12(1) doi: 10.1371/journal.pone.0169368. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Siber G., Klugman K., Mäkelä P.H. ASM Press; 2008. Pneumococcal vaccines: the impact of conjugate vaccine. [Google Scholar]
- 15.Frank O., De Oliveira Bernardo C., González-Chica D.A., Macartney K., Menzies R., Stocks N. Pneumococcal vaccination uptake among patients aged 65 years or over in Australian general practice. Hum Vaccines Immunother. 2020;16(4):965–971. doi: 10.1080/21645515.2019.1682844. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Gatwood J., Shuvo S., Hohmeier K.C. Pneumococcal vaccination in older adults: an initial analysis of social determinants of health and vaccine uptake. Vaccine. 2020;38(35):5607–5617. doi: 10.1016/j.vaccine.2020.06.077. [DOI] [PubMed] [Google Scholar]
- 17.Robert Koch Institute R Epidemiologisches Bulletin 34/2020. August. 2020:68. Published online. [Google Scholar]
- 18.Schutzimpfungen - 02/2018 - 2018 - KVS-Mitteilungen - Mitglieder - kassenärztliche vereinigung sachsen. https://www.kvs-sachsen.de/mitglieder/kvs-mitteilungen/2018/02-2018/schutzimpfungen/. Accessed September 24, 2020.
- 19.van der Linden M., Imöhl M., Perniciaro S. Limited indirect effects of an infant pneumococcal vaccination program in an aging population. PLoS ONE. 2019;14(8) doi: 10.1371/journal.pone.0220453. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Hurley D., Griffin C., Young M., et al. Safety, tolerability, and immunogenicity of a 20-valent Pneumococcal Conjugate Vaccine (PCV20) in adults 60 to 64 years of age. Clin Infect Dis. doi:10.1093/cid/ciaa1045 [DOI] [PMC free article] [PubMed]
- 21.Stacey H.L., Rosen J., Peterson J.T. Safety and immunogenicity of 15-valent pneumococcal conjugate vaccine (PCV-15) compared to PCV-13 in healthy older adults. Hum Vaccines Immunother. 2019;15(3):530–539. doi: 10.1080/21645515.2018.1532249. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Reinert R.R., Haupts S., Linden M.V.D. Invasive pneumococcal disease in adults in North-Rhine Westphalia, Germany, 2001–2003. Clin Microbiol Infect. 2005;11(12):985–991. doi: 10.1111/j.1469-0691.2005.01282.x. [DOI] [PubMed] [Google Scholar]
- 23.Broome C.V., Facklam R.R., Fraser D.W. Pneumococcal disease after pneumococcal vaccination. http://dx.doi.org/10.1056/NEJM198009043031003. doi:10.1056/NEJM198009043031003
- 24.Firth D. Bias reduction of maximum likelihood estimates. Biometrika. 1993;80(1):27–38. doi: 10.2307/2336755. [DOI] [Google Scholar]
- 25.Heinze G., Schemper M. A solution to the problem of separation in logistic regression. Stat Med. 2002;21(16):2409–2419. doi: 10.1002/sim.1047. [DOI] [PubMed] [Google Scholar]
- 26.Djennad A., Ramsay M.E., Pebody R. Effectiveness of 23-valent polysaccharide pneumococcal vaccine and changes in invasive pneumococcal disease incidence from 2000 to 2017 in those aged 65 and over in England and wales. EClinicalMedicine. 2018;6:42–50. doi: 10.1016/j.eclinm.2018.12.007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Matanock A. Use of 13-valent pneumococcal conjugate vaccine and 23-valent pneumococcal polysaccharide vaccine among adults aged ≥65 years: updated recommendations of the advisory committee on immunization practices. MMWR Morb Mortal Wkly Rep. 2019;68 doi: 10.15585/mmwr.mm6846a5. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.AER_for_2016-invasive-pneumococcal-disease_0.pdf. https://www.ecdc.europa.eu/sites/default/files/documents/AER_for_2016-invasive-pneumococcal-disease_0.pdf. Accessed January 28, 2021.
- 29.Theidel U., Kuhlmann A., Braem A. Pneumococcal Vaccination Rates in Adults in Germany. Dtsch Ärztebl Int. 2013;110(44):743–750. doi: 10.3238/arztebl.2013.0743. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Robert Koch Institute . 2019. Epidemiologisches bulletin; p. 44.https://www.rki.de/DE/Content/Infekt/EpidBull/Archiv/2019/Ausgaben/44_19.pdf?__blob=publicationFile [Google Scholar]
- 31.Koch-Institut R. 2020. Epidemiologisches bulletin 47/2020; p. 34. November. Published online. [Google Scholar]
- 32.Betsch C., Rossmann C., Pletz M.W. Increasing influenza and pneumococcal vaccine uptake in the elderly: study protocol for the multi-methods prospective intervention study Vaccination60. BMC Public Health. 2018;18(1):885. doi: 10.1186/s12889-018-5787-9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Neufeind J., Betsch C., Bach Habersaat K., Eckardt M., Schmid P., Wichmann O. Barriers and drivers to adult vaccination among family physicians – Insights for tailoring the immunization program in Germany. Vaccine. 2020;38:4252–4262. doi: 10.1016/j.vaccine.2020.04.052. [DOI] [PubMed] [Google Scholar]
- 34.Kim J.H., Chun B.C., Song J.Y. Direct effectiveness of pneumococcal polysaccharide vaccine against invasive pneumococcal disease and non-bacteremic pneumococcal pneumonia in elderly population in the era of pneumococcal conjugate vaccine: a case-control study. Vaccine. 2019;37(21):2797–2804. doi: 10.1016/j.vaccine.2019.04.017. [DOI] [PubMed] [Google Scholar]
- 35.Suzuki M., Dhoubhadel B.G., Ishifuji T. Serotype-specific effectiveness of 23-valent pneumococcal polysaccharide vaccine against pneumococcal pneumonia in adults aged 65 years or older: a multicentre, prospective, test-negative design study. Lancet Infect Dis. 2017;17(3):313–321. doi: 10.1016/S1473-3099(17)30049-X. [DOI] [PubMed] [Google Scholar]
- 36.Kraicer-Melamed H., O'Donnell S., Quach C. The effectiveness of pneumococcal polysaccharide vaccine 23 (PPV23) in the general population of 50 years of age and older: a systematic review and meta-analysis. Vaccine. 2016;34(13):1540–1550. doi: 10.1016/j.vaccine.2016.02.024. [DOI] [PubMed] [Google Scholar]
- 37.Data from Pfizer's adult and pediatric clinical trial programs for 20-Valent pneumococcal conjugate vaccine presented at IDWeek 2020 | Pfizer. https://www.pfizer.com/news/press-release/press-release-detail/data-pfizers-adult-and-pediatric-clinical-trial-programs-0. Accessed January 28, 2021
- 38.Vollmer N. May 22, 2020. Article 9 eu general data protection regulation (EU-GDPR) https://www.privacy-regulation.eu/en/article-9-processing-of-special-categories-of-personal-data-GDPR.htm. Published. Accessed September 22, 2020. [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.