Challenges in Phase 4 post-licensure safety studies using real world data in the United States: Hepatitis B vaccine example

Highlights

• •Post-licensure vaccine safety studies can identify adverse events that were not detected in pre-licensure trials.
• •Study design in real-world settings involves many pragmatic and scientific decisions.
• •We discuss these considerations in design of a post-licensure hepatitis B vaccine safety study.
• •This example can inform design of other post-licensure vaccine safety studies.

Abstract

Post-licensure vaccine safety studies are essential to identify adverse events that may not have been detected in pre-licensure clinical trials and to address questions that arose during the pre-licensure phase. These studies are increasingly conducted using real-world data collected as part of routine health care delivery. However, design of post-licensure vaccine safety studies involves many pragmatic and scientific decisions, which must be made while balancing diverse stakeholder opinions. Challenges include selecting exposure and comparison groups, deciding on the most appropriate outcome, determining sample size and length of follow-up time, and other analytic considerations. As an example of this process and to inform other post-licensure vaccine safety studies in real-world settings, we discuss our experience with design of an FDA-required Phase 4 post-licensure safety study of a hepatitis B vaccine in a large integrated health care organization in the United States.

1. Introduction

Rigorous safety studies are of paramount importance for licensure of vaccines and public confidence in their safety. In the United States, vaccine safety is evaluated through a robust series of pre-licensure studies, post-marketing surveillance led by the Food and Drug Administration (FDA) and the Centers for Disease Control and Prevention (CDC), and often Phase 4 post-licensure studies [1], [2], [3]. This system ensures that vaccine safety is sufficiently demonstrated prior to licensure and continues to be monitored when the vaccine is on the market.

Pre-licensure vaccine safety studies, required by the FDA as part of the licensure process, include preclinical laboratory studies and a series of Phase 1 to 3 clinical trials [1]. These studies provide critical data on vaccine safety and efficacy that determine licensure approval. Importantly, pre-licensure vaccine safety studies are designed to achieve high internal validity but may have less generalizability; these studies are usually conducted in select populations of relatively healthy individuals and can have limited sample sizes and follow-up time that may preclude detection of rare adverse events [4].

Post-licensure vaccine safety studies are conducted once a vaccine is on the market and used in the general population. The purpose of post-licensure vaccine safety studies is to identify adverse events that may not have been detected in pre-licensure studies, particularly those that are rare, occur long after vaccination, or occur only in certain sub-populations [5]. In addition, these studies often address questions that arise during the pre-licensure phase. FDA- and CDC-administered safety surveillance systems, discussed in detail elsewhere [3], [6], [7], [8], conduct essential pharmacovigilance, but studies of adverse events in recently licensed or new vaccines are precluded until the vaccines are broadly used. The FDA therefore frequently requests or mandates that manufacturers conduct Phase 4 post-licensure vaccine safety studies, as is the focus in this report. These studies can utilize various randomized and non-randomized designs, but increasingly use data generated as part of routine health care delivery (e.g., data from electronic health records [EHR]) to assess vaccine safety in a real-world context [9].

Conduct of real-world Phase 4 post-licensure vaccine safety studies, however, can be complex. These studies usually include diverse stakeholders (e.g. industry sponsors, regulators and policy makers, health care administrators and insurers, institutional review boards, clinicians and staff, and patients), who can have numerous and often conflicting demands. In considering the manifold options for study design and analysis, divergent stakeholder opinions must be creatively and judiciously accommodated. Here, we discuss the process of designing a real-world post-licensure vaccine safety study, presenting as an example our experience with an FDA-required post-licensure safety study of a hepatitis B vaccine in a large integrated health care organization.

2. Example: Hepatitis-B vaccine post-licensure safety study

2.1. HEPLISAV-B [hepatitis B vaccine (recombinant), adjuvanted] vaccine

HEPLISAV-B® [Hepatitis B Vaccine (Recombinant), Adjuvanted] (“HepB-CpG” hereafter) is a hepatitis B vaccine developed by Dynavax Technologies [10]. The vaccine is composed of recombinant yeast cell-derived hepatitis B surface antigen (HBsAg) and a novel adjuvant that is an agonist for toll-like receptor 9 (TLR9). HepB-CpG induced higher and earlier seroprotection than Engerix-B® (“comparator vaccine” hereafter) in clinical trials and requires only two doses over one month compared to the comparator vaccine’s schedule of three doses over six months. In pre-licensure clinical trials, the rates of serious adverse events were similar between HepB-CpG and comparator vaccine, except for an unanticipated “imbalance” in the Medical Dictionary for Regulatory Activities (MedDRA) preferred term acute myocardial infarction (AMI) in one of the trials (HepB-CpG 0.25% [n = 14]; comparator vaccine: 0.04% [n = 1]) [11]. The AMI events occurred in individuals with a high prevalence of cardiovascular disease risk factors, without apparent temporal association to vaccine administration, and at lower than expected incidence rates in the comparator vaccine group.

HepB-CpG was approved in 2017 for prevention of hepatitis B in adults, after FDA’s Vaccines and Related Biological Products Advisory Committee (VRBPAC) voted 12 to 1 that the HepB-CpG safety data supported licensure. The expected Prescription Drug User Fee Act (PDUFA) date, the deadline for the FDA’s review of the Biologics License Application (BLA), was August 2017, but the review clock was extended to finalize and agree upon the key details of the post-licensure safety study [12], required to provide further data on the risk of AMI among HepB-CpG recipients. The FDA approved HepB-CpG in November 2017. In February 2018, the Advisory Committee on Immunization Practices (ACIP) recommended HepB-CpG for use as a hepatitis B vaccine in adults, and these recommendations were published in April 2018 in the Morbidity and Mortality Weekly Report (MMWR) [13]. These dates were important in preparing for the post-licensure safety study at Kaiser Permanente Southern California (KPSC), an ideal setting for this type of study.

2.2. Study setting

KPSC is an integrated health care system currently serving 4.6 million members with diverse racial/ethnic and socioeconomic backgrounds largely representative of the underlying population [14]. KPSC has 15 hospitals and over 230 medical offices, organized in 15 medical service areas, each surrounding a central medical center. Health plan members enroll through their employer or a family member’s employer, through individual and family plans, or through state or federal programs (e.g. Covered California, Medi-Cal, and other programs).

The KPSC HealthConnect® EHR system tracks members’ health care encounters, including diagnosis and procedure codes, vaccinations, medications, and laboratory results. The EHR are generally considered comprehensive, as members have strong financial incentive to seek care at KPSC facilities. All ACIP-recommended vaccines are provided free to members and offered proactively at any visit regardless of co-pay status. Although members may seek emergency medical care from outside health care providers, for outside facilities to be reimbursed, claims must be submitted including documentation substantiating a clinical diagnosis. The claims allow outside utilization to be linked to the member through the unique medical record number.

The KPSC EHR system has features in place to identify which members can benefit from vaccinations. Customized best practice alerts follow ACIP guidance for vaccine indications, dosing schedules, and interchangeability of products. For example, providers are alerted through the EHR if patients with diabetes, or those for whom sexually transmitted infection tests are ordered, have not received hepatitis B vaccine. These alerts have increased rates of adult vaccination at KPSC [15].

Moreover, the EHR system facilitates research at KPSC. Historical data on trends in vaccination and disease rates can provide accurate parameters for sample size estimation and can potentially be used for historical comparisons. Demographic and clinical characteristics of patients at the time of vaccination can be described in detail, and data points after vaccination can be obtained by passive follow-up through the EHR.

3. Considerations for design of post-licensure vaccine safety studies

3.1. Stakeholder engagement

While post-licensure vaccine safety studies at KPSC are conducted for vaccines that have been licensed by FDA, recommended by ACIP, and are considered safe and effective, they are often undertaken to address questions that arise in prelicensure trials. Accordingly, the complexity of arriving at study design decisions is magnified by the large number of stakeholders, many of whom are more familiar with conduct of randomized clinical trials than real-world observational studies. For example, industry sponsors and regulators may recommend methods that are more suitable for clinical trials than for a vaccine that has been implemented in routine clinical care. Clinicians may be eager to expand use of the new vaccine or, on the other hand, may be hesitant due to patient safety concerns. Independent review of data by a Data Monitoring Committee (DMC) acts as a safeguard to reassure the scientific community and the public about study conduct and results. All these stakeholders demand final results as soon as possible to determine continued use of the vaccine. In the following sections, we highlight these tensions and discuss challenges in designing the post-licensure HepB-CpG vaccine safety study.

3.2. Vaccine implementation

The first study design consideration was based on how to deliver the vaccine to eligible KPSC members. Randomization of individuals to HepB-CpG or to comparator vaccine was not an option for a real-world study; an individually randomized study would assess vaccine safety among individuals willing to be randomized, rather than among those receiving vaccine through routine clinical care. Administering HepB-CpG as part of routine clinical care required approval by the KPSC Regional Immunization Practices Committee (RIPC), which sets guidelines for vaccine use at KPSC medical centers, largely based on recommendations from ACIP. This meant that the logistics for provision of HepB-CpG to KPSC members, including developing EHR order sets, could not be finalized until after the ACIP recommendations for use of HepB-CpG were published in the MMWR, such that use of HepB-CpG began at KPSC in August 2018.

Although KPSC research department functions independently of pharmacy operations, as a real-world study, we utilized KPSC distribution channels in which new products can be rolled out to specific medical centers, allowing provision of HepB-CpG to pre-specified medical centers and departments while continuing to supply comparator vaccine to other medical centers and departments. Medical centers usually prefer stocking one vaccine product at a time to avoid administration and documentation errors. This was consistent with the need of the study to limit selection and misclassification biases that could result from having both vaccine products available in the same facility. Furthermore, EHR order sets could be modified for specific groups of medical centers sharing the same EHR instance. This allowed HepB-CpG to become the only adult hepatitis B vaccine used in 7 of 15 medical centers in Family Medicine and Internal Medicine departments, where > 90% of hepatitis B vaccine doses are given. In these departments in the 7 medical centers, the default EHR order sets associated with adult hepatitis B vaccination were changed from comparator vaccine to HepB-CpG. The order sets facilitated the administration of HepB-CpG according to the appropriate dosing schedule. No changes were made in the other 8 medical centers, which continued to administer comparator vaccine.

3.3. Comparison group

The next consideration in study design was selection of the comparison group. This required thinking through potential sources of bias, such as occurrence of selection bias if the comparison group was not representative of hepatitis B vaccine recipients, and occurrence of confounding if differences in AMI risk factors between the comparison group and the HepB-CpG group were not measured or controlled [16], [17]. A self-controlled comparison, which can minimize these sources of bias [18], [19], was not feasible since a long follow-up period would be needed, and pre-specifying a comparison window would be difficult because of unpredictable second dose receipt and timing in the course of routine care in real-world settings.

Thus, options for the comparison group were individuals who did not receive hepatitis B vaccine (unvaccinated cohort) or individuals who received another hepatitis B vaccine (comparator vaccine cohort), either of which could be concurrent or historical. The unvaccinated comparison group had high potential for confounding, since vaccinated and unvaccinated individuals would likely differ in health care utilization and comorbidities, particularly diabetes. The comparator vaccine cohort was also subject to selection bias resulting from a physician’s or patient’s decisions based on a patient’s risk profile, but this was reduced since only one adult hepatitis B vaccine product (HepB-CpG or comparator vaccine) would be available per facility. A historical cohort of comparator vaccine recipients in the target medical centers could further reduce cluster-level differences in population characteristics and was preferred by some stakeholders. However, because of secular trends in cardiovascular disease or changes in health care practice, a historical cohort could be systematically different from the HepB-CpG cohort.

After weighing these options, we decided to use a concurrent comparison of comparator vaccine recipients. To address stakeholder preferences, a secondary analysis was planned to compare HepB-CpG recipients with historical recipients of comparator vaccine at the same medical centers. We also decided to exclude individuals who had received chronic dialysis prior to hepatitis B vaccination, as vaccination recommendations differ for these individuals [20]. We also planned to compare characteristics of vaccine recipients between HepB-CpG and comparator vaccine groups, including sociodemographic characteristics, comorbidities, cardiovascular disease risk factors, cardiovascular disease medications, and concomitant vaccinations.

3.4. Outcome

Next, we needed to define the study outcome. The options were to use a 3-point Major Adverse Cardiovascular Events (MACE) composite outcome, including AMI, stroke, and cardiovascular death [21], or to focus on AMI. The MACE outcome was used in the HepB-CpG clinical trials, and stakeholders with cardiovascular clinical trial backgrounds tended to prefer this outcome. However, the clinical trial “imbalance” had occurred in type 1 AMI and not in other MedDRA preferred terms. A composite indicator such as MACE could be appropriate if the number of expected AMI events was small, but we expected sufficient AMI events given the large population of KPSC adults at risk of AMI who receive hepatitis B vaccine, such as individuals with diabetes. Moreover, including other events in the outcome could dilute detection of potential imbalance in AMI events.

For identification of AMI, the options were to rely on International Classification of Diseases (ICD) diagnosis codes alone, or to conduct detailed chart review to adjudicate potential events identified by codes. Using diagnosis codes from the EHR would be simple and quick, but potentially inaccurate. While prior studies had indicated a high validity of ICD-9 codes [22], [23], the validity of ICD-10 codes for AMI had not been established. Furthermore, ICD-10 included new codes for AMI types 2–5 based on the Universal Definition of AMI [24], which could further impact the validity of coding. Although our outcome of interest was first occurrence of type 1 AMI during follow-up, we included ICD-10 codes for all AMI types (I21.*, I22.*) in any diagnosis position in our search algorithm due to potential misclassification. ICD-10 codes for historical AMI (I25.2) were not included. In addition, we had to consider the care setting. True AMI events should result in admission to the hospital. However, some patients may die in the emergency room and others may be placed under observation but never formally admitted. Therefore, we included for chart review all patients with AMI diagnosis codes in the inpatient setting as well as those from the ED where the patient died that day or the following day and those who were transferred from ED to observation but not formally admitted.

We then needed to determine how type 1 AMI events would be adjudicated. Although research associates often conduct chart review for studies at KPSC, we thought that KPSC cardiologist reviewers were required to adjudicate occurrence and type of AMI. Chart review was conducted independently by two cardiologist reviewers masked to vaccine exposure, thereby strengthening the validity of results. A third cardiologist reviewer adjudicated cases with discrepant results. If the third reviewer disagreed with both prior reviewers, options for the final decision included having all three reviewers meet to come to a consensus, or deeming these cases as indeterminate. The first option seemed less objective. For example, a more senior reviewer could sway other reviewers toward their view. Therefore, we decided on the latter option and added a sensitivity analysis that did and did not include indeterminate cases in the primary outcome.

3.5. Follow-up time and sample size

Once the outcome was decided upon, the length of follow-up time needed to be determined, considering the two-dose vaccine schedule. This required weighing biological plausibility, statistical considerations, and stakeholder preferences. In the last HepB-CpG clinical trial, the resulting imbalance in AMI occurred after 120 days, with most AMI events during the trial occurring after 180 days after the first dose. The clinical trial used a 13-month follow-up period from the first dose, since this allowed a 1-year follow-up after the second dose which was consistently administered one month after the first dose. In the real-world post-licensure study, adherence to the vaccine schedule was unknown. Limiting follow-up to patients who received the second dose would greatly reduce the sample size, indefinitely lengthen the duration of the study, and bias the study population. For HepB-CpG there was no clear biological mechanism for AMI, and data were not compelling for a dose–response. Therefore, for comparability with the clinical trial, we decided to follow up individuals for 13 months after their first dose, regardless of timing of receipt of second dose.

Sample size considerations were based on historical rates of AMI among hepatitis B vaccine recipients. Assuming approximately 30,000 individuals each would receive HepB-CpG and comparator vaccine and be followed for up to 13 months, and assuming an event rate of 1.5/1000 person-years and 10% loss to follow-up, we estimated that we would have approximately 92% power to exclude HR ≥ 2.0. Since these considerations were based on assumptions, the DMC monitored the aggregate AMI rate and vaccine accrual on a quarterly basis and could recommend extension of the study if needed.

3.6. Analytical considerations

Stakeholders such as the FDA, ACIP, the industry sponsor, and KSPC clinicians were eager for preliminary data from the study. In order to not spend alpha (type 1 error), we did not want to conduct many interim analyses. However, we agreed to conduct two interim analyses on unadjudicated AMI using ICD-10 codes (rather than the primary outcome of adjudicated AMI). Conditional power analyses at each interim analysis would provide the likelihood of ruling out a pre-prescribed hazard ratio if the study were to continue under its original design.

For the primary analysis, we needed to determine if we would conduct a superiority or non-inferiority analysis. Because the FDA was interested in knowing if the risk of AMI was lower than a pre-specified margin, we decided to power the study to test the hypothesis: H₀: HR ≥ 2, H_a: HR < 2.

We also considered methods to control for residual confounding in analyses. The most easily interpretable option, a multivariable Cox proportional hazards model, would assess the crude association between exposure and outcome, examine the effect of each potential confounder on the estimate, and control for factors simultaneously. Propensity scores would control for a vector of confounders with a single variable, reserving degrees of freedom for testing the primary association [25]. Inverse probability of treatment weighting (IPTW) would use weights based on propensity scores to create a synthetic sample in which distribution of measured covariates would be well-balanced. Although IPTW can be unstable if there is strong confounding, this was not expected in this study. Another option was doubly robust methods, such as targeted maximum likelihood estimation (TMLE) [26]. These methods are robust to misspecification of the exposure or outcome model, but are theoretically complex and less widely used in pharmacoepidemiology. We decided to use IPTW for the primary analysis and to conduct a sensitivity analyses using a Cox proportional hazards model adjusting for propensity score and a traditional multivariable Cox proportional hazards model, with the DMC to advise if the primary and sensitivity analyses differ in their conclusions.

A number of exploratory analyses were also considered. Because this post-licensure vaccine safety study had limited exclusion criteria and reflected real-world use, it could be used to answer other relevant questions that could not be addressed as part of clinical trials. Potential exploratory analyses included evaluating AMI risk by prior receipt of hepatitis B vaccine (i.e., initial versus subsequent dose of hepatitis B vaccine) and by receipt of concomitant vaccines (e.g., influenza, pneumococcal, Tdap, or other adjuvanted vaccines). We also had the ability to assess adherence to the recommended schedule (i.e., receipt of 1 versus 2 doses, timing of completion of the 2-dose series).

While outside the scope of this AMI study, a separate post-marketing commitment study was also planned to evaluate the incidence of new onset immune-mediated diseases, herpes zoster, and anaphylaxis among HepB-CpG versus comparator vaccine recipients. This study involves a number of outcomes with varying follow-up periods, in contrast to the AMI study which has a single outcome with a single follow-up period.

4. Conclusions

This brief report has discussed some of the challenges of designing real-world post-licensure vaccine safety studies, describing as an example the considerations, complexities and time required in designing an observational safety study of HepB-CpG. We highlight the tensions between implementing the vaccine as part of routine clinical practice, while ensuring appropriate comparison groups, outcome definitions, follow-up time, and analytic methods. Cross-discipline collaboration is essential; divergent opinions of stakeholders from the real-world clinical perspective and the regulatory perspective must be negotiated. Real-world studies by definition are not “one size fits all” and will require different decisions based on context and needs. When there are multiple ways robust decisions could be made, one option must be selected using best available evidence, but alternative options should be considered in sensitivity, secondary, or exploratory analyses. Studies conducted in less integrated settings could encounter additional challenges ensuring complete capture of data on exposures, outcomes, and potential confounders, and could require more resources and time to obtain medical records needed for adjudication of outcomes from disparate institutions. In conclusion, the design of this post-licensure safety study of HepB-CpG highlights a number of considerations that are important to ensure that real-world evidence generated from observational studies are robust and valid for clinical, regulatory, and policy decision makers.

Declaration of Competing Interest

The authors declare the following financial interests/personal relationships which may be considered as potential competing interests: Katia Bruxvoort received research support from Dynavax Inc. for work related to this paper and unrelated research support from GlaxoSmithKline, Pfizer, and Seqirus Inc.

Lina S. Sy received research support from Dynavax Inc. for work related to this paper and unrelated research support from GlaxoSmithKline, Seqirus Inc., and Novavax.

Bradley K. Ackerson received research support from Dynavax Inc. for work related to this paper and has received research support from GlaxoSmithKline, Moderna, Pfizer, Seqirus Inc and Novartis for work unrelated to this study.

Jeffrey M. Slezak received research support from Dynavax Inc. for work related to this paper and unrelated research support from GlaxoSmithKline and Novovax Inc.

Lei Qian received research support from Dynavax Inc. for work related to this paper and unrelated research support from GlaxoSmithKline Inc.

William Towner received grant support paid to SCPMG from Dynavax for work related to this paper and unrelated research grant support paid to SCPMG from Gilead Sciences, Pfizer, Janssen, ViiV and Merck.

Kristi Reynolds received research support from Dynavax Inc. for work related to this paper.

Zendi Solano received research support from Dynavax Inc. for work related to this paper. Zendi Solano has received research support from GlaxoSmithKline and Gilead for work unrelated to this paper.

Cheryl Mercado received research support from Dynavax Inc. for work related to this paper.

Steven J. Jacobsen received research support from Dynavax Inc. for work related to this paper.