Developing Outcomes Assessments as Endpoints for Registrational Clinical Trials of Antibacterial Drugs: 2015 Update From the Biomarkers Consortium of the Foundation for the National Institutes of Health

At the Food and Drug Administration's request, we have been developing updated and new outcome measures for use as endpoints in the indications of community-acquired bacterial pneumonia, acute bacterial skin and skin structure infections, hospital-acquired bacterial pneumonia, and ventilator-associated bacterial pneumonia.

Abstract

One important component in determining the benefits and harms of medical interventions is the use of well-defined and reliable outcome assessments as endpoints in clinical trials. Improving endpoints can better define patient benefits, allowing more accurate assessment of drug efficacy and more informed benefit-vs-risk decisions; another potential plus is facilitating efficient trial design. Since our first report in 2012, 2 Foundation for the National Institutes of Health Biomarkers Consortium Project Teams have continued to develop outcome assessments for potential uses as endpoints in registrational clinical trials of community-acquired bacterial pneumonia and acute bacterial skin and skin structure infections. In addition, the teams have initiated similar work in the indications of hospital-acquired bacterial pneumonia and ventilator-associated bacterial pneumonia. This report provides an update on progress to date in these 4 diseases.

(See the Editorial Commentary by Toerner and Cox on pages 608–9.)

Bringing new, safe, and efficacious antibacterial therapies to patients and physicians requires the commitment of many stakeholders. Regulatory agencies play an important role in articulating scientifically valid and feasible clinical trial designs. In this context, the US Food and Drug Administration (FDA) has recently focused on (1) improving methodology for the design, conduct, and interpretation of clinical trials for anti-infective agents, especially noninferiority trials; (2) using expedited approval pathways for much-needed new anti-infectives; and (3) understanding and incorporating the patient “voice” in the drug development process, including use of patient-reported outcomes (PRO) instruments that measure how patients feel and/or function in response to a therapeutic intervention.

In 2010, the FDA asked the Biomarkers Consortium of the Foundation for the National Institutes of Health (FNIH) to help advance the scientific process of developing well-defined and reliable outcome assessments for use as endpoints in clinical trials in community-acquired bacterial pneumonia (CABP) and acute bacterial skin and skin structure infections (ABSSSIs). Two years later, the FDA requested an expansion of scope into hospital-acquired bacterial pneumonia (HABP) and ventilator-associated bacterial pneumonia (VABP). This manuscript provides an update on our work in these 4 important bacterial disease indications.

BACKGROUND

Recently, the FDA has undertaken a thorough scientific review of critical elements of antibacterial registrational trial design, including the appropriate criteria for use of a noninferiority trial design [1]. The FDA concluded that further work was required on design elements of noninferiority studies, including enrollment criteria, scientific justification for trial design, and well-defined and reliable outcome assessments as endpoints for registrational trials, in a number of indications, including ABSSSIs and CABP. This work would also aid in more efficiently designing superiority trials.

Historically, efficacy endpoints for CABP and ABSSSI registrational trials were based on clinician global assessments of resolution/improvement of signs and symptoms of infection after completion of antimicrobial therapy, with an important component being the clinician's judgment as to whether additional antimicrobial therapy was needed to successfully treat the infection. Interpretation of treatment effect based on clinician global assessments is more challenging when the exact clinical variables that clinicians should measure are inconsistently defined, or when it is not defined exactly how clinicians should combine these variables into a measure of “improvement” or “success” [2]. Different clinicians may, for example, have different definitions of directional change or success, and additional antimicrobial therapy may be prescribed for various reasons (lack of efficacy, an adverse effect, or another reason). A lack of clarity raises issues of reliability and reproducibility of clinician global assessments. Any resulting increased variability in outcome assessment can increase the needed sample size of trials (greater variability in a measured outcome variable means a greater number of observations are required to achieve adequate statistical power to detect or exclude differences caused by the trial intervention). Notably, documenting treatment effects on patient-centered outcomes can be more directly achieved by using PRO assessments than clinical global assessments, whereas observer-reported outcomes may be used in young children who cannot self-report [3, 4].

An additional goal of the FDA was to identify well-defined and reliable endpoints that could be evaluated at time points for which there was a maximal amount of prior evidence for a robust treatment effect. This information is required to construct reliable noninferiority margins for modern noninferiority trials. New FDA Guidances for ABSSSI and CABP registrational trials accordingly focus on assessment of efficacy at earlier time points than previously recommended in order to justify noninferiority hypotheses [5, 6]. For example, the ABSSSI Guidance references historical data from the preantibiotic era of treatment effects at 48–72 hours after initiation of therapy.

In 2010, given the FDA's uncertainty about appropriate noninferiority trial design, the Agency asked the Biomarkers Consortium of the FNIH to research and develop potential reliable, well-defined, and clinically relevant endpoints for registrational trials in CABP and ABSSSIs. The Consortium agreed to do so because of the urgent public health need, even though the task lay outside its usual purview related to biomarkers, and constituted a Project Team that included representatives from the FDA, the National Institute of Allergy and Infectious Diseases, academia, the Infectious Diseases Society of America (IDSA), and industry. Individual team participants (aside from FNIH staff) contribute their time entirely on a pro bono basis. The project aims to modernize and standardize the approach to the outcome assessments used as endpoints in clinical trials, thereby providing better information to patients and clinicians, increasing trial efficiency and limiting costs, and shortening the time to bringing new, safe, and efficacious antimicrobials to patients. Notably, patient symptom data have been captured in previous trials, but often in nonstandardized ways, so the team's efforts focused on standardizing assessments to decrease variability and increase reliability; in other words, the team should not be viewed as developing entirely “new” endpoints. These efforts should help address the concerns voiced by IDSA, among others, about the hurdles to developing improved antibacterial therapeutics [7–10].

PRIOR FNIH WORK IN ABSSSI AND CABP

An analysis of CABP and ABSSSI clinical trial data from previously completed studies contributed in-kind by FNIH Project Team members led to a series of interim recommendations submitted to the FDA docket in 2011 [11, 12] and subsequently summarized in this journal in 2012 [13]. Specifically, review of historical and modern data confirmed the FDA's conclusion that antimicrobial treatment effects are most apparent during the first few days of therapy. Based on evidence from the data reviewed, the FNIH recommended modifications to both the CABP and ABSSSI endpoints as proposed by the FDA. A further conclusion was that early clinical response endpoints provided a scientific justification for noninferiority hypotheses in CABP and ABSSSI registrational trials, thereby allowing evidence-based drug development to continue while further research on outcomes was conducted (Table 1). Of note, the FDA editorial accompanying the FNIH manuscript reassured pharmaceutical sponsors that the FDA would “accept efficacy endpoints based on improvement in symptoms for CABP and control of lesion spread for ABSSSI, even as further work is being done by the FNIH [project team] on its next phase of the project” [15].

Table 1.

Summary of Foundation for the National Institutes of Health Analyses and Interim Conclusions for Conduct of Registrational Trials in the Community-Acquired Bacterial Pneumonia, Acute Bacterial Skin and Skin Structure Infection, Hospital-Acquired Bacterial Pneumonia, and Ventilator-Associated Bacterial Pneumonia Indications

Indication	Methodology	Major Findings, Conclusions, and Recommendations	Date [Reference]
CABP	Review of literature; analysis of modern-day clinical trial data submitted in-kind by pharmaceutical sponsors	“Progressive improvement in four symptoms (cough, dyspnea, chest pain, and sputum production) during the first 4 d of therapy was sufficiently well documented that an early response endpoint measure could be proposed. To assess durability of response and other late events, supportive information should be obtained by assessing outcomes at a fixed timepoint after therapy has been completed. Such information could include a late response endpoint similar to the traditional test-of-cure endpoint. Although based on limited data and requiring further research, an early response endpoint can be used to anchor a non-inferiority trial for this indication. The early response endpoint is thus suggested for possible use by FDA in review of registrational trials and approval of applications in CABP while further research into this area is conducted.”	2011 [11]
ABSSSI	Review of literature; analysis of modern-day clinical trial data submitted in-kind by pharmaceutical sponsors	“Control of lesion spread at 48 to 72 h after randomization was sufficiently well documented that an early response endpoint measure could be proposed. To assess sustained response and other late events, supportive information should be obtained by assessing outcomes at a fixed time point after therapy has been completed. Such information could include a late response endpoint similar to the traditional test-of-cure (TOC) endpoint but more clearly defined. Although incompletely validated under the proposed conditions of use and requiring further research, an early response endpoint can be used to anchor a non-inferiority hypothesis in a trial for this indication. Thus, the Project Team supports a primary endpoint based on early response in review of registrational trials and approval of applications in ABSSSI while further research into outcomes at later time points in this area is conducted.”	2011 [12]
HABP	Review of literature	“A clinically meaningful endpoint of symptom improvement plus survival for non-ventilated patients could be based on the historical data for community-acquired bacterial pneumonia, for which there is a large treatment effect to day 7 of antibacterial drug therapy.” “There was some concern  …  that mortality and other differences between HABP and VABP suggest these are different diseases, meaning that combining both in a single trial could raise methodological issues.”	2013 [14]
VABP	Review of literature	“Despite the potential clinical trial implementation feasibility issues that have been raised with current FDA HABP/VABP Guidance, including an all-cause mortality (ACM) endpoint, most [FNIH] participants are comfortable with ACM as an endpoint, especially for VABP, if trial feasibility could be addressed by changing other parameters of study design.” “The outstanding questions for use of ACM relate to timing of its assessment, as well as to whether there are suitable intermediate clinical endpoints. One concern with ACM is its lower incidence in registrational trials versus “real life.” It is hypothesized that making exclusion criteria less restrictive, and thereby increasing the severity of illness in the enrolled population, has the potential to facilitate enrollment.” “A number of candidate changes to other aspects of trial design (eg, primary analysis set) were identified as promising potential approaches to improving feasibility, while maintaining scientific validity.”	2013 [14]

Abbreviations: ABSSSI, acute bacterial skin and skin structure infection; ACM, all-cause mortality; CABP, community-acquired bacterial pneumonia; FDA, US Food and Drug Administration; FNIH, Foundation for the National Institutes of Health; HABP, hospital-acquired bacterial pneumonia; VABP, ventilator-associated bacterial pneumonia.

The FNIH work informed new FDA ABSSSI and CABP Guidance documents and helped form the basis for FDA approval in 2014 of 3 new antimicrobial agents (tedizolid, oritavancin, dalbavancin) for treatment of ABSSSIs [16–18], as well as initiation of recently completed and ongoing registrational studies in CABP. These developments also stimulated retrospective and prospective analyses of the operational (study conduct) aspects of the early response endpoint, and its relationship to the traditional end-of-therapy and test-of-cure endpoints (Supplementary Materials). In general, the analyses confirmed that an early response endpoint provides clinically relevant, quantifiable, and reproducible data that justify noninferiority hypotheses, and are consistent with observations made later in the course of, or after, treatment that would allow for superiority hypotheses later in the disease course.

NEW FNIH RESEARCH INITIATIVES IN CABP AND ABSSSI

Subsequently, research was initiated to establish short- and long-term outcome measures that are well defined, reliable, and reflective of how patients feel, function, or survive (Supplementary Materials). The initial focus was development of a PRO instrument for ABSSSIs. The team subsequently began a similar project for CABP, and more recently for HABP.

PRO instruments capture the “patient voice”; that is, directly measure how patients describe and quantify their symptoms of infection (Supplementary Materials). PRO instruments have been of particular interest to the FDA recently, as exemplified by the release of its initial Guidance on PRO measures in 2006 and finalization of that Guidance in 2009 [19], and the Guidance for Qualification of Drug Development Tools, including PRO instruments [20]. Because mortality often is low with effective treatments, many patients who survive still experience significant symptoms, which also can be positively impacted by effective interventions.

The first 2 stages of PRO instrument development include (1) a review of the literature and qualitative research through patient and clinician interviews to determine which patient concepts of their illness are important to measure and (2) development of draft PRO questions based on results of the qualitative interviews, followed by interviews with a second independent group of patients to ascertain the completeness and patient understanding of the draft questions. These results for ABSSSI and CABP PRO development are summarized in the Supplementary Materials.

Using ABSSSI PRO development as an example, a 2-stage method of development was conducted with adult patients diagnosed with an ABSSSI within the past 7 days in the United States. Patients with a wound infection, cellulitis (including erysipelas), or major abscess were included. Cross-sectional, qualitative 1:1 telephone interviews were performed by trained interviewers using a semistructured interview guide. Item and concept generation was also augmented through a comprehensive ABSSSI literature review and interviews with 9 clinical experts in the United States and Europe.

Thirty-four patients from 4 clinical sites participated in concept elicitation interviews. Thirteen patients were diagnosed with major abscess, 12 with wound infection, and 9 with cellulitis. The main themes to emerge included signs (eg, enlargement, color), symptoms (eg, pain, swelling), and impacts on functionality (eg, social, physical) related to the skin infection. The most commonly reported symptoms included experiencing pain (n = 32), swelling (n = 31), and drainage or leakage at the site of the infection (n = 27). The CABP and ABSSSI draft instruments are now ready to enter the third stage of psychometric evaluation of measurement properties via testing in selected registrational trials.

The FNIH Biomarkers Consortium Project Teams do not view PRO instruments as the sole outcome assessment for use as the endpoint for registrational trials in these indications. PRO measures may be assessed with other outcomes. In addition, it is not possible to measure symptoms in patients who are unable to communicate or who have died, so PRO measures are a useful adjunct to other outcomes such as survival and development of disease complications, and also could be used in addition to clinicians' assessments of skin infection lesion size. The availability of validated and FDA-qualified PRO instruments would add to the “toolbox” of options for sponsors to use in future registrational trials in these indications to measure outcomes in patients who survive but have experienced significant disease symptoms. In addition, appropriately evaluated PRO instruments can be used outside the setting of clinical trials evaluating medical interventions. They can be used to standardize measurements in epidemiological studies evaluating natural history and burden of disease, as well as form part of development of “severity” scales that could be included among the inclusion criteria for future trials.

INITIATIVES IN HABP AND VABP

Prior FDA draft HABP-VABP Guidance documents have elicited concerns from stakeholders about high logistical challenges in terms of trial sample size, time, and cost. The focus on all-cause mortality at 28 days after initiation of therapy as the primary endpoint elicited multiple comments, including that comorbid conditions alone could exert a substantial effect on mortality rates (although antibiotic effects in community-acquired pneumonia have been shown to be large and reproducible) [21]. Also, all-cause mortality does not measure the effects of interventions on symptoms and function in the more numerous group of patients who survive, and therefore does not assess other important outcomes [22].

The core question of HABP-VABP trial feasibility relates not just to choice of the primary endpoint, but also to other interrelated issues of trial design including enrollment criteria, statistical analysis populations, and sample size. Development of well-defined and reliable outcome assessments has the potential to decrease HABP-VABP trial sample size without compromising the scientific integrity of the data produced and the robustness of conclusions drawn.

The FNIH team prepared a HABP-VABP “Interim Considerations” document that was submitted to the FDA docket in 2013 (Supplementary Materials) [14]. The assessment confirmed the relevance of all-cause mortality as an endpoint in HABP-VABP trials, but noted its limitations. Recommendations included allowing, in some scenarios, registration based on a single pivotal trial, with the primary analysis conducted in the intention-to-treat analysis population instead of the microbiologically confirmed analysis population. Further work includes consideration of a “mortality-plus” endpoint—that is, use of a multicomponent assessment of all-cause mortality plus selected serious adverse events/complications of disease of clear relevance to how patients feel and function, such as pleural empyema or respiratory failure requiring mechanical ventilation. The effort is evidence-based as it will analyze data contributed in-kind by sponsors of recent HABP-VABP trials. The most recent effort is development of a PRO instrument for HABP trials, an initiative generously funded by the FDA itself through a Broad Agency Announcement (FDABAA-13-0019).

FURTHER CONSIDERATIONS

Global harmonization of endpoints in the CABP and ABSSSI indications is critically important, as the European Medicines Agency continues to rely on physician global assessments of outcomes late in the time course of therapy. The current situation is problematic for sponsors conducting registrational trials for marketing authorization in the United States and the European Union because of the need in each trial for 2 very different statistical analysis plans with resulting conflicting sample size requirements. Developing endpoints for pediatric trials in these indications, and probably others (eg, osteomyelitis), is also vitally important. For example, minimum skin infection size requirements in adults may be impossible and irrelevant to apply to children.

CONCLUSIONS

Advancing the science of outcome assessments helps all stakeholders make better decisions in development of new interventions and can provide patients and clinicians with evidence on patient-centered outcomes reflecting the benefits and harms of medical interventions. The efforts of the Biomarkers Consortium of the FNIH aim to enrich the latest developments in the science of clinical trials, while addressing concerns about the scientific validity, feasibility, and rigor of such studies. New evidence-based recommendations for trial design plus the introduction of new PRO instruments will represent steps forward in the continuing process of incorporating new understandings of regulatory science into the regulatory framework, to the benefit of patients and their physicians.

Supplementary Data

Supplementary materials are available at http://cid.oxfordjournals.org. Consisting of data provided by the author to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the author, so questions or comments should be addressed to the author.