Moving Beyond Mortality: Development and Application of a Desirability of Outcome Ranking (DOOR) Endpoint for Hospital-Acquired Bacterial Pneumonia and Ventilator-Associated Bacterial Pneumonia

Abstract

Background

Hospital-acquired and ventilator-associated bacterial pneumonia (HABP/VABP) are frequently caused by multidrug-resistant organisms. Patient-centered endpoints in clinical trials are needed to develop new antibiotics for HABP/VABP. Desirability of outcome ranking (DOOR) is a paradigm for the design, analysis, and interpretation of clinical trials based on a patient-centered, benefit-risk evaluation.

Methods

A multidisciplinary committee created an infectious diseases DOOR endpoint customized for HABP/VABP, incorporating infectious complications, serious adverse events, and mortality. We applied this to 2 previously completed, large randomized controlled trials for HABP/VABP. ZEPHyR compared vancomycin to linezolid and VITAL compared linezolid to tedizolid. For each trial, we evaluated the DOOR distribution and probability, including DOOR component and partial credit analyses. We also applied DOOR in subgroup analyses.

Results

In both trials, the HABP/VABP DOOR demonstrated similar overall clinical outcomes between treatment groups. In ZEPHyR, the probability that a participant treated with linezolid would have a more desirable outcome than a participant treated with vancomycin was 50.2% (95% confidence interval [CI], 45.1%­−55.3%). In VITAL, the probability that a participant treated with tedizolid would have a more desirable outcome than a participant treated with linezolid was 48.7% (95% CI, 44.8%–52.6%). The DOOR component analysis revealed that participants treated with tedizolid had a less desirable outcome than those treated with linezolid when considering clinical response alone. However, participants with decreased renal function had improved overall outcomes with tedizolid.

Conclusions

The HABP/VABP DOOR provided more granular information about clinical outcomes than is typically presented in clinical trials. HABP/VABP trials would benefit from prospectively using DOOR.

A multidisciplinary committee developed an infectious diseases desirability of outcome ranking (DOOR) endpoint and demonstrated how this could be applied to 2 randomized trials for hospital-acquired and ventilator-associated bacterial pneumonia.

Hospital-acquired bacterial pneumonia (HABP) and ventilator-associated bacterial pneumonia (VABP) are common healthcare-associated infections with high rates of morbidity and mortality [1–3]. Treatment decisions can be complex as HABP/VABP is frequently caused by multidrug-resistant organisms (MDROs) [3, 4]. In an analysis by the United States Food and Drug Administration (FDA) of 4 HABP/VABP randomized controlled trials (RCTs), nearly 20% of all gram-negative pathogens were resistant to meropenem, including 79% of Acinetobacter baumannii isolates [3]. As the development of antibiotic resistance continues to outpace the availability of new antibiotics, there is a critical need to develop and assess antibiotics for HABP/VABP [5–7].

HABP/VABP clinical trials are challenging and resource intensive due to high patient acuity, complex protocols, and low patient enrollment [3, 6, 8]. It is therefore critical to ensure that trial endpoints directly inform treatment decisions. The 2020 FDA guidance for HABP/VABP drug development allows the use of 14- or 28-day all-cause mortality as the primary endpoint [9]. However, both the Foundation for the National Institutes of Health (FNIH) Biomarkers Consortium and the Clinical Trials Transformation Initiative, with input from the FDA, have raised concerns about using mortality alone in noninferiority HABP/VABP trials, and recommended combining mortality with other relevant adverse events (AEs) [6, 10, 11]. The FDA has also encouraged novel endpoint development aimed at understanding how patients feel and function [11].

Desirability of outcome ranking (DOOR) is a paradigm for the design, analysis, and interpretation of research based on patient-centric benefit-risk evaluation [12–14]. Using ordinal ranking, DOOR evaluates a patient's entire clinical course. Unlike in traditional registrational trials where the safety population is analyzed separately from the efficacy population, DOOR combines the safety and efficacy evaluations and allows for a more comprehensive understanding. DOOR partial credit analyses also allow patients and clinicians to choose the relative weight of DOOR events.

The Antibacterial Resistance Leadership Group (ARLG) has established the use of DOOR in observational studies addressing antibiotic resistance [15–17]. More recently, through the work of a multidisciplinary committee, we created an infectious diseases DOOR and demonstrated its use in complicated urinary tract infection (cUTI) and complicated intra-abdominal infection (cIAI) trials [18, 19]. Here, we used data from 2 multicenter, double-blind RCTs [20, 21] to develop a DOOR endpoint for HABP/VABP and demonstrate how DOOR can be applied.

METHODS

DOOR Task Force and Development of the DOOR Analysis Strategy

As previously described, in 2020 ARLG created a multidisciplinary DOOR Task Force [18]. This group includes experts in infectious diseases, trial design, statistical analysis, drug regulation, quality of life, and patient advocacy and also includes individuals from academia, the FDA, the National Institutes of Health, the pharmaceutical industry, and a patient advocacy group. Our aim was to develop a DOOR analysis strategy that could be tailored to common infectious diseases and used in registrational trials for novel anti-infectives. Through consensus building and iterative feedback we agreed upon a DOOR analysis strategy (Figure 1A) that was adapted from work in Staphylococcus aureus bacteremia [14]. We then defined each event category for infectious diseases commonly used as entry indications for new anti-infectives, including HABP/VABP, cUTI, cIAI, and acute bacterial skin and skin structure infections [18]. The HABP/VABP DOOR endpoint is presented in this manuscript (Figure 1B).

Figure 1.

Desirability of outcome ranking (DOOR) analysis strategy. A, Generalized DOOR analysis strategy that could be applied to any infectious diseases clinical trial [18]. B, How DOOR component events were defined a priori for complicated hospital-acquired and ventilator-associated bacterial pneumonia (HABP/VABP) trials. ^aHealth-related quality of life indicators, when available, could be used as a tiebreaker for patients with the same rank. ^bDefined as lack of global resolution of index infection or recurrence of index infection before test of cure. ^cDefined as a newly identified complication or progression of the original infection that was not present at enrollment, including the development of Clostridioides difficile.  ^dDefined according to International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use E6 Good Clinical Practice guidelines. ^eOsteomyelitis was added after the initial review of adverse events from the trials with agreement by the Antibacterial Resistance Leadership Group (ARLG) Innovations Committee. ^fCould not be used as an event in VITAL study as all patients were ventilated at enrollment.

Selection and Description of the HABP/VABP Trials

The DOOR Task Force contacted 2 pharmaceutical companies and the FNIH to inquire about performing DOOR analyses on HABP/VABP datasets. Pfizer and Merck agreed to share deidentified data, in kind, from ZEPHyR [20] and VITAL [21], respectively. Both trials were multicenter, double-blind RCTs [20, 21]. ZEPHyR enrolled participants with hospital-acquired or healthcare-associated methicillin-resistant S. aureus (MRSA) pneumonia and compared intravenous vancomycin to linezolid. VITAL enrolled ventilated participants with hospital-acquired or ventilator-associated pneumonia caused by gram-positive organisms and compared intravenous linezolid to tedizolid. ZEPHyR was conducted prior to the updated FDA Guidance on HABP/VABP [9] and used the clinical outcome at end of study (EOS) as the primary endpoint. VITAL, conducted more recently, analyzed 2 primary endpoints: clinical outcome at test of cure (TOC) and 28-day all-cause mortality. Both studies followed participants for AEs for a similar duration; however, in ZEPHyR the TOC visit was the same as the EOS visit (Table 1).

Table 1.

Study Characteristics of the Randomized Controlled Trials Analyzed Using Desirability of Outcome Ranking

Characteristic	ZEPHyR	VITAL
No. of participants in mITT populationa	448	718
Study design	Phase 4, multicenter, double-blind RCT	Phase 3, multicenter, double-blind, double-dummy RCT
Study population	Hospital-acquired or healthcare-associated MRSA pneumonia	Ventilated hospital-acquired or ventilator-associated gram-positive pneumonia
Dates of enrollment	Oct 2004–Jan 2010	Jun 2014–Jun 2018
Study drugs	Vancomycin vs linezolid	Linezolid vs tedizolid
Duration of therapy	7–14 d	7 d tedizolid; 10 d linezolid
Original primary endpoint	Clinical outcome at end of study	Clinical outcome at TOC and 28-d all-cause morality
Test of cureb	EOT + 7–30 d	EOT + 7–14 d
Time frame for monitoring adverse events, d	EOT + 7–30b (∼days 14–44)	Days 28–32

Abbreviations: EOT, end of therapy; mITT, modified intention to treat; MRSA, methicillin-resistant Staphylococcus aureus; RCT, randomized controlled trial; TOC, test of cure.

^amITT was defined as all randomized patients who received ≥1 dose of study drug.

^bIn ZEPHyR, this was labeled the end of study visit.

Application of HABP/VABP DOOR and Variable Abstraction

We retrospectively abstracted relevant data from ZEPHyR and VITAL to determine how many DOOR events each participant experienced. The DOOR events included absence of clinical response, infectious complications, and serious adverse events (SAEs) (Figure 1).

Absence of Clinical Response

For this event, we used the primary study's assessment of clinical outcome at TOC. In the primary DOOR analysis, we considered any participant who did not meet the study definition of clinical success (ie, classified as clinical failure or indeterminate) as having absence of clinical response. Clinical cure was defined differently in each study but generally included participants who were alive with resolution of their presenting signs and symptoms, without any new symptoms of pneumonia or need for additional antibiotics for pneumonia. For ZEPHyR, we used the blinded, sponsor-adjudicated assessment of clinical outcome (differences described in Supplementary Table 1).

Infectious Complications

Two board-certified infectious disease clinicians (J. H.-A., H. W. B.) reviewed all AE events in the Medical Dictionary for Regulatory Activities (MedDRA) system organ class of “infections and infestations” and “respiratory, thoracic and mediastinal disorders” to determine which met criteria for the prespecified DOOR infectious complications (Figure 1B). AEs were reviewed according to a DOOR Task Force standard operating procedure. During the review, an additional AE (osteomyelitis due to the same bacteria identified in the enrollment culture) was included as an infectious complication as the reviewers determined it to be consistent with the infectious complication definition. All infectious complications had to be identified after enrollment and occur during the AE monitoring period (Table 1). Respiratory failure requiring mechanical ventilation could not be used as an infectious complication in VITAL as mechanical ventilation was a requirement for study enrollment. In ZEPHyR, we determined if participants met criteria for respiratory failure requiring mechanical ventilation by assessing all AEs included in the “respiratory, thoracic, and mediastinal disorders” system organ class and including anyone with a new requirement for mechanical ventilation after the start of treatment. The reviewers were blinded to the treatment and agreement between reviewers had to be unanimous. Any events that were unable to be resolved were taken back to the full DOOR Task Force for review.

SAEs

We included all participants coded as having an SAE during the follow-up period used for AE monitoring (Table 1). SAEs were defined according to International Council for Harmonisation of Technical Requirements for Pharmaceuticals for Human Use E6 Good Clinical Practice guidelines (21 Code of Federal Regulations 312.32) and included any event that (1) resulted in death, (2) was life-threatening, (3) required inpatient hospitalization or prolongation of existing hospitalization, (4) resulted in persistent or significant disability/incapacity, or (5) was a congenital anomaly/birth defect [22].

Ranking

After determining how many DOOR events (absence of clinical response, infectious complications, and SAEs) occurred for each participant, a mutually exclusive rank was assigned. Rank 1 is the most desirable outcome and includes participants who were alive and did not experience any of the undesirable events. Rank 5 is the least desirable outcome and includes participants who died. Ranks 2 through 4 include participants who were alive but had 1, 2, or 3 events, respectively (Figure 1A). If a participant had >1 event in the same category (eg, 2 SAEs), this was classified as meeting only 1 event category. However, if an infectious complication also met criteria for an SAE, this was categorized as having 2 event categories.

Statistical Analysis

For both trials, our primary analysis used the modified intention-to-treat (mITT) population, which was defined as all randomized participants who received at least 1 dose of study drug. For ZEPHyR, the mITT population only included participants with MRSA as the cause of pneumonia. In an exploratory analysis of ZEPHyR, we also analyzed the primary study's evaluable, per-protocol population to allow for a direct comparison with the published results [20].

In the primary analysis we compared the DOOR distribution between treatment groups and calculated the DOOR probability (ie probability of having a more desirable outcome in 1 treatment group compared to the other) by Wilcoxon-Mann-Whitney statistic with a 2-sided 95% confidence interval (CI) [23]. A DOOR probability of 50% indicates no statistical difference between groups. For each DOOR component, we also calculated the probability specific to that event. Additionally, we derived prioritized DOOR probabilities, 1 prioritizing efficacy and 1 prioritizing safety. When comparing 2 participants with the same number of undesirable events, the efficacy DOOR prioritizes avoidance of clinical failure over SAEs or infectious complications, whereas the safety DOOR prioritizes the avoidance of SAEs and infectious complications over clinical failure [18].

We performed subgroup analyses in which we calculated the DOOR probability for participants with specific clinical characteristics. The subgroups were chosen by the DOOR Task Force prior to analysis, based on the clinical relevance and the availability of the variables in both datasets. In a sensitivity analysis, we changed our classification of participants with indeterminate or missing clinical outcomes in 3 ways: (1) Participants with indeterminate or missing outcomes were ranked above those with clinical failure if they otherwise had the same rank (“tiebreaker” analysis); (2) participants with indeterminate or missing outcomes were counted as “clinical cure”; and (3) participants with indeterminate or missing outcomes were excluded.

Partial Credit Analysis

We completed a DOOR partial credit analysis using 3 hypothetical scoring keys (Scenarios A, B, and C) that were also used in the cUTI DOOR analysis [18]. One advantage of the partial credit scoring approach is that it can allow for personalized grading of DOOR ranks. As previously described, in a partial credit analysis, the DOOR categories are scored like an academic test. Rank 1 (most desirable) is given a score of 100 and Rank 5 (least desirable) is given a 0. Ranks 2–4 are given “partial credit,” which can be any score between 0 and 100, as long as the original rank order is maintained. Patients or clinicians can adjust the scores given to Ranks 2–4 based on their own preferences or values. For example, one patient may believe that having 1 undesirable event may not be that impactful and give Rank 2 a score of 90, but another patient may decide that any undesirable events would severely hinder their quality of life and give Rank 2 a score of 30. Treatment arms are compared by calculating the difference between the mean partial credit scores in each group. A difference of zero indicates no significant difference between the 2 groups. In a prospective clinical trial, the partial credit grading key should be prespecified to ensure transparency and reproducibility.

Analyses were not adjusted for multiple comparisons. All analyses were conducted using SAS version 9.4 (SAS Institute, Cary, North Carolina) or R version 4.0.3 (R Foundation for Statistical Computing, Vienna, Austria) statistical software.

RESULTS

Four hundred forty-eight participants were included in the mITT population for ZEPHyR and 718 participants were included in the mITT population for VITAL (Table 1). In ZEPHyR, <40% of the participants had the most desirable outcome (alive with zero events), and the mortality rate was <20%. In VITAL, a greater proportion had the most desirable outcome (45%–48%), but the mortality rate in both arms was higher (approached 30%) (Figure 2).

Figure 2.

Desirability of outcome ranking distribution by treatment groups for ZEPHyR and VITAL. The events include absence of clinical response, infectious complications, serious adverse events, and death (definitions included in Figure 1). Abbreviations: LZD, linezolid; TZD, tedizolid; VAN, vancomycin.

For both trials the DOOR distribution between treatment arms was similar (Figure 2). In ZEPHyR, the probability that a participant treated with linezolid would have a more desirable outcome than a participant treated with vancomycin was 50.2% (95% CI, 45.1%­–55.3%). In VITAL, the probability that a participant treated with tedizolid would have a more desirable outcome than a participant treated with linezolid was 48.7% (95% CI, 44.8%–52.6%). Significant differences between groups were not demonstrated with the DOOR probabilities. The probabilities were similar in the DOORs prioritized for efficacy or safety (Figure 3).

Figure 3.

Forest plot displaying the overall desirability of outcome ranking (DOOR) probability as well as the probability for the prioritized DOOR scales and the DOOR components (treatment failure, infectious complications, serious adverse events, and death). Abbreviations: CI, confidence interval; DOOR, desirability of outcome ranking; LZD, linezolid; SAE, serious adverse event; TZD, tedizolid; VAN, vancomycin.

As DOOR can be considered a composite outcome, it is important to analyze the individual components of DOOR (Figure 3, Supplementary Table 2). For ZEPHyR, we did not observe a difference between treatment arms in any of the individual DOOR components (absence of clinical response, infectious complications, SAEs, or death). However, in VITAL, participants in the tedizolid group had a less desirable outcome when considering clinical response (probability, 46.3% [95% CI, 42.8%–49.9%]). This was balanced by participants in the tedizolid group having a more favorable outcome when assessing SAEs, although this difference was not significant (probability, 52.3% [95% CI, 48.7%–55.9%]).

To compare our results directly with the original ZEPHyR analysis, we performed an exploratory analysis using the evaluable, per-protocol population used in the primary publication (n = 339). In this subset, the primary DOOR analysis revealed no difference in the overall global outcome between the 2 groups (DOOR probability, 52.7% [95% CI, 46.9%–58.5%]). However, in the DOOR component analysis, participants receiving linezolid had a more desirable outcome for clinical efficacy than participants receiving vancomycin; no difference was observed in the other DOOR components (Supplementary Figure 1).

In subgroup analyses (Figure 4), we did not observe a difference between linezolid and vancomycin in any of the groups defined by age, presence of diabetes, type of pneumonia, and renal function. However, in VITAL, participants with the best renal function (creatine clearance [CrCl] ≥90 mL/minute) had a more desirable outcome with linezolid, while participants with the worst renal function (CrCl <15 mL/minute) had a more desirable outcome with tedizolid. Additionally, those with moderately decreased renal function (CrCl <60 mL/minute) had a DOOR probability >50% indicating a trend toward improved outcomes with tedizolid.

Figure 4.

Estimated desirability of outcome ranking probabilities and associated 95% confidence intervals for all subgroup analyses. Abbreviations: CI, confidence interval; DOOR, desirability of outcome ranking; eGFR, estimated glomerular filtration rate; HAPB, hospital-acquired bacterial pneumonia; LZD, linezolid; TZD, tedizolid; VABP, ventilator-associated bacterial pneumonia; VAN, vancomycin.

In the DOOR partial credit analysis, we did not observe differences between treatment groups for any of the 3 hypothetical scenarios (Supplementary Figure 2). Results did not meaningfully change in sensitivity analyses where we modified how we categorized participants with missing and indeterminate outcomes (Supplementary Figure 3).

DISCUSSION

We have demonstrated that an infectious diseases DOOR analysis strategy can be adapted and successfully applied to HABP/VABP RCTs. The HABP/VABP DOOR is designed to be patient-centered, encompassing more than a single assessment of clinical efficacy or mortality. DOOR includes issues that may arise throughout a patient's follow-up or recovery period including infectious complications, SAEs, or death. DOOR provides clinicians with more detailed information than typically presented in clinical trials and we believe DOOR should be prospectively included in future HABP/VABP trials. A free web-based application (https://methods.bsc.gwu.edu) is now available to conduct comprehensive DOOR analyses.

In both ZEPHyR and VITAL, the primary DOOR analysis revealed similar global outcomes between treatment groups. However, in VITAL, the DOOR component analysis revealed that linezolid may be superior to tedizolid in terms of clinical efficacy. This was balanced by a nonsignificant trend of increased SAEs in the linezolid group. This is an example of how DOOR can help elucidate potential tradeoffs between benefits and harms of new medications and can be used to inform shared decision-making discussions between patients and clinicians. Additionally, regulatory agencies could explore utilizing the prioritized versions of DOOR to help determine if new treatments should be approved and in what clinical scenarios the benefit: risk ratio is most favorable.

Notably, in ZEPHyR, the clinical effectiveness of linezolid was superior to vancomycin for the treatment of MRSA pneumonia [20]. However, using DOOR, we found that the overall clinical outcome was similar between the 2 groups. The possible improved clinical efficacy seen with linezolid was offset by nonsignificant increases in infectious complications, SAEs, and death. This analysis has the potential to change practice patterns of clinicians who frequently use linezolid to treat MRSA pneumonia.

Using DOOR in subgroup analyses can also identify which patients would most benefit from new antibiotics. In VITAL, we observed that outcomes were better in participants with kidney disease treated with tedizolid compared with linezolid. Given the small number of participants with kidney disease and the number of subgroups analyzed, this finding should be interpreted with caution, viewed as hypothesis generating, and viewed in context of prior work demonstrating that renal impairment can increase the risk of thrombocytopenia on linezolid [24]. Patients with HABP/VABP are often critically ill with MDROs [3]. In these cases, clinicians may have to choose between a new antibiotic that may not have much published data versus an older antibiotic with known risks of AEs. DOOR can allow for a better understanding of these risks, considering baseline comorbidities or specific infection characteristics.

Our study is strengthened by the fact that we analyzed 2 large RCTs for HABP/VABP spanning many years. Additionally, the DOOR endpoint was informed by a multidisciplinary group of stakeholders including patient advocates and experts in antibiotic development, regulation, and clinical trial design. Data were provided by pharmaceutical companies to support endpoint development, but they did not sponsor this study or directly participate in the analysis.

The study also has limitations. First, the review of infectious complications was limited to coded data, which often do not capture the full extent of the AE. Specifically we were not able to determine the cause of mechanical ventilation. Second, as our study was retrospective, we could not change the definition of clinical efficacy or include any patient-reported assessments of health-related quality of life. We believe this is important information to capture in future clinical trials. Third, the studies were not originally designed to detect differences in DOOR outcomes. During trial design, the sample size would be selected to detect a clinically meaningful difference in DOOR distributions based on the DOOR probability. The ARLG is creating a DOOR sample-size and power assessment tool that will be included in the web-based application. Fourth, because we analyzed large RCTs that have stringent enrollment criteria, our DOOR analysis may not be generalizable to the highest-risk patients frequently diagnosed with HABP/VABP.

In conclusion, we have demonstrated that DOOR is feasible to use in HABP/VABP clinical trials and allows for a more comprehensive understanding of the risks and benefits of novel therapeutics. We believe DOOR should be used prospectively in RCTs as an endpoint that provides more actionable information to patients, clinicians, and researchers. Future work is needed to understand how to incorporate patient-reported outcomes, specifically those related to health-related quality of life, into the DOOR endpoint.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Acknowledgments. The authors are grateful to all study investigators, staff, and participants who participated in the trials analyzed in this manuscript (ZEPHyR and VITAL). The authors acknowledge Pamela Sears, Carisa De Anda, Amanda Paschke, and Natalya Broyde from Merck & Co for sharing data from VITAL and answering questions related to the dataset. The authors are also thankful to Rienk Pijpstra from Pfizer for collaborating and sharing data from ZEPHyR.

ARLG Innovations Working Group Members. Helen Boucher (Chair), Tufts University; Chip Chambers, University of California San Francisco; Sara Cosgrove, Johns Hopkins University; Sarah Doernberg, University of California San Francisco; Scott Evans, George Washington University; Toshi Hamasaki, George Washington University; Tom Holland, Duke University; Jessica Howard-Anderson, Emory University; Vance Fowler, Duke University; Heather King, Duke University; Sumati Nambiar, Johnson & Johnson; Ephraim Tsalik, Danaher; Ramya Gopinath, United States Food and Drug Administration; Peter Kim, United States Food and Drug Administration; Mukil Natarajan, United States Food and Drug Administration; Mark Needles, United States Food and Drug Administration; Dan Rubin, United States Food and Drug Administration; Ursula Waack, United States Food and Drug Administration; Deborah Collyar, Patient Advocates in Research.

Disclaimer. The content of this work is solely the responsibility of the authors and should not be construed to represent the official views or policies of the National Institutes of Health (NIH) or the US Food and Drug Administration (FDA).

Financial support. This work was supported by the National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health (award number UM1AI104681). This work was also supported in part by an appointment to the Research Participation Program at the US Food and Drug Administration administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the US Department of Energy and the FDA. D. C. also reports honorarium to author from the Antibacterial Research Leadership Group (ARLG)/NIH in support of this work. J. H.-A. reports funding paid to institution for this work from ARLG/NIH. V. G. F. reports support for this work from NIH (grant number 1R01-AI165671).