Exploration of a Potential Desirability of Outcome Ranking Endpoint for Complicated Intra-Abdominal Infections Using 9 Registrational Trials for Antibacterial Drugs

Abstract

Background

Desirability of outcome ranking (DOOR) is a novel approach to clinical trial design that incorporates safety and efficacy assessments into an ordinal ranking system to evaluate overall outcomes of clinical trial participants. Here, we derived and applied a disease-specific DOOR endpoint to registrational trials for complicated intra-abdominal infection (cIAI).

Methods

Initially, we applied an a priori DOOR prototype to electronic patient-level data from 9 phase 3 noninferiority trials for cIAI submitted to the US Food and Drug Administration between 2005 and 2019. We derived a cIAI-specific DOOR endpoint based on clinically meaningful events that trial participants experienced. Next, we applied the cIAI-specific DOOR endpoint to the same datasets and, for each trial, estimated the probability that a participant assigned to the study treatment would have a more desirable DOOR or component outcome than if assigned to the comparator.

Results

Three key findings informed the cIAI-specific DOOR endpoint: (1) a significant proportion of participants underwent additional surgical procedures related to their baseline infection; (2) infectious complications of cIAI were diverse; and (3) participants with worse outcomes experienced more infectious complications, more serious adverse events, and underwent more procedures. DOOR distributions between treatment arms were similar in all trials. DOOR probability estimates ranged from 47.4% to 50.3% and were not significantly different. Component analyses depicted risk-benefit assessments of study treatment versus comparator.

Conclusions

We designed and evaluated a potential DOOR endpoint for cIAI trials to further characterize overall clinical experiences of participants. Similar data-driven approaches can be utilized to create other infectious disease–specific DOOR endpoints.

Utilizing patient-level data from 9 registrational phase 3 clinical trials for complicated intra-abdominal infection (cIAI), we defined and evaluated a desirability of outcome ranking (DOOR) endpoint for cIAI that provides a novel approach to understanding overall outcomes among trial participants.

Desirability of outcome ranking (DOOR) uses an ordinal ranking system to incorporate safety and efficacy assessments into a single endpoint [1], potentially allowing for a more granular evaluation of the spectrum of experiences that clinical trial participants may have within categories of clinical response. This may distinguish DOOR from traditional trial analyses, which evaluate safety and efficacy separately, albeit in the same study population.

The primary objective of our study was to define and evaluate a DOOR endpoint in clinical trials for complicated intra-abdominal infections (cIAIs). Per United States Food and Drug Administration (FDA) guidance, the recommended primary endpoint for a cIAI trial is clinical response at the test-of-cure visit using a noninferiority margin of 10% [2]. “Clinical success” is defined as resolution of baseline signs and symptoms of cIAI. “Clinical failure” includes death, surgical site wound infection, unplanned procedure for complication or recurrence of cIAI, or initiation of nontrial antibacterial drug therapy [2]. Through finer elucidation of the spectrum of participant experience, a DOOR endpoint may allow us to better understand the clinical benefit of the primary endpoint and could provide a more nuanced, benefit-risk assessment.

We performed our study through a unique collaboration between the Antibacterial Resistance Leadership Group (ARLG) Innovations Working Group (IWG) and the FDA. The ARLG IWG previously developed a DOOR endpoint for Staphylococcus aureus bloodstream infections [3]; this endpoint has been used as a template for other infectious diseases such as complicated urinary tract infections [4], and led to a proposed a priori DOOR prototype containing infectious complications deemed by the IWG to be clinically relevant to cIAI (Supplementary Tables 1 and 2).

We tested the a priori DOOR prototype on the FDA's database of registrational trials for cIAI with the intent to refine the prototype based on observations from the data. The work evolved through a 2-step process: (1) application of the a priori DOOR prototype to our database, modification of the prototype based on the totality of participant experiences, and derivation of a cIAI-specific DOOR endpoint; and (2) application of the cIAI-specific DOOR endpoint to our database with estimation of the DOOR probabilities, analyses of DOOR components, and subgroup analyses.

METHODS

We compiled a list of adult antibacterial trials from new drug applications (NDAs) for cIAI submitted to the Division of Anti-Infectives, Office of Infectious Diseases, Center for Drug Evaluation and Research, FDA, from 2005 through 2019. Patient-level data were analyzed from 9 phase 3, randomized, noninferiority, double-blind clinical trials (labeled trials A–I) from the 6 NDAs submitted; all but 1 NDA was approved. Due to variation in trial design, the DOOR endpoint was applied and evaluated in individual trials rather than in aggregated data.

The study population included hospitalized participants with cIAIs, including appendicitis with perforation or peri-appendiceal abscess, cholecystitis with perforation or abscess, diverticulitis with perforation or abscess, gastric or intestinal perforation, intra-abdominal abscess, or purulent peritonitis due to a perforated viscus, all necessitating acute surgical or percutaneous intervention within 24–48 hours of enrollment. In accordance with trial protocols, all 9 trials excluded patients who required staged abdominal repair, laparostomy, or marsupialization. We analyzed all participants in the microbiological intention-to-treat (micro-ITT) population which, with minor variations, was defined as all randomized participants who received any amount of study drug and had a cIAI pathogen identified at baseline. The micro-ITT population was chosen as this was the primary analysis population in all trials in our dataset.

Application of DOOR Prototype and Subsequent Derivation of cIAI-Specific DOOR Endpoint

First, we applied the a priori DOOR prototype (Supplementary Tables 1 and 2) to our database and evaluated the extent to which the prototype's existing components of absence of clinical response, infectious complications (ICs), and serious adverse events (SAEs) captured the overall outcomes of individual participants with cIAIs. We utilized the trial investigator's classification of clinical response at the test-of-cure visit, as well as their reporting of both adverse events (AEs) and SAEs.

Absence of clinical response included participants classified, according to trial protocol definitions, as having an indeterminate response (Supplementary Figure 1) or clinical failure. If a participant's case was reviewed by the surgical review panel (SRP), the SRP's clinical assessment prevailed.

Infectious complications in the a priori DOOR prototype were prespecified by the ARLG IWG (Supplementary Table 1) and defined as clinically relevant infectious AEs that occurred in the Medical Dictionary for Regulatory Activities (MedDRA) System Organ Class (SOC) “Infections and Infestations.”

SAEs were categorized by the investigator as defined by the Code of Federal Regulations 21 C.F.R. § 312.32 [5].

All ICs, SAEs, and deaths that occurred after enrollment and during the period of monitoring for AEs, ranging from 28 to 45 days across trials, were included in our analysis. In addition to these 3 categories of clinical experiences, we noted that a greater proportion of participants underwent additional surgical/percutaneous procedures, compared to the proportions of participants who had at least 1 IC or SAE (Figure 1). In our estimation, this warranted modification of the DOOR prototype to include a separate category for additional procedures relevant to cIAI, contributing to the derivation of a cIAI-specific DOOR endpoint (Table 1).

Figure 1.

More participants experienced either 1 or 2 infectious complications (ICs), serious adverse events (SAEs), or additional procedures than those who experienced ≥3 events per category. Additional procedures occurred more commonly than ICs or SAEs.

Table 1.

Component Definitions for Complicated Intra-abdominal Infection–Specific Desirability of Outcome Ranking Endpoint

Component Events	Definition
Absence of clinical response	Includes outcomes of clinical failure or indeterminate as assessed by the investigator at the TOCa visit. If the participant's clinical course was reviewed by the SRP, the SRP's clinical assessment prevails.
Infectious complications	Newly identified infections that were not initially diagnosed at the start of the trial, including those related and unrelated to the original cIAI.
Surgical/percutaneous procedures	Any additional abdominal interventions, to include surgical, percutaneous, or endoscopic procedures, that the participant has after their first operation for cIAI. Any postoperative wound-related surgical or percutaneous interventions that the participant has after their first operation for cIAI.
Serious adverse events	Includes SAEs as defined by the Code of Federal Regulations (21 C.F.R. § 312.32)b,c

Abbreviations: cIAI, complicated intra-abdominal infection; SAE, serious adverse event; SRP, surgical review panel; TOC, test of cure.

The time frame for the TOC visit varied slightly by trial, as did the point from which the TOC was measured. For example, some trials measured the TOC from randomization, while others measured from the first or last dose of therapy. The earliest TOC visit was conducted 14 days after randomization, while the latest TOC visit was conducted 50 days after randomization.

Any medical event that (1) results in death, (2) is life-threatening, (3) requires inpatient hospitalization or prolongation of existing hospitalization, (4) results in persistent or significant disability/incapacity, or (5) is a congenital anomaly/birth defect.

If an SAE was also in the infectious complication component, this counted as 2 events for the desirability of outcome ranking.

Application of cIAI-Specific DOOR Endpoint and Statistical Analysis

Next, we applied the cIAI-specific DOOR endpoint to our existing database. We assigned each participant a DOOR between 0 and 8 using our newly derived endpoint (Tables 1 and 2) to count the number of component events experienced by a study participant. A DOOR of 0 represented the most desirable outcome and included participants who were alive and did not experience any undesirable component events. DOORs of 1 through 7 represented less desirable outcomes and included participants who were alive but experienced from 1 to 7 component events. A DOOR of 8 represented the least desirable outcome of death.

Table 2.

Complicated Intra-abdominal Infection–Specific Desirability of Outcome Ranking Endpoint

DOOR Ranka	Alive?	Count of Eventsb
0 (most desirable)	Yes	0 of 7
1	Yes	1 of 7
2	Yes	2 of 7
3	Yes	3 of 7
4	Yes	4 of 7
5	Yes	5 of 7
6	Yes	6 of 7
7	Yes	7 of 7
8 (least desirable)	No (death)	Any

Abbreviation: DOOR, desirability of outcome ranking.

Each participant was assigned a DOOR between 0 and 8 by counting the number of events a participant experienced within each component category (see Table 1). Study participants classified as having the outcomes of “indeterminate” or “clinical failure” were counted as having 1 event in the “absence of clinical response” category. Up to 2 events were counted in the categories of infectious complications, surgical/percutaneous procedures, and serious adverse events. These events comprised the final DOOR.

Absence of clinical response: 0 or 1 event; infectious complications: 0, 1, or 2 events; surgical percutaneous interventions: 0, 1, or 2 events; serious adverse events: 0, 1, or 2 events.

Using R version 4.0.5 software, we conducted a DOOR probability analysis in which we estimated the probability that a participant assigned to the study treatment arm would have a more favorable DOOR than a participant assigned to the comparator arm (estimated using Wilcoxon-Mann-Whitney U statistic adjusted for tie, divided by the product of group sample sizes) and formed a corresponding 2-sided 95% confidence interval and P value for the DOOR probability as described [6].

As DOOR is a composite outcome, we assessed the treatment effect on each component by repeating the probability analysis for absence of clinical response, ICs, additional procedures, SAEs, and death using only the DOORs generated for each respective component. We also repeated the DOOR probability analysis for subgroups defined by age, sex, type of cIAI, baseline creatinine clearance, and presence or absence of diabetes mellitus. These subgroups were selected due to their clinical relevance and because these data were readily available in all 9 trials.

RESULTS

Nine clinical trials were separately analyzed and included a total of 5022 study participants from the micro-ITT populations of each trial.

Application of DOOR Prototype and Subsequent Derivation of cIAI-Specific DOOR Endpoint

When the a priori DOOR prototype was applied to the clinical trials data, we found that its component definitions did not fully encompass the array of events that participants experienced, especially the impact of additional surgical procedures. After careful consideration, we modified the component definitions as follows:

1. Absence of clinical response: Across the trials, there were a variety of reasons for an “indeterminate” classification (Supplementary Figure 1). Due to this ambiguity and given prior precedence in considering both indeterminates and clinical failures as representing participants without clinical cure [7–9], we included both indeterminates and clinical failures in the “absence of clinical response” category.

2. Infectious complications: We found that infectious AEs were uncommon across the trials with the most frequent events of “wound infection” or “fever” occurring in 4.3% and 3.5% of aggregated study participants, respectively (Supplementary Figure 2). However, because we noted that other relevant infectious AEs (such as peritonitis or cholangitis) were found in MedDRA SOCs other than “Infections and Infestations,” we expanded the original IC component definition to include all newly identified infections in any MedDRA SOC diagnosed after randomization. Of note, an IC that was also counted as an SAE received 2 points in the cIAI-specific DOOR.

3. Surgical/percutaneous procedures: In the DOOR prototype, unplanned surgical procedures for progression/complication of the original infection were counted as ICs. However, by grouping procedures and ICs together, critical elements of a participant's overall experience would potentially be missed. Indeed, 17% of participants underwent additional surgical procedures, more than those who experienced any SAEs (9.9%) or ICs (14.6%) (Figure 1). Thus, we defined and added a separate component to the cIAI-specific endpoint to include procedures relevant to cIAI (Table 1) performed after the participant's first surgical intervention for cIAI. Given that initial cIAI interventions occurred on study day −1 or study day 1, this component included procedures that occurred on and after study day 2.

4. Accounting for multiple events: A number of participants experienced 1 or 2 ICs, SAEs, or additional procedures (Figure 1), but few experienced 3 or more events per category. Therefore, we modified the DOOR prototype ranking system [3, 4] (Supplementary Table 2) to count up to 2 ICs, SAEs, or additional procedures experienced by participants (Table 2). When stratified by clinical response, we noted that participants without clinical cure were more likely to have multiple ICs (Figure 2A), SAEs (Figure 2B), and procedures (Figure 2C).

Figure 2.

Participants with worse clinical outcomes were more likely to have more infectious complications (A), serious adverse events (B), and surgical/percutaneous procedures (C).

Application of cIAI-Specific DOOR Endpoint and Statistical Analysis

Each trial was analyzed using the cIAI-specific DOOR endpoint (Table 1). We compared the DOOR distribution in study treatment and comparator arms (Figure 3) and found they were similar within each trial. However, the DOOR distributions varied among trials, particularly in the percentage of participants with a DOOR of zero, which ranged from 47.6% in the study treatment arm for trial G to 81.0% in the study treatment arm for trial E.

Figure 3.

Desirability of outcome ranking distribution by treatment arm for 9 randomized control trials for complicated intra-abdominal infection (trials A–I). On the x-axis, uppercase letters represent the study treatment arm and lowercase letters represent the comparator arm.

We then formed DOOR probability estimates for each trial, which conveyed the likelihood that a participant randomly assigned to the study treatment would have a more desirable outcome than a participant assigned to the comparator. The DOOR probability estimates for the 9 trials ranged from 47.4% to 50.3% (Table 3) with no statistically significant differences between treatment and comparator arms (P > .05 for each trial). Results were generally insensitive to handling of indeterminates in sensitivity analyses (Supplementary Figure 3).

Table 3.

Desirability of Outcome Ranking Probabilities for 9 Randomized Controlled Trials for Complicated Intra-abdominal Infections

Trial	DOOR Probability, %a	(95% CI)b	P Value
A	50.3	(46.9–53.7)	.836
B	48.3	(44.5–52.2)	.386
C	47.7	(44.1–51.2)	.196
D	48.3	(45.3–51.4)	.279
E	48.2	(44.5–51.9)	.337
F	49.7	(44.4–54.9)	.897
G	47.4	(43.2–51.7)	.234
H	50.0	(44.9–55.2)	.990
I	48.1	(44.0–52.1)	.349

Abbreviations: CI, confidence interval; DOOR, desirability of outcome ranking.

The DOOR probability represents the probability of more a desirable result with study treatment relative to the comparator. A DOOR probability of 50% indicates no treatment difference; <50% favors the comparator; >50% favors study treatment.

A CI containing 50% indicates no statistically significant difference in the distribution of DOOR between the study treatment and the comparator arms.

The component analysis is designed to dissect the composite DOOR and evaluates the impact of each component separately. The analyses of 2 trials are highlighted as examples (Figure 4). In trial A, participants in the study treatment arm had more desirable outcomes regarding the IC component but less desirable outcomes for absence of clinical response, procedures, SAEs, and death. These results were not statistically significant (Figure 4A). In trial B, the study treatment was nominally statistically inferior to the comparator regarding the occurrence of SAEs (Figure 4B and Supplementary Figure 4). All other component analyses were not statistically significant (Supplementary Figure 5).

Figure 4.

Forest plot of the desirability of outcome ranking (DOOR) probabilities for the DOOR endpoint and each DOOR component. Trial A has no significant differences between the treatment arms in the component analysis (A). The study treatment arm was shown to be nominally statistically inferior for serious adverse events in trial B (B). Abbreviations: CI, confidence interval; DOOR, desirability of outcome ranking; ICs, infectious complications; SAEs, serious adverse events.

In the subgroup probability analyses, study treatment and comparator arms were generally similar, except for 1 instance where a study treatment was nominally statistically inferior to the comparator in participants with creatinine clearance <60 mL/minute (Supplementary Figure 6).

DISCUSSION

By combining clinical efficacy and safety into a novel endpoint, DOOR represents an innovation in clinical trial design that may better reflect associations among component outcomes and better evaluate the cumulative impact of multiple clinical events, thus being more reflective of an individual participant's overall outcome. We derived a comprehensive, cIAI-specific DOOR endpoint uniquely informed by our review of data from >5000 participants enrolled in 9 clinical trials for cIAI.

In our initial application of the DOOR prototype, we identified important clinical events, such as additional ICs and surgical procedures, that would likely significantly impact patients but were not captured by the DOOR prototype. In the cIAI-specific DOOR endpoint, the ICs component was expanded to include any new infection acquired by the participant during the trial, as patients who develop intercurrent nosocomial infections while being treated for intra-abdominal infections are known to have poorer outcomes including higher mortality rates, lengthier hospital stays, and longer durations of antibacterial therapy [10]. The targeted surgical/percutaneous procedures component was added to capture related surgical complications after the initial cIAI intervention. Major surgical complications (especially those requiring additional interventions) have been reported to impact mortality rates, length of hospital stay, and level of care required at discharge [11] as well as a patient's psychological well-being and quality of life (QoL) [12, 13].

By counting multiple ICs, procedures, and SAEs experienced by individual participants, our work differs from previously published DOOR endpoints that included only the presence or absence of these events [3, 4]. We capped the number of events at 2 per category, based on the observation that few participants experienced 3 or more ICs (1.1%), procedures (3.2%), or SAEs (1.1%) (Figure 1). However, with high clinical cure rates of >70% in all 9 trials, the 8-level ranking system offers a nuanced insight into differences in individual participant experience regardless of the success of the study treatment. For example, patients A and B may both have been classified as experiencing clinical cure, but patient A had no adverse events while patient B developed a hospital-acquired pneumonia and superficial postoperative wound infection. Presumably, patient B had a less desirable clinical course overall, which would be reflected as a DOOR of 2 compared to patient A's DOOR of 0.

Importantly, our study demonstrated that participants who experienced clinical failure were also those who had more intercurrent events such as ICs, procedures, and SAEs (Figure 2). This key observation underscores the potential interrelationship of these components and their contribution to a participant's outcome of clinical failure, as well as differences in participant experiences within this category. For example, a participant with an outcome of clinical failure due to the initiation of nontrial antibacterial drug therapy may be unaware of this change, while a participant who failed due to the need for an unplanned surgical procedure for complication or recurrence of cIAI is clearly impacted significantly. Though QoL data related to cIAI were lacking in these trials, previous evidence shows that surgical complications and additional procedures negatively impact a patient's overall well-being [11–13].

The DOOR probability analysis for each of the 9 trials was not statistically significant, and in general, reflected the results obtained through traditional safety and efficacy analyses. While DOOR is not meant to mirror traditional evaluations of efficacy and safety per se, it is not surprising that results of the probability analysis between treatment arms were similar for trials in which the efficacy of the study drug was noninferior to comparator and that traditional safety analyses did not find major imbalances between the arms. In our estimation, this finding does not necessarily imply a lack of sensitivity of the DOOR endpoint but may rather reflect the relatively controlled populations with fewer comorbidities and risk factors for severe outcomes who were enrolled in these trials [1]. While clinical cure rates in the 9 trials were generally high, the observed variation in DOOR distribution among trials, particularly in numbers of participants with a DOOR of zero (clinical cures with no component events), may be partly ascribed to differences in data collection among trials. Notably, variation in DOOR distribution could also reflect differences among patient populations, geographic regions, and local surgical practice, and bears further investigation.

Component analyses are important given the composite nature of DOOR and may allow for a deeper understanding of the treatment effects. The forest plot provides a visual risk-benefit assessment of a study treatment versus comparator by displaying probability estimates for DOOR and its components. While comparing occurrence of these components in each treatment arm is a key part of traditional safety analyses, DOOR calculates the probability of these events occurring in one study arm compared to another. Subgroup analyses permitted additional benefit-risk assessments, comparing DOOR probabilities within subgroups defined by baseline patient comorbidities, demographics, disease location, or microbiological factors deemed relevant to the infectious disease under study. Overall, no significant differences were noted in subgroup analyses for the 9 trials.

Our study highlights the derivation of a data-driven, disease-specific DOOR endpoint, but standardization to support comparability is needed. Furthermore, our work leads to interesting considerations of the potential applicability of DOOR in various settings. Interpretation of a trial in which, for instance, the component analysis shows study treatment to be safer than the comparator, but inferior to the comparator for efficacy, needs to be considered. From a clinical perspective, however, knowing that one drug is likely to be safer than another while having an equivalent probability for efficacy may help inform medical decision-making for an individual patient, considering their comorbidities, disease site, risk or presence of antimicrobial-resistant infection, and economic or QoL considerations.

Study limitations include heterogeneity in trial design, data collection, and definitions, which precluded aggregation of data and assessment of additional factors such as antimicrobial susceptibility. Differences in reporting practices for AEs and ICs may have influenced the outcome of the multiple event analysis. Furthermore, given the exploratory nature of these evaluations, the component and subgroup analyses were not adjusted for multiple comparisons. Notably, the retrospective approach also excluded the patient voice and necessitated assumptions of what events matter most to patients. However, in the absence of QoL measurements, using data from >5000 clinical trial participants to inform our endpoint derivation was the most objective and comprehensive approach available. Some of these limitations could be addressed in future trials by utilizing a well-defined DOOR and associated components and developing standardized and targeted data collection instruments to better support prospective evaluation of patient outcomes.

In conclusion, we derived, applied, and analyzed a novel cIAI-specific DOOR endpoint. Performance of this endpoint could be explored prospectively as a secondary endpoint in future cIAI trials. Future work will include exploration and potential inclusion of other factors that impact patients (such as length of hospital stay and illness-associated costs) in DOOR endpoints, incorporating the patient voice through patient reported outcomes and QoL data, and deriving other infectious disease–specific DOOR endpoints using our comprehensive data-driven approach. Notably, the derivation of disease-specific DOOR endpoints may further advance trial design and improve our understanding of patients’ overall experiences in clinical trials.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Disclaimer. This article reflects the views of the authors and should not be construed to represent the views or policies of the US Food and Drug Administration (FDA).

Financial support. This work was supported in part by an appointment to the Research Participation Program at the FDA, administered by the Oak Ridge Institute for Science and Education through an interagency agreement between the US Department of Energy and the FDA, reported by T. K. This work was also supported by the National Institute of Allergy and Infectious Diseases, National Institutes of Health (NIH) (award number UM1AI104681 reported by V. G. F., S. R. E., and H. F. C.). J .H. A., T. L. H., and S. B. D. were in part supported by the Antibacterial Resistance Leadership Group (NIAID UM1AI104681).