Desirability of outcome ranking and quality life measurement for antimicrobial research in organ transplantation

Abstract

Background:

Current clinical trials of new antibiotics facilitate bringing new drugs to market but often fail to provide useful information for clinical decision making.

Methods:

Literature review of desirability of outcome ranking (DOOR) in antibiotic clinical trials and description how DOOR can fit into design, administration, and assessment of clinical trials.

Results:

DOOR is an approach that addresses many of the shortcomings of current trials by incorporating efficacy and safety into a single outcome to analyze the patient experience in its entirety. Application of partial credit, tiebreaker strategies including response adjusted for duration of antibiotic risk (RADAR), and DOOR for Management of Antimicrobial Therapy (DOOR MAT) provides additional nuance and granularity. To address pitfalls of DOOR, investigators must develop the DOOR a priori, ideally with input from and incorporation of patient voices, and should perform component analysis to ensure that imbalances in key components are detected.

Conclusion:

Inclusion of DOOR in clinical trials will enrich our understanding of how new antibiotics might benefit patients who have had transplantation. Additional work to develop best practices for DOOR selection, analysis, and interpretation must continue, and incorporation of the patient perspective is essential.

Proposed tweet:

Desirability of outcome ranking (DOOR) incorporates the entire patient experience into a single clinical trial outcome, providing actionable information to transplant patients and clinicians alike.

The current state of clinical trials for new antibiotics

Antibiotics have saved millions of lives in the modern medical era and helped pave the way for modern surgical development, including solid organ transplantation.^1–2 However, in the setting of immunosuppression, anatomic issues that arise after surgery, donor-derived infections, and use of antibiotics, infections due to multidrug-resistant organisms (MDROs) are rising.^3–9 While prevention of these infections remains critical, we must also maintain access to antibacterial agents for when MDRO infections do occur. Important deficiencies persist in the pipeline¹⁰ of antibacterial agents directed against the most urgent bacterial threats^11,12 despite new incentives and funding to support development that provide reason for optimism.

As more antibiotics come to market, we must reexamine the information being generated by late-stage clinical trials. At its core, if a new intervention is successful, a clinical trial should illustrate an improvement in patient outcome—such as survival or how a patient feels or functions—according to treatment assignment.¹³ For situations where researchers expect a new treatment is better than the current standard (or placebo), a superiority design should be selected. For many new drugs, including antibacterial agents, researchers expect performance to match—rather than exceed—standard therapies.¹⁴ In this case, the researcher will need to use either an equivalence design or a non-inferiority design. Rather than demonstrating better outcomes, noninferiority studies intend to show that a new intervention is not markedly worse¹⁵ than the current standard of care.¹⁶

In infectious diseases, novel antibiotics may offer benefit because of their expanded spectrum against MDROs. At first glance, a superiority design might appear appropriate since the goal of a clinical trial for a new drug offering expanded coverage would be to confirm superiority compared to the current standard. However, for difficult-to-treat resistant infections, studying a new treatment compared to an insufficient current standard would be unethical and impractical.¹⁷ Instead, the common approach is to demonstrate non-inferiority for a new antibiotic against drug-susceptible versions of the infectious syndrome of interest. Critiques of non-inferiority trials include the following:¹⁶

• The null hypothesis is that the investigational agent is inferior to the standard. Is it ethical to enroll into a trial with this type of null hypothesis?

• Selection of the non-inferiority margin can be arbitrary¹⁸ and may not be acceptable, especially if the disease under study is very serious and any decrement in outcome would be undesirable for a patient. If each new agent incrementally leads to a non-inferior decrement in efficacy, there could be reduction in outcomes over time.¹⁹ Selection of an active comparator that is not the “best” approved treatment may also contribute to this phenomenon of biocreep.

• Design, conduct, and analysis flaws, such as underpowering, allowing for concomitant antibiotics, or large attrition rates, can lead to bias towards non-inferiority.¹⁵

• Especially for large trials, it is possible to have such a precise point estimate that the entire 95% confidence interval falls within the non-inferiority margin but still below zero. That is, there is the potential to be both non-inferior and inferior simultaneously.²⁰

• If a new agent is non-inferior to the old agent, why use the new agent? Often, there is an implied ancillary benefit,²¹ such as convenience, cost, safety, or spectrum, but researchers frequently neglect to incorporate rigorous statistical design and testing on these claims. If there is an implied claim of benefit, should there be an assessment of superiority?

• Sample sizes are often large and may not be practical.

Ascertaining non-inferiority importantly enables new antibiotics to be brought to market yet provides little clinically relevant information about the overall benefits of a new agent for the population it will be used to treat. In an analysis²² of new Food and Drug Administration (FDA) approvals for antibiotics between 2010–2015, Deak and colleagues demonstrate that all 6 broad-spectrum agents approved for common bacterial syndromes (pneumonia, skin and skin structure infection, urinary tract infection, intra-abdominal infection) utilized non-inferiority designs, including one agent where the non-inferiority margin had not been prespecified and was allowed a broad margin. None of these registrational trials specifically focused on efficacy versus MDROs. Furthermore, most of the main endpoints included clinician assessment of the composite of available subjective and objective information available rather than focusing on mortality or patient-reported outcomes. While designing superiority trials for drug-resistant bacteria poses a significant clinical and statistical challenges,¹⁷ improvements in the design and reporting of non-inferiority trials will facilitate more informative clinical trials even for studies of drug-susceptible organisms.

Whether there is a superiority or a noninferiority design, current outcome assessments have significant shortcomings,²³ including:

• Trials often combine outcomes with different importance to patients without assigning relative weights. For example, the FDA guidance²⁴ for evaluation of new drugs for complicated urinary tract infection suggests an endpoint evaluating both clinical (resolution of symptoms) and microbiological (bacteria in the urine decreased to < 10³ CFU/mL on follow-up culture) response. Imagine two patients (patients A and B) enrolled in the trial with pyelonephritis. Patient A has complete resolution of symptoms, but the urine culture at test of cure has 10,000 CFU/mL of E. coli on culture. Patient B clinically deteriorates and ends the study in the intensive care unit with multisystem organ failure. For this study, both patients would be classified as “failures,” though clearly the first patient had a better outcome than the second one. Should these patients be considered differently? Doesn’t it matter if one drug leads to more failures of the type experienced by patient A compared to those experienced by patient B?

• Current trials fail to integrate the entire patient experience²³ and instead focus on analyzing specific outcomes separately. By segregating efficacy and safety outcomes, trials limit interpretation of the interplay between these two and inclusion of quality of life. With any treatment, some patients may experience benefit and some harm. When these two outcomes are reported separately, we do not gain understanding of the degree to which benefits and harms may be related and experienced within each patient. More nuanced understanding of results can be gained from understanding the make-up of the groups experiencing efficacy without toxicity, efficacy with toxicity, neither efficacy nor toxicity, and no efficacy with toxicity. Evans and Follmann illustrate this concept in their paper on “Using outcomes to analyze patients rather than patients to analyze outcomes: A step toward pragmatism in benefit:risk evaluation.”²³ As part of their example, they conceive a fictional trial to illustrate this point. Imagine 200 patients comparing two agents—A and B—reports that treatment A leads to 50% efficacy with 30% safety events while treatment B leads to 50% efficacy with 50% safety event. Ostensibly, it might seem that treatment A should be preferred. However, it turns out that treatment A led to the following: 35 experiencing efficacy without toxicity, 15 efficacy with toxicity, 35 with neither efficacy nor toxicity, and 15 with no efficacy with toxicity. Treatment B led to the following: 50 experiencing efficacy without toxicity, 0 efficacy with toxicity, 0 with neither efficacy nor toxicity, and 50 with no efficacy with toxicity. When viewed from this angle, the comparison confers very different information, and choice A no longer clearly trumps choice B.

• Competing risks must be accounted for carefully. For instance, if the primary outcome of interest is length of stay in the hospital, this may look falsely good for an intervention associated with more death and hence shorter stays.

• Traditional endpoints do not account for the cumulative nature of beneficial and harmful events.

What is desirability of outcome ranking (DOOR)?

DOOR is an approach to design, administration, and assessment of clinical trials that addresses many of the drawbacks highlighted above. While traditional designs separate efficacy and safety on the individual level and then combine these at the population level, DOOR^23,25 integrates benefits and harms within patients as an ordered composite outcome and then aggregates the information for comparison. Unlike other composite endpoints where each component is treated with equal importance, DOOR facilitates hierarchical ordering. Here is how it works:^23,25

Step 1:

Creation of an ordinal composite outcome incorporating benefits and harms. Patients in different DOOR ranks experience importantly different outcomes while those within ranks experience outcomes that are not importantly different.²⁵ Mancini and Schulzer²⁶ conceptualize this approach well in their description of “unqualified success” and “unmitigated failure,” illustrated in Table 1. The exact features of what is considered a success and what makes a success qualified or a failure unmitigated will differ based on the condition under study. In addition, there may be different layers of success or levels of toxicity. Qualified success and unqualified failure may also be switched or combined.

Table 1.

Conceptual framework of DOOR

Category	Description
Unqualified success	Treatment success without side effects
Qualified success	Treatment success with side effects
Unqualified failure	Survives, treatment failure without side effects
Unmitigated failure	Survives, treatment failure with side effects
Death	Dies

Step 2.

All trial participants receive a DOOR assignment based on the outcomes in a trial. When comparing patients, the one with a better outcome has a better rank. Though not required, when two patients have the same rank, there is the option for a tiebreaker, such as a measure of health-related quality of life or duration of antibiotic therapy received, an approach called Response Adjusted for Duration of Antibiotic Risk, or RADAR.²⁵ In the RADAR approach, patients within the same DOOR rank—e.g. with similar clinical outcomes—will be compared based on the actual duration of antibiotic use in days such that patients with shorter durations will be prioritized over those receiving longer durations. Duration of therapy is not incorporated into comparisons between patients in different DOOR ranks since these patients have experienced different clinical outcomes. In addition to antibiotic duration, other types of antibiotic exposure could be used to break ties, including spectrum or use of a certain class of antibiotic.

Step 3.

Compare the DOOR distributions between interventions to estimate the probability that a randomly selected patient will have a better DOOR if assigned to a new treatment versus the comparator treatment. A DOOR probability of 50% indicates that the outcomes are comparable while a confidence interval that excludes and exceeds 50% suggests superiority. One that excludes and falls below 50% indicates inferiority.

Study teams should calculate sample sizes based on a superiority test with the null hypothesis of no difference in DOOR and an alternative hypothesis that the probability of a randomly selected patient in the treatment group will have a better DOOR than one in the control group will exceed 50%.²⁵ Standard calculations using the Wilcoxon-Mann-Whitney test can yield numerical estimates for trials, and simulations add information.

The DOOR probability-based analysis may be used in trials as primary or secondary endpoints. DOOR also can provide useful information for personalizing approaches to individual patients with different preferences and values. Best practices for presentation and analysis of DOOR includes reporting and analyzing the components of the DOOR separately as key secondary endpoints. Because each DOOR rank may not carry equivalent weight or importance to patients, Evans and Follman developed an extension to DOOR analysis whereby different weights can be assigned to each rank.²³ Termed partial credit, similar to how an academic test might be graded with different points assigned to different answers, this strategy allows for analysis of DOOR based on different patient priorities. For example, a patient for whom quality of life weighs most important might assign a lower point value to the qualified success outcome than one for whom survival at any cost takes priority (Table 2).

Table 2.

Partial credit approach

Category	Description	Survival at all costs	Quality of life as a focas
Unqualified success	Treatment success without side effects	100	100
Qualified success	Treatment success with side effects	100	40
Unqualified failure	Survives, treatment failure without side effects	90	30
Unmitigated failure	Survives, treatment failure with side effects	80	20
Death	Dies	0	0

Below, I review some recent and ongoing studies that have prospectively incorporated a DOOR approach and several others that have retrospectively applied DOOR.

Approaches to build and apply DOOR

While many clinical trials incorporating DOOR to date have done so in a post-hoc manner, the ideal approach involves prospective development in the planning phase of trials. The NIH-funded Antibacterial Resistance Leadership Group (ARLG)²⁷ has pioneered the a priori development of DOOR endpoints for clinical trials with an initial focus on Staphylococcus aureus bloodstream infection.²⁸ To develop a DOOR endpoint for use in future trials, the group developed a survey of clinician researchers using a series of case vignettes to understand what features of the patient experience should be included. The survey resulted in a 6-level DOOR incorporating treatment failure, infectious complications, ongoing symptoms, grade 4 adverse events, and death. The group applied the resulting DOOR retrospectively to the Combination Antibiotics for Methicillin-Resistant Staphylococcus aureus (CAMERA-1) trial, which studied the addition of flucloxacillin versus placebo on the backbone of vancomycin therapy.^28,29 A modified version of this DOOR will be the primary endpoint for the Dalbavancin as an Option for Treatment of Staphylococcus aureus bacteremia (DOTS) trial,³⁰ which is currently recruiting. Participants in DOTS with complicated S. aureus bacteremia or right-sided endocarditis who have cleared bacteremia with initial therapy will be randomized to ongoing standard intravenous therapy versus two doses of dalbavancin one week apart.

Several other clinical trials have incorporated or plan to incorporate DOOR prospectively:

• SCOUT-CAP:³¹ This placebo-controlled, double-blinded RCT compared short-course (5 days) antibiotic treatment plus 5 days of placebo with long-course (10 days) antibiotic treatment for pediatric pneumonia treated in ambulatory patients. The primary outcome included an 8-level DOOR incorporating clinical response, symptom resolution, and maximal adverse events with RADAR as a tiebreaker. Each participant received a rank on the 8-level DOOR. Within each level, the participants who received fewer antibiotic days were assigned higher ranks. The null hypothesis of the study was that the probability of a more desirable DOOR-RADAR for the short-course strategy would be 50% compared to the long-course strategy. The probability of a more desirable rank for a patient assigned to the short-course arm versus the long-course arm was calculated at 69% (95% CI, 63%–75%), indicating superiority of the short-course approach overall. Without the RADAR analysis, the clinical response between the arms was comparable with DOOR probability of 48% (95% CI, 42%–53%). From this, one can conclude that the short-course approach was equally effective but allowed for less antibiotic exposure. Supporting the use of RADAR as an important tiebreaker, the study also demonstrated significant decreases in number of antibiotic resistance genes per prokaryotic cell in those receiving the short-course strategy as compared to the longer course. Resistance genes could be used as a tie-breaker of antibiotic use in future studies.

• DigiSep-Trial:³² This study, currently recruiting,³³ aims to examine whether a strategy of diagnosing infections in patients with sepsis using next-generation sequencing will lead to better clinical outcomes than one using conventional microbiology diagnostics alone. The primary endpoint is a DOOR with RADAR as a tiebreaker. The 5-level DOOR component incorporates mortality, length of stay in intensive care, and acute kidney injury requiring renal replacement therapy.

• SHORTEN2:³⁴ This study, which is not yet recruiting, will randomly assign patients with Pseudomonas aeruginosa bacteremia to 7 versus 14 days of antibiotic therapy. The primary outcome will be a DOOR with RADAR tiebreaker. The 5-level DOOR incorporates clinical cure, recurrence, serious adverse event, and death.

Many others have applied DOOR analysis retrospectively to previously performed clinical trials.^35–38 These analyses must work within the framework of data collected as part of the original trial and may be limited, especially in ability to measure patient-reported outcomes. Other groups have applied DOOR analysis to observational studies.³⁹

DOOR MAT:

Trials focusing on MDR organisms pose many challenges.¹⁷ For non-registrational trials, one approach is to compare management strategies that take into account antibiotic resistance, clinical syndrome, and desire to avoid antibiotic-related harms. Evans and colleagues⁴⁰ have extended the DOOR concept for this type of study to account for appropriateness of antibiotic choice in the context of drug resistance with an approach called Desirability of Outcome Ranking approach for the Management of Antimicrobial Therapy (DOOR MAT). This approach ranks treatment based on two basic rules: (1) antibiotics active against the isolated bacteria are more desirable than those that do not have activity and (2) narrower antibiotics are preferred over broader-spectrum antibiotics. Using this reasoning, the stepwise approach to developing a DOOR MAT follows:⁴¹

1. For the organism of interest, rank antibiotic spectrum levels from narrow to broad. This works best when comparing within a class, such as β-lactam antibiotics, where it can be reasonably expected that if an organism if resistant to a broader-spectrum category agent, it would also be resistant to the narrower-spectrum agents as well. The categories can be named by spectrum, for instance, narrowest, medium narrow, medium, medium broad, and broadest.

2. Determine the possible phenotypic susceptibility patterns based on the categories. For instance, a pan-susceptible organism would be S-S-S-S-S. One only susceptible to the broadest category would be classified as R-R-R-R-S. A pan-resistant organism would be R-RR-R-R.

3. Generate a grid combining the antibiotic spectrum with the phenotypic resistance such that for each resistance pattern, selection of antibiotics from the spectrum category would be assigned a category as follows, organized from most to least desirable such as: most suitable, slight overtreatment, moderate overtreatment, heavy overtreatment, severe overtreatment, and undertreatment. Table 3 shows a simplified version of this categorization.

4. Assign scores to the categories from the prior step with the most desirable category receiving 100 and the least 0 and all others something in between.

5. Employ, analyze, and interpret the DOOR MAT score.

Table 3.

DOOR MAT example. The ranked order of treatment is: 1) Most suitable (most desirable); 2) moderate overtreatment; 3) severe overtreatment; 4) undertreatment (least desirable)

	S-S-S	R-S-S	R-R-S
Narrow	Most suitable (1)	Undertreatment (4)	Undertreatment (4)
Intermediate	Moderate overtreatment (2)	Most suitable (1)	Undertreatment (4)
Broad	Severe overtreatment (3)	Moderate overtreatment (2)	Most suitable (1)

Examples of the DOOR MAT approach include an examination⁴¹ of treatment of Escherichia coli and Klebsiella pneumoniae bloodstream infections at the Veteran’s Health Administration and a comparison⁴² of potential effect of different rapid diagnostic platforms on antimicrobial prescribing. As a metric of antibiotic prescribing appropriateness, this analysis approach could be complementary to antimicrobial stewardship programs wanting to measure quality of prescriptions in addition to the typical metrics that mainly address quantity of prescriptions (e.g. days of therapy). This method does rely on having positive cultures so is more appropriate for treatment situations where having positive cultures is expected, such as bacteremia or urinary tract infection, rather than scenarios where positive cultures may or may not occur, such as pneumonia or sepsis.

What are criticisms of the DOOR approach?

DOOR is not a panacea for all challenges related to antibacterial endpoint evaluation. Critics^43–45 have identified several areas of concern, including:

• Selection of DOOR ranks (definition and number) may be subjective, poorly reflective of the disease state, and lack patient perspective,^43,46 especially if applied in a post-hoc manner. Moreover, manipulation of the number and content of the DOOR ranks can affect the results and interpretation, further emphasizing the importance of a priori selection.⁴⁵

• As with all composite endpoints, important imbalances in the distributions in individual DOOR ranks may be missed unless there is analysis of the components.^44,47 Furthermore, if sample size selection is based solely on overall DOOR, there may not be power to detect important imbalances, such as a difference in mortality.

• Interpretation⁴⁵ of a new effect measure may complicate interpretation of clinical trial data rather than simplify it.

• The addition of RADAR incorporates a process measure of antibiotic use that may not directly improve outcomes for the individual patient^43,45 and may mask small but important differences in rare clinical outcomes, such as death.

• As with any approach to observational studies, application of DOOR to observational studies can be prone to biases and imbalances that might affect interpretation.^{46, 46–49}

To avoid these pitfalls, investigators must pre-specify and carefully design the DOOR prospectively. Power calculations should account for detection of critical components, and component analysis should be incorporated into the analysis plan. RADAR may be a complementary component, but the DOOR analysis based on clinical outcomes should be included also. If possible, direct measures of harms associated with antibiotic exposure, such as Clostridioides difficile or antibiotic-associated renal failure, should be measured rather than duration of therapy.

How can DOOR better incorporate the patient voice?

One important gap to date in the development and application of DOOR to clinical trials has been the patient voice. Patient perspectives should be incorporated into the process at multiple stages, including:

• Development and incorporation of measures of health-related quality of life (HR-QOL) into DOOR measures. Because the DOOR approach can incorporate multiple aspects of patient experience, measurement of HR-QOL related to infectious syndromes will be a critical aspect. Rather than rely on clinician assessment of clinical success or cure, which may vary^50,51 significantly from patient assessment and may be a poorer predictor of outcomes,^52,53 patient reported outcome could be added or substituted as a measure of success and of ongoing functional impact of an intervention.

• Development of DOOR endpoints: In addition to incorporating patient-reported outcomes into DOOR measures, patients should also have a voice in what measures should be incorporated into the DOOR.

• Partial-credit score assignment: Patient perspectives should be considered when assigning scores as well as when interpreting results for clinical decision making.

Infectious diseases lags other fields in developing and validating patient-reported outcome measures, particularly for bacterial syndromes commonly caused by antibiotic-resistant organisms.⁵⁴ In recent years, progress^55–57 towards measures for pneumonia, skin and skin structure infection, and bloodstream infection have been encouraging. More work will be required to validate these measures and incorporate them prospectively into clinical trials.

Conclusion and future steps

Extending this work to infections in transplant patients will enrich our understanding of how new antibiotics might benefit patients who have had transplantation. Unique features, such as organ rejection and donor-derived infection, could be incorporated into DOORs for these populations to allow a tailored analysis for these patients.

Incorporation of DOOR will facilitate more informative antibiotic clinical trials for all stakeholders. Additional work to develop best practices for DOOR selection, analysis, and interpretation must continue, and incorporation of the patient perspective will be a vital part. Ongoing collaborations to develop consensus in approach between regulators, clinical researchers, representatives from industry, clinicians, and patients will be critical.

Abbreviations:

AMR

Antimicrobial resistance

ARLG

Antibacterial Resistance Leadership Group

DOOR

Desirability of outcome ranking

DOOR MAT

Desirability of Outcome Ranking approach for the Management of Antimicrobial Therapy

CFU

Colony-forming unit

FDA

Food and Drug Administration

HR-QOL

Health-related quality of life

MDR

Multidrug-resistant

MDRO

Multidrug-resistant organism

mL

Milliliter

NIH

National Institutes of Health

RADAR

Response adjusted for duration of antibiotic risk