Desirability of Outcome Ranking for the Management of Antimicrobial Therapy (DOOR MAT): A Framework for Assessing Antibiotic Selection Strategies in the Presence of Drug Resistance

Abstract

The complexities of antibiotic resistance mean that successful stewardship must consider both the effectiveness of a given antibiotic and the spectrum of that therapy to minimize imposing further selective pressure. To meet this challenge, we propose the Desirability of Outcome Ranking approach for the Management of Antimicrobial Therapy (DOOR MAT), a flexible quantitative framework that evaluates the desirability of antibiotic selection. Herein, we describe the steps required to implement DOOR MAT and present examples to illustrate how the desirability of treatment selection can be evaluated using resistance information. While treatments and the scoring of treatment selections must be adapted to specific clinical settings, the principle of DOOR MAT remains constant: The most desirable antibiotic choice effectively treats the patient while exerting minimal pressure on future resistance.

To meet the challenges of antibiotic treatment and stewardship evaluation in the presence of antibiotic resistance, the Desirability of Outcome Ranking for the Management of Antimicrobial Therapy (DOOR MAT) offers a flexible quantitative framework that evaluates the desirability of antibiotic selection.

The benefit of early and effective antibiotic therapy in the treatment of acute bacterial infections is established in multiple settings [1–3]. In instances where antibiotics are warranted, inappropriate therapy is conventionally understood as effective therapy having a spectrum of antimicrobial activity that exceeds what is necessary to treat the infecting pathogen (ie, overtreatment). Inappropriate therapy also includes antimicrobials that are ineffective (ie, undertreatment); some multidrug-resistant pathogens require broad-spectrum agents as effective therapy. Inappropriate antimicrobial use contributes to the emergence of antimicrobial-resistant bacteria. Identifying and reducing such use is a goal of antimicrobial stewardship [4–7]. Appropriateness indicates that antibiotic agents with a narrow spectrum of activity are preferable to broad-spectrum agents, but not at the expense of patient outcomes [8, 9]. Antibiotic treatments that incorporate stewardship aims have been shown to be beneficial [10, 11]. When considered in the context of both patient outcomes and the potential for selective pressure to drive emergence of future antibiotic resistance, the appropriateness of antibiotic agents can be categorized, with some agents being more desirable than others.

Both the spectrum of activity for individual antibiotics as well as antimicrobial resistance among bacterial isolates introduce further complexity into decisions about desirability. Metrics exist to group and compare antibiotics based on their spectrum of activity: or their expected activity given historic resistance prevalence [12, 13]. While an antibiotic’s spectrum of activity is fixed, the prevalence of antimicrobial resistance and the clinical implications are more fluid, changing over time, across regions, and depending on clinical circumstances. The varied combinations of spectrum and resistance pose a challenge in evaluating treatment selections. We propose the Desirability of Outcome Ranking approach for the Management of Antimicrobial Therapy (DOOR MAT), a quantitative structured framework that evaluates antimicrobial treatment selection based on desirability considered in the context of antibiotic resistance, clinical circumstances, and stewardship goals.

This strategy combines the quantitative element of scoring antibiotic agents based on their spectrum with the ranking element of DOOR to evaluate appropriateness of treatment using simple statistical analysis techniques. Consider the example of a serious infection caused by common enterobacteria, such as Klebsiella pneumoniae (Kp). A carbapenem could be used in the treatment of 3 types of strains: one susceptible to cefazolin, ceftriaxone, and piperacillin-tazobactam; one resistant to these agents but carbapenem susceptible; and one resistant to carbapenems. An agent-based spectrum index considers these uses of a carbapenem to be identical, failing to capture differences in expected patient outcomes between effective and ineffective therapies. DOOR allows these 3 to be ranked, but the degree of the preference—how many times to overtreat to avoid ineffective treatment or the relative clinical risk of overtreatment compared to that of optimal treatment—is not measured. DOOR MAT seeks to address these limitations by combining elements of both approaches.

OVERVIEW OF DOOR MAT

Working with an existing classification of agents by spectrum and a known resistance/activity profile, we use DOOR MAT to evaluate treatment selections based on their desirability. These steps align with Desirability of Outcome Ranking (DOOR) strategies previously described, then expands on them by assigning scores to the treatment selections to quantify the relative desirability of one treatment selection over another [14]. The most desirable and highest scored treatment selection is one that effectively treats the patient while exerting minimal pressure on future resistance. All other treatment options fall into one of 2 types of less desirable treatments: undertreatment, in which the treatment selection is not active against the isolate; or overtreatment, in which the treatment selection is broader than at least one available active agent. The scores assigned to these reflect the relative clinical and resistance risks of undertreatment and overtreatment.

Focusing on activity and spectrum for simplicity, we can classify and rank treatment selection based on 2 rules: (1) treatment selections containing an active agent are more desirable than treatment selections that do not contain an active agent; and (2) narrower antibiotics are more desirable than broad antibiotics.

DOOR MAT can be adapted to antimicrobial treatments in a range of scenarios. Differences between treatment selection strategies can be quantified, informing the analysis of clinical and stewardship activities.

DETAILED DESCRIPTION AND EXAMPLES

Below we describe the steps of implementing the DOOR MAT framework, walking through examples using the model of Kp causing bacteremia—an infection requiring effective treatment but where effective antibiotic therapy is conditioned by resistance. We have simulated a cohort that reflects the typical prevalence of resistance to extended-spectrum cephalosporins and carbapenems, roughly 12% and 3%, respectively, reported in nationwide studies published in recent years [15–18].

Step 1: Establish and Rank Antibiotic Spectrum Levels

Determine the antibiotic spectrum levels of interest in order to classify antibiotics according to their spectrum of antibiotic activity. Levels are ordinal (narrow to broad), and the number of levels may vary according to the antibiotics included in the study.

Example

For Kp bacteremia, we focus on the most commonly used agents, cephalosporins, β-lactam/β-lactamase inhibitor combinations, and carbapenems, grouping them into 4 ordinal categories. A fifth category includes broad-spectrum agents that are typically considered medications of “last resort” and used to treat isolates resistant to all β-lactams (Table 1).

Table 1.

β-Lactams, β-Lactam/β-Lactamase Inhibitors, and “Last Resort” Antibiotics of Interest, Classified According to Spectrum of Activity

Spectrum of Activity	Cephalosporins, β-Lactams, and β-Lactam/β-Lactamase Inhibitorsa
Narrow	Cefazolin, ampicillin, ampicillin-sulbactam
Intermediate I	Ceftriaxone, cefoxitin, cefotetan, cefotaxime, ceftaroline, ceftazidime, cefuroxime
Intermediate II	Piperacillin-tazobactam, cefepime
Broad	Imipenem, doripenem, meropenem, ertapenem
Last resort	Colistin, tigecycline

Bolded terms indicate the representative agent used to determine the resistance profile.

^aWe do not include newly available agents such as ceftazidime-avibactam, meropenem-vaborbactam, or imipenem-relebactam, focusing on antibiotics that were widely available over the timeframe when the reference Klebsiella pneumonia isolates were described.

Step 2: Determine Possible Resistance Profiles

Determine possible resistance profiles based on the antibiotic spectrum levels. In general, if there is resistance to an antibiotic, then there is usually resistance to antibiotics from the same class that have a narrower spectrum of activity. This allows profiles to be ordered from least to most resistant and captures nearly all isolates.

Then, for the treatment selections classified into spectrum levels in step 1, determine the antibiotic susceptibility results that indicate which treatment selections would be active or inactive. These susceptibility results can be observed directly for an antibiotic agent, inferred using a representative agent within a drug class, or derived from susceptibilities of multiple agents or a proxy.

Example

For each β-lactam spectrum level, we chose a representative agent (Table 1) that we used to determine the resistance profile: cefazolin for “narrow”; ceftriaxone for “intermediate I,” piperacillin-tazobactam for “intermediate II,” and imipenem for “broad.” Mechanistically, we do not expect Kp isolates to frequently violate the assumption that resistance to a given antibiotic implies resistance to narrower agents on the antibiotic spectrum. For this example, we assume that our last-resort agents are active against all isolates.

In our example, there are 5 possible resistance profiles capturing the susceptibility (S) or resistance (R) to increasing spectra of agents:

• S-S-S-S-S: Susceptible to all 4 representative β-lactams and last-resort agents.

• R-S-S-S-S: Resistant to narrow; susceptible to others.

• R-R-S-S-S: Resistant to narrow and intermediate I; susceptible to others.

• R-R-R-S-S: Resistant to narrow, intermediate I, and intermediate II; susceptible to broad.

• R-R-R-R-S: Resistant to all 4 representative β-lactams; susceptible to last-resort agents.

Step 3: Create a DOOR MAT Grid

Create a DOOR MAT grid by cross-classifying the antibiotic spectrum levels, as assigned in step 1, with the possible resistance profiles. The resistance profile of the infecting pathogen can be determined or inferred from the results of antimicrobial susceptibility testing. Antibiotic treatments are then classified into one of the cells of the DOOR MAT grid. Subsequently, the treatment selections are characterized and ranked based on their desirability, applying the 2 principles described above.

Example

We developed 2 DOOR MAT grids from our example (Figure 1). The first is color-coded according to active and inactive treatments for each of the resistance profiles developed in step 2. The second stratifies the treatments by their desirability or appropriateness. Undertreatment is the least desirable treatment based on expected clinical outcomes, the narrowest active treatment is the most desirable, and all other options representing overtreatment fall between these 2 extremes. Overtreatment is further stratified by the number of spectrum groups between the given treatment and the most desirable.

Figure 1.

A, Desirability of Outcome Ranking approach for the Management of Antimicrobial Therapy (DOOR MAT) grid classifying combinations of antibiotic treatment spectrum and resistance profile into active and inactive combinations with combinations in which the selected treatment (row) is active against isolates of a given resistance pattern (column) shown in light green and combinations in which the treatment is not active shown in dark red. B, Further classification of the active treatments from (A) stratifies the desirability of treatments. C, The most desirable or appropriate treatment selections are represented in bright green. Dark red cells (upper right of the DOOR MAT grid) represent undertreatment, which are the least desirable treatments based on expected clinical outcomes, and off-diagonal stripes among the active treatments group overtreatment combinations into ordered groups ranging from slight overtreatment (light green) to severe overtreatment (bright red). Abbreviations: R, resistant; S, susceptible.

Step 4: Define and Assign Scores

Determine a scoring system and apply it to the treatment gradations that emerge from the DOOR MAT grid created in step 3. The scores should reflect clinical risk assessments and the relative importance of antibiotic stewardship goals and will vary according to the clinical scenario to which DOOR MAT is applied.

We recommend constructing a DOOR MAT scale that ranges from 0 to 100. The most desirable treatment selections are assigned a score of 100, whereas the least desirable are assigned a score of 0. All other selections are given partial credit with scores assigned between 0 and 100 [9], translating scores and summaries into a familiar percentage scale.

Development of DOOR MAT scores falls beyond the scope of this introduction, but principles of average weighted accuracy analysis offer weighting schemes based on clinical importance [19]. Similarly, the Delphi method and judgment matrices offer approaches for developing rankings or scorings within teams of stakeholders that can be implemented and documented [20, 21].

Example

For 6 ranked categories, we might consider several scoring schemes. The simplest approach uses equal increments to address the 4 levels that fall between the extremes of least to most desirable treatments, with resulting scores of 0, 20, 40, 60, 80, and 100 (“Approach A” in Table 2). While easy to impose, this approach may not reflect clinical priorities. Approach B penalizes overtreatment to a greater degree than approach A, imposing a wider interval between ideal treatment and slight overtreatment (scores of 100 vs 60) and a smaller interval between severe overtreatment and inactive treatment (scores of 10 vs 0). Approach C considers inactive treatment to be far less desirable than all active treatments (scores of 0 vs 88–100) with small differences between overtreatment and ideal treatment.

Table 2.

DOOR MAT Groups in Descending Order of Desirability and Example Scores

Treatment Selection Group	Approach A	Approach B	Approach C
Ideal treatment (most desirable)	100	100	100
Slight overtreatment	80	60	97
Moderate overtreatment	60	50	94
Heavy overtreatment	40	30	91
Severe overtreatment	20	10	88
Inactive treatment (least desirable)	0	0	0

We present these 3 approaches to demonstrate ways in which scores for the same combination of treatments and resistance profile may shift depending on the priorities of the DOOR MAT user(s) and the intended application.

Step 5: Implement and Summarize

Apply the DOOR MAT scores to a clinically relevant situation. The situation may be an individual case, a cohort of patients, or the resistance patterns for a specific group of bacterial isolates such as might be used to develop an antibiogram. For an individual case, once the resistance profile and treatment selection are known, the score for that case be determined. For a cohort of patients, the frequencies of each combination of resistance profile and treatment selection can be tallied, and then scoring of the entire cohort can occur at the aggregate level. Similarly, based on an institution’s antibiogram, combinations of resistance profiles and treatment selections can be tallied to inform treatment protocols.

Examples

First we will consider a single case of Kp bacteremia caused by an R-S-S-S-S isolate and a protocol that recommends empiric treatment with piperacillin-tazobactam. For this case, the treatment selection is an intermediate II, which our ranked and grouped combinations classified as a slight overtreatment. Approaches A, B, and C would score this case at 80, 60, and 97 respectively.

Looking to our previous example of 1000 Kp isolates and using the same treatment protocol, we observe 7% optimal treatment (R-R-S-S-S isolates), 12% slight overtreatment (R-S-S-S-S isolates), and 77% moderate overtreatment (S-S-S-S-S isolates). While we will never observe heavy or severe overtreatment, 5% of isolates (R-R-R-S-S and R-R-R-R-S) will receive inactive treatment. For our full cohort, approach A (equally spaced intervals) yields an average of 62, approach B (penalizing overtreatment heavily) an average of 52, and approach C (penalizing inactive treatment heavily) an average of 90. These differences emphasize the careful consideration needed for the DOOR MAT scoring system. For a formal comparison, specifying scores a priori is essential to the integrity of the analysis. However, planned sensitivity analyses summarizing the effects of small scoring changes on results are recommended.

Step 6: Analyze and Interpret

DOOR MAT scores represent a continuous metric of the desirability of a treatment selection given the resistance profile of the infecting pathogen. DOOR MAT scores can be interpreted as measures of the desirability of treatments under a given set of definitions and scores.

Analysis methods for continuous variables could be employed to compare DOOR MAT scores and should be chosen to reflect the study design, the unit of analysis, the posed hypothesis, and relevant confounders. Rank-based methods could assess the desirability ranking, but they would notably fail to incorporate the scoring aspect of DOOR MAT.

Examples

Using DOOR MAT, we can compare the appropriateness of a given treatment protocol across cohorts with different resistance prevalence. We can compare our example of Kp isolates causing bacteremia with 3% carbapenem nonsusceptibility, where using piperacillin-tazobactam to treat infections caused by these isolates would result in a mean score of 62. In another cohort of the same size, simulated using a carbapenem resistance prevalence of 10% observed in Kp bacteremia from the New York/New Jersey area [22], the mean score would be 57. The use of piperacillin-tazobactam in the latter cohort would be ineffective in more than 10% of cases, and the associated scores of zero decrease the average appropriateness across the cohort. Since the appropriateness of a given treatment protocol across a cohort depends on the resistance prevalence, DOOR MAT can quantify and facilitate analysis of differences in appropriateness (Figure 2A).

Figure 2.

Prevalence of Desirability of Outcome Ranking approach for the Management of Antimicrobial Therapy (DOOR MAT) resistance profiles in cohorts simulated assuming 3% vs 10% carbapenem resistance, shaded according to treatment group assuming a treatment protocol of piperacillin-tazobactam where isolates of patterns S-S-S-S-S, R-S-S-S-S, and R-R-S-S-S receive effective treatment (green and yellow bars) based on piperacillin-tazobactam susceptibility, but isolates of patterns R-R-R-S-S and R-R-R-R-S receive ineffective therapy (dark red) based on piperacillin-tazobactam resistance, and the greater prevalence of carbapenem resistance in the right panel leads to fewer isolates effectively treated and lower average DOOR MAT scores (A). Two treatment protocols considered for the simulated cohort with 3% carbapenem resistance, the left panel assuming treatment with piperacillin-tazobactam and the right panel assuming therapy selected using rapid genetic tests. The rapid genetic tests detect susceptibility to the narrowest active agent in most S-S-S-S-S, R-R-S-S-S, and R-R-R-R-S isolates where optimal treatment is seen (bright green), but the tests fail to identify resistance in the R-S-S-S-S and R-R-R-S-S isolates and undertreatment (dark red) occurs frequently. Depending on the scores assigned to each treatment category, the rapid genetic test may or may not represent an improvement in appropriate treatments (B). Abbreviations: R, resistant; S, susceptible.

For a further example, we again consider Kp bacteremia and address the relative desirability of 2 empiric treatment protocols. The first uses piperacillin-tazobactam for all bloodstream infections where Kp is suspected. The second treatment protocol uses rapid genotypic diagnostic tests to determine the narrowest active agent. Reported results from a study of rapid genotypic diagnosis tests for Kp isolates [7] allow us to simulate such treatments for our population (Figure 2B). Using approach A that we described in step 4 above, the mean DOOR MAT of the treatments under the first protocol would be 62, compared to an appreciably higher score of 85 under the second protocol but a different scoring approach may not imply such improvement (Supplementary Appendix).

DOOR MAT can further be used to combine the concepts of these 2 examples, comparing the difference between the 2 different treatment protocols in cohorts of different resistance prevalence by first determining the average score within each resistance pattern, then using weighted averages according to the prevalence of the resistance patterns (Supplementary Appendix). Such an analysis could be used to identify populations where one treatment protocol might increase appropriate therapy over another treatment protocol, informing how best to allocate laboratory and stewardship resources and efforts.

DISCUSSION

Resistant pathogens pose many challenges for patient treatment and antimicrobial stewardship. Therefore, a more structured approach is indicated for measuring the appropriateness of antibiotic treatment selection in the presence of resistance.

DOOR MAT is designed to offer a flexible, quantitative framework for assessing treatment selection given the resistance profile of the pathogen. In our examples, we compared a given treatment protocol among Kp isolates with different resistance prevalence and compared 2 treatment protocols. Additional scenarios might include summarizing provider behavior temporally, comparing prescribing practices before and after a stewardship intervention, or quantifying the appropriateness of empiric vs definitive therapies.

The scores and antibiotic groupings presented in the examples above are not suggested “defaults” of DOOR MAT, but chosen simply to motivate concepts. Selecting scores requires quantifying clinical risk and stewardship goals. As we saw in our example, different scoring systems applied to the same set of treatment selections and resistance profiles can yield different results. The intervals between ideal, inactive, and overtreatment correspond to the degree of penalization associated with undertreatment and overtreatment. The selection of levels and scores must be based upon specific study settings.

While the ranking of treatment selections prioritizes active treatment and narrow spectrum, patient outcomes are not directly incorporated into DOOR MAT scores. Thus, these scores must be interpreted as measuring desirability of treatment, not of outcomes. While we expect the 2 to be strongly associated, this distinction in interpretation is crucial. Future work is needed to inform scoring strategies based on outcome data, assessing desirability from the point of view of the patient, the provider, the healthcare system, and society.

There are complexities that can challenge implementation of DOOR MAT. Additional dimensions of antibiotic selection have not been addressed here: toxicity, cost, availability, ease of administration, and drug–drug interactions. But where these contribute to appropriateness, they can be incorporated into DOOR MAT. DOOR MAT may fail to capture resistance patterns caused by unusual mechanisms such as, in this example, inhibitor-resistant β-lactamases [23]. A lack of consensus regarding the desirability of antibiotic treatments may hinder the ranking and scoring steps [24, 25]. New evidence may challenge paradigms informing DOOR MAT, as has occurred with the treatment of infections caused by Kp and Escherichia coli with extended-spectrum β-lactamasaes [26–28]. Last, synergistic combinations of antimicrobial agents may be effective and clinically appropriate despite nonsusceptibility of the comprising agents, and would be difficult to assess with DOOR MAT without adjustments not discussed here.

We acknowledge that no categorization of antibiotics will fully capture their mechanistic differences, spectrum of activity, or selective pressure. However, DOOR MAT’s flexibility allows for the addition of resistance profiles or specific drug combinations and the re-ranking or re-scoring of treatment options either in an adaptive manner described a priori or in a sensitivity analysis.

DOOR MAT offers a pragmatic tool that assumes the occasional need of broad-spectrum agents, rewards optimal treatment that cannot be de-escalated, and reflects the priorities of the clinical setting in which it is applied. Selecting scores in a meaningful way requires considering and weighing both long-term pooled societal risks and immediate patient benefits, with perhaps more data and immediacy granted to the latter. Subsequent research will focus on patient outcomes and resistance trends to better inform and validate DOOR MAT scores.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Disclaimer. The findings and conclusions in this document are those of the authors, who are responsible for its content, and do not necessarily represent the views of the Department of Veterans Affairs (VA), the National Institutes of Health (NIH), or the United States government.

Financial support. This work was supported by the National Institute of Allergy and Infectious Diseases (NIAID) of the NIH (grant numbers R01AI100560, R01AI063517, R21AI114508, and R01AI072219 to R. A. B.; UM1AI104681 to the Antibacterial Resistance Leadership Group supporting S. R. E. and Y. J.); funds and/or facilities provided by the Cleveland Department of Veterans Affairs; the Biomedical Laboratory Research and Development Service of the VA Office of Research and Development (grant number 1I01BX001974 to R. A. B.); and the Geriatric Research Education and Clinical Center Veterans Integrated Service Network 10 (to B. M. W., R. A. B., F. P., and R. L. P. J.).

Potential conflicts of interest. R. A. B. receives research grants from Merck, Allergan, Wockhardt, Achaogen, Shionogi, VenatoRx, VA Merit Review, NIAID, Entasis, and GlaxoSmithKline (GSK). F. P. and R. L. P. J. receive research funding from Pfizer, Merck, and Accelerate; R. L. P. J. has also participated in advisory boards for Pfizer and Roche. S. R. E. reports personal fees from Takeda/Millennium, Pfizer, Roche, Novartis, Achaogen, the Huntington Study Group, ACTTION, Genentech, Amgen, GSK, the American Statistical Association, the US Food and Drug Administration, Osaka University, the National Cerebral and Cardiovascular Center of Japan, the NIH, the Society for Clinical Trials, Statistical Communications in Infectious Diseases (DeGruyter), AstraZeneca, Teva, the Austrian Breast and Colorectal Cancer Study Group/Breast International Group, the Alliance Foundation Trials, Zeiss, Dexcom, the American Society for Microbiology, Taylor and Francis, Claret Medical, Vir, Arrevus, Five Prime, Shire, Alexicon, Gilead, Spark, the Clinical Trials Transformation Initiative, Nuvelution, Tracon, the Deming Conference, the Antimicrobial Resistance and Stewardship Conference, the World Antimicrobial Congress, WAVE, Advantagene, Braeburn, Cardinal Health, Lipocine, Microbiotix, and Stryker. R. A. V. reports grants from Gilead, outside the submitted work. All other authors report no potential conflicts of interest.

All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.