Abstract
Most clinical trials of treatment efficacy evaluate benefits and harms separately. Investigators generally rate the primary outcome of a trial with a binary outcome measure and consider harms separately as adverse events. This approach fails to recognize finer gradations of patient response, correlations between benefits and harms, and the overall effects on individual patients. For example, in status epilepticus trials, efficacy is often defined as the absence of clinically apparent seizures with recovery of consciousness. Such an efficacy outcome fails to recognize that some causes of status epilepticus, such as subarachnoid hemorrhage or stroke, may not be accompanied by return of consciousness, and the need to intubate a patient may be classified as treatment failure even if status was successfully terminated. The Desirability of Outcome Ranking (DOOR) method uses a different approach. The DOOR method involves comparing the experiences of trial participants in different treatment arms by the desirability of the overall patient outcome. Using status epilepticus treatment as an example, a patient who experiences successful termination of status epilepticus but with major side effects would have a less desirable outcome than a patient with treatment success and minor side effects, who in turn would have a less desirable outcome than a patient with treatment success but no side effects. This is a patient-centered approach because it considers treatment efficacy in the context of the costs borne by the patient, for example, toxicity in achieving efficacy. Thus, DOOR considers both the benefits and harms to individual patients in assessing the outcome of a clinical trial. In this article, we present the rationale for the use of DOOR, the issues involved in the development of and statistical analyses of an ordinal outcome, and an example of the potential application of the DOOR method to a clinical trial of convulsive status epilepticus.
Introduction
Most studies of treatment efficacy evaluate benefits and harms separately. Investigators generally rate the primary outcome of a trial with a binary, often composite, outcome measure and consider harms separately as adverse events. This approach fails to recognize finer gradations of patient response, correlations between benefits and harms, and the overall effects on individual patients. For example, in the Established Status Epilepticus Treatment Trial (ESETT),1 endotracheal intubation was considered a treatment failure because rapid EEG was unavailable to determine whether a patient was still having subclinical seizures. Many of these patients may have stopped seizures but required intubation because of respiratory depression caused by treatment medications combined with the effects of the status itself. Arguably, stopping seizures but requiring intubation is a better outcome than not stopping the seizures, and it would be important to recognize this distinction. In the absence of EEG recording, the ESETT's primary outcome required recovery of consciousness as evidence of the termination of seizures. However, many causes of convulsive SE, such as subarachnoid hemorrhage, large stroke, or lobar hemorrhage, lead to depressed consciousness, and such a patient is likely to remain unconscious despite the termination of SE. Failure to recognize these distinctions may distort the measurement of drug efficacy.
The Desirability of Outcome Ranking (DOOR) method2 was developed to address these issues. It was first applied in infectious disease studies,3-6 where gradations of effectiveness must be balanced by the adverse events of taking antibiotics. DOOR has also been applied to outcomes for cardiovascular disease and neurologic emergencies.7-9 We face conceptually similar challenges in the evaluation of the treatment of status epilepticus (SE), where there is a balance between the benefits of more aggressive treatment (i.e., termination of the seizures) and the adverse outcomes such as respiratory failure and hypotension. In this article, we present the conceptual framework for such a DOOR outcome measure, discuss issues related to analysis, and present a hypothetical comprehensive DOOR outcome measure for convulsive status epilepticus treatment trials.
Methods
Conceptual Framework
The DOOR methodology involves comparing the experiences of trial participants in different treatment arms by the desirability of the overall patient outcome rather than using a binary efficacy outcome. Using antibiotic treatment as an example, a patient who experiences treatment success but with acute kidney injury would have a less desirable outcome than a patient with treatment success and diarrhea, who in turn would have a less desirable outcome than a patient with treatment success but no side effects. This is a patient-centered approach because it considers treatment efficacy and the costs borne by the patient, for example, toxicity in achieving efficacy. Thus, DOOR considers both the benefits and harms to individual patients in assessing the outcome of a clinical trial. The prespecified DOOR measure can incorporate both short-term temporary effects and long-term effects such as quality of life.10
When using the DOOR methodology, an important first step is to define an ordinal DOOR outcome representing a global patient response considering benefits and harms. This may be straightforward for some diseases, such as acute emergencies with an immediate response to treatment. For clinical scenarios with complex tradeoffs among benefits and harms, such as balancing short-term and long-term benefits and harms, this is challenging and could involve expert panels, patients, and other stakeholders.11,12 Such scenarios could involve the measurement of quality-adjusted life years13 or health indices such as the Health Utilities Index.14
DOOR Analyses
There are 2 complementary approaches to DOOR analyses.
The first approach uses the concept of pairwise comparisons of the DOOR responses between treatments. Once global patient responses have been classified into DOOR, we compare all possible pairwise comparisons of the DOOR from patients in one treatment arm with the outcomes from patients in the other treatment arm. For example, there are N1*N2 possible pairwise comparisons in a study with N1 patients in one treatment arm and N2 patients in the other treatment arm. Pairwise comparisons will result in one patient having a better outcome, a worse outcome, or a tie result. The DOOR probability (i.e., the probability of a more desirable result [adjusted for tied desirability]) in one treatment relative to another treatment is then calculated using these pairwise comparisons.
 |
A DOOR probability of 50% implies that there is no between-treatment group difference in the desirability of the outcomes. Although different from statistics traditionally used in clinical trials (e.g., the difference in proportions or a hazard ratio), this metric has intuitive appeal for clinicians when selecting a therapy because it would be very useful to know the probability that a patient will have a more desirable overall outcome on one treatment relative to a therapeutic alternative. CIs for the DOOR probability can best estimate using published methods.15 Methods for incorporating stratification variables16 and censoring17 into the calculation are available.
DOOR is a composite outcome. A fundamental tenet to the analysis of a composite outcome is to thoroughly analyze its components. Thus, comprehensive DOOR analyses include the estimation of the DOOR probability for individual components and the overall ranking. The DOOR probability for each of these binary outcomes can be calculated using the same methods. An advantage of the DOOR probability is that it is an absolute metric that allows for informed interpretation of multiple outcomes simultaneously, in contrast to relative measures with different baseline rates. The resulting analyses will include CI estimates of a more desirable result with respect to DOOR and each of its components. Given the ordinal nature of DOOR, a plot of the cumulative DOOR probability and associated confidence bands, with sequential dichotomization of the DOOR, is also performed.
A concern with the pairwise comparison approach is that a more frequent advantage in an outcome of lesser importance could offset a small decrement in a very important component despite appropriate prioritization. For example, the step between the worst outcome and the next best outcome may be viewed as a larger step than steps between other categories. To address this concern, one can perform partial credit analyses.18 Partial credit analyses involve grading the levels of the DOOR on a scale from 0% to 100%. The most desirable outcome is given a score of 100%, and the worst outcome is given a score of 0%; intermediate outcomes are scored with partial credit. The specific scores used for partial credit can be determined by patients using quality-of-life instruments or from a survey of expert clinicians, for example. The mean partial credit scores are used to compare treatment arms.
The partial credit scoring approach is easily interpreted because of the 100-point scale, and it is useful for nonuniform steps between categories, but scoring partial credit may be more difficult than simple ranking. Partial credit scoring can be prespecified in clinical trial design for transparency. However, the treatment contrast can also be displayed, as partial credit assignment varies in sensitivity analyses, for example. Such analyses allow for (1) the assessment of robustness (e.g., if the same conclusion is reached across most reasonable grading systems, then the result is robust) and (2) personal preference regarding the value of the levels; for example, a 20-year-old patient may value certain outcomes differently from a 70-year-old patient, and analyses could be performed for each of their perspectives.
An R-Shiny app tool for analyzing DOOR has been developed. It provides primary and supportive analyses with associated downloadable data presentations. It is freely available online at extapps.bsc.gwu.edu/shinypubtst/Methods/DOOR_Shiny/.
Application to a Theoretical Trial of SE Treatment
Status epilepticus is one of the most common neurologic emergencies. For the acute treatment of convulsive SE, the clinical challenge is to rapidly terminate seizures and minimize complications related to the treatment itself. Anticonvulsant medications may cause prolonged sedation, respiratory depression, or respiratory failure, and some medications have adverse cardiovascular effects. For example, status epilepticus could be treated effectively with early, aggressive use of general anesthetic agents. However, the tradeoffs for the patient would be the need for endotracheal intubation, admission to an intensive care unit, possibly hypotension, and the potential complications of these. Whether we treat SE based on clinical suspicion or continuous EEG evidence, these tradeoffs exist.
In general, the long-term effects of SE are not directly measurable because they may not be attributable to SE (or its therapy) per se but rather to the underlying cause and are confounded by long-term care factors. Therefore, this example focuses on short-term emergency department (ED) outcomes, and we do not touch on quality-adjusted life years or other long-term outcomes. Furthermore, because of the acute emergency nature of SE, where patients are unconscious, in clinical practice, patients and caregivers are not provided with treatment options; medication choice and the decision to intubate, for example, are made by the treating clinician and are discussed with the caregivers after acute care for seizures has been provided. Therefore, in this example, we focus on outcomes ranked in desirability by clinicians, not patients.
For a clinical trial comparing an investigational study drug infused over 5 minutes with an initial expected pharmacodynamic effect by 15 minutes and ED outcome determined at 60 minutes, Table 1 depicts a proposed 5-level DOOR scale for STatus EPilepticus (DOOR STEP). We based this scale on the clinically important outcomes witnessed in the ESETT.1 Clinical outcomes are ranked by the order of desirability. The levels are mutually exclusive and exhaustive in the sense of including important clinical outcomes for all study patients for the overall patient-level experience.
Table 1.
DOOR for a Clinical Trial of a Study Drug for Status Epilepticus (DOOR STEP), With Expected Clinical Effects 15 Minutes After the Start of Infusion
| DOOR level (most to least desirable) | Rationale |
| No clinically evident or electrographic seizures requiring treatment after 15 min, not intubated, and improving mental status by 60 min | Anchor for the ideal overall patient outcome based on previous studies of status epilepticus and modern availability of rapid bedside EEG |
| There was no clinical or electrographic evidence of seizures requiring treatment after 15 min, not intubated, but mental status had not improved at 60 min | A prolonged postictal state may require further diagnostic testing or intensive care unit monitoring |
| There was no clinical or electrographic evidence of seizures, but the patient was intubated or additional seizure medication was used | Control of status epilepticus but with associated respiratory failure or the need for rescue medications. Variability among clinicians and institutions in the thresholds for endotracheal intubation is noted |
| There was clinically evident seizure or electrographic seizure requiring rescue medicine between 15 and 60 min or a continuous seizure lasting more than 5 min, or 20% of the time (9 min) occupied by seizures | Failure to control status epilepticus. Failure to control seizures is a worse outcome than respiratory failure because of ongoing status-associated complications |
| Life-threatening adverse event or death | Anchor for the worst possible outcome |
In patients without improving mental status, DOOR levels 2 and 3 require the absence of both clinical and EEG evidence of seizures. Level 4 includes either clinical or EEG evidence of seizures.
Table 2 depicts a hypothetical partial credit approach anchored on 100 for the best outcome (SE resolved and mental status improving at 60 minutes) and 0 for life-threatening complications or death using DOOR. In this example, a nearly perfect score is given to the patient who stops seizing but has not recovered mental status by 60 minutes, and a score of 40 is given to a patient whose seizures were not terminated but did not suffer a life-threatening adverse event.
Table 2.
Example of Partial Credit Scoring for DOOR STEP Outcomes
| DOOR level (most to least desirable) | Partial credit score |
| No clinically evident or electrographic seizures requiring treatment after 15 min, not intubated, and mental status improving by 60 min | 100 |
| No clinical or electrographic evidence of seizures requiring treatment after 15 min, not intubated, but mental status not improving at 60 min | 95 |
| No clinical or electrographic evidence of seizures, AND either intubation OR the use of additional seizure medications | 70 |
| Any clinically evident seizure or electrographic seizure requiring rescue medicine between 15 and 60 min or a continuous seizure lasting more than 5 min, or 20% of the time (9 min) occupied by seizures | 40 |
| Life-threatening adverse event or death | 0 |
Sensitivity analyses can be used to test study results under varying assumptions for the scores.
Table 3 shows the application of DOOR STEP to a hypothetical trial of 2 drugs for the treatment of status epilepticus. The probability of a more desirable outcome with drug A is 54.7% with a 95% CI of (50.4%, 58.9%). The advantages of drug A are evident in better rates of control of seizures and lower rates of intubation. Such a difference would not have been detected using a simple binary outcome requiring cessation of status and improving mental status at 1 hour because both groups had nearly identical rates for this outcome. The cumulative DOOR probability and associated confidence bands are shown in the Figure. In this specific example, the safety outcome of death/life-threatening adverse events can be analyzed directly because it is included in the DOOR, whereas other safety outcomes would be handled in secondary analyses.
Table 3.
Application of DOOR STEP to a Hypothetical Trial of 2 Drugs for Status Epilepticus
| Treatment group | ||
| DOOR level | A (N = 300) | B (N = 300) |
| No clinically evident or electrographic seizures requiring treatment after 15 min, not intubated, and mental status improving by 60 min | 150 (50%) | 145 (48.3%) |
| No clinical or electrographic evidence of seizures requiring treatment after 15 min, not intubated, but mental status not improving at 60 min | 75 (25%) | 34 (11.3%) |
| No clinical or electrographic evidence of seizures, AND either intubation OR the use of additional seizure medications | 50 (16.7%) | 75 (25%) |
| Any clinically evident seizure or electrographic seizure requiring rescue medicine between 15 and 60 min or a continuous seizure lasting more than 5 min, or 20% of the time (9 min) occupied by seizures | 24 (8.83%) | 45 (15%) |
| Life-threatening adverse event or death | 1 (0.3%) | 1 (0.3%) |
Figure. Application of the DOOR Method to a Hypothetical Trial of 2 Drugs for the Treatment of Status Epilepticus.
The analysis detects a higher probability of a better outcome with drug A. This result would have been missed with a traditional binary outcome measure because both groups had similar rates of the traditional binary outcome (DOOR level 1).
Discussion
We describe an approach to the design and analyses of clinical trials evaluating interventions for convulsive status epilepticus based on an ordinal outcome focused on short-term emergency department outcomes. The 5-level DOOR STEP ranges from an ideal response of efficacy with no major side effects to the worst response of life-threatening adverse events or death. DOOR STEP incorporates typical intermediary outcomes of status treatment, including a prolonged postictal state, the need for rescue medications or respiratory failure, and failure to stop seizure activity. Rapid EEG is required to distinguish among several of these outcomes. Investigators conducting clinical trials might consider including other intermediary outcomes, such as an absence of clinical seizures but the presence of seizures on EEG, or suspected aspiration pneumonia. Sample size requirements and the likely importance of outcomes to treating clinicians will be important additional considerations in balancing the inclusion of more vs fewer outcome categories.
The DOOR approach has advantages over separate analyses of efficacy and safety. First, prior studies have created de facto composite outcomes that fail to recognize important gradations of individual patient responses. For example, the ESETT Study1 used a primary outcome of no seizures, no rescue medications, and improving mental status at 1 hour. By definition, failure to meet the primary efficacy outcome was a composite outcome encompassing multiple different clinical outcomes: a prolonged postictal state, subclinical seizures, rescue medications but with eventual control of status, and intubation. These were all combined with ongoing status as “treatment failure.” The limitation with this approach is that it does not incorporate the useful information that the patient with a prolonged postictal state is better off than the patient who still has subclinical seizures and the patient who was intubated. The DOOR scale, by contrast, recognizes the tradeoffs between benefits and harms inherent in the treatment of status. For example, consider 2 patients with status epilepticus. Patient A requires intubation but has successful termination of their seizures. Patient B also requires intubation but does not have termination of their seizures. Clearly, patient A has a better outcome, but this would not be captured in a traditional binary efficacy outcome. Furthermore, they would be grouped similarly in the safety outcome of respiratory failure. Second, DOOR allows the determination of benefits and harms at the patient level. As Evans points out,18 separating efficacy and safety obscures the relationship between the 2. If efficacy and adverse events are occurring in different subgroups of patients, then the goal of therapy will be to identify these subgroups and treat them differently. If patients are experiencing both efficacy and adverse events, as occurs in most clinical scenarios, then an ordinal outcome allows investigators to measure directly the net balance of benefits and harms in each patient and to compare the overall likelihood of improved outcomes in one arm vs another.
It is important to note that the DOOR example presented here does not represent a utility ranking. Rather, this is a simple ordinal scale ranking the most to least desirable rankings and does not represent a ratio scale. A difference between 0 and 1 is not necessarily the same as the difference between 2 and 3, for example. Partial credit analyses could be mapped to utilities using a technique such as the standard gamble.12 Simple rating scales have an advantage over utility ranking in that the ranking process may be easier to understand than the standard gamble. Simple rating scales are easier to apply than health indices.12
Other investigators have used ordinal outcomes for the analysis of clinical outcomes. For example, investigators19 applied DOOR retrospectively to the patients participating in the Trial of Short-Course Antimicrobial Therapy for Intraabdominal Infection (STOP-IT). Because this trial compared short-course with standard course antibiotics, the investigators added the response adjusted for the duration of antibiotic risk analyses (RADAR), which incorporates the duration of therapy into ranking (i.e., for 2 patients with the same outcome, the patient with a shorter course of antibiotics is ranked higher). In these analyses, the DOOR-RADAR provided important additional information that was not uncovered with the original binary outcome, and such results would have been available with smaller sample sizes than the original study and likely would have resulted in earlier completion of the study. A similar concept could be used in DOOR STEP; in 2 patients with the same clinical outcome, a higher rank could be applied to the patient who has faster termination of seizures (when applicable). This recognizes the importance of increasing the duration of seizures in increasing the risk of neuronal injury. Perez et al.20 used the Desirability of Outcome Ranking Approach for the Management of Antimicrobial Therapy (DOOR MAT) to evaluate the treatment of bloodstream infections in the Veterans Health Administration. DOOR MAT scores improved in the decade of study, indicating the reduced use of broad-spectrum antibiotics without a negative impact on survival. This study focused on the desirability of treatment selection rather than the desirability of outcome but is conceptually similar. The Influence of Cooling Duration on Efficacy in Cardiac Arrest Patients (ICECAP) is prospectively enrolling patients with a primary outcome of a weighted modified Rankin Scale (mRS) determined 90 days after enrollment.17 The mRS is a 7-level ordinal scale of disability that ranges from 0 (no symptoms at all) to 6 (death). ICECAP uses weighting of mRS states to capture changes in functional status. Similarly, both the Brain Oxygen Optimization in Severe Traumatic Brain Injury—Phase 318 and the Hyperbaric Oxygen Brain Injury Treatment trial19 use the ordinal Glasgow Outcome Scale–Extended as the primary outcome to measure cognitive, behavioral, and disability outcomes.
Conclusions
The DOOR is a useful conceptual framework for designing and analyzing clinical trials based on the balance of benefits and harms. For long-term health outcomes, patient input regarding ranking and weighting of the component scores should be used in conjunction with standard decision-making tools such as the standard gamble. We present an example of DOOR in the context of clinical trials of the treatment of status epilepticus (DOOR STEP), incorporating the most common and important outcomes for the acute phase of treatment.
Glossary
- DOOR
Desirability of Outcome Ranking
- ED
emergency department
- ESETT
Established Status Epilepticus Treatment Trial
- ICECAP
Influence of Cooling Duration on Efficacy in Cardiac Arrest Patients
- mRS
modified Rankin Scale
- SE
status epilepticus
Appendix. Authors
| Name | Location | Contribution |
| James M. Chamberlain, MD | Division of Emergency Medicine, Children's National Hospital; Departments of Pediatrics and Emergency Medicine, George Washington University | Drafting/revision of the manuscript for content, including medical writing for content; study concept or design; and analysis or interpretation of data |
| Jaideep Kapur, MD, PhD | Department of Neurology, University of Virginia | Drafting/revision of the manuscript for content, including medical writing for content, and study concept or design |
| Robert S. Silbergleit, MD | Department of Emergency Medicine, University of Michigan | Drafting/revision of the manuscript for content, including medical writing for content, and study concept or design |
| Jordan J. Elm, PhD | Medical University of South Carolina | Drafting/revision of the manuscript for content, including medical writing for content, and study concept or design |
| Eric S. Rosenthal, MD | Harvard Medical School; Massachusetts General Hospital | Drafting/revision of the manuscript for content, including medical writing for content, and study concept or design |
| Thomas P. Bleck, MD | Ken and Ruth Davee Department of Neurology, Northwestern University Feinberg School of Medicine | Drafting/revision of the manuscript for content, including medical writing for content, and study concept or design |
| Shlomo Shinnar, MD, PhD | Albert Einstein College of Medicine and Montefiore Medical Center | Drafting/revision of the manuscript for content, including medical writing for content, and study concept or design |
| Shahriar Zetabchi, MD | Department of Emergency Medicine, Downstate Medical Center | Drafting/revision of the manuscript for content, including medical writing for content, and study concept or design |
| Scott R. Evans, PhD | Milken Institute School of Public Health, George Washington University | Drafting/revision of the manuscript for content, including medical writing for content; study concept or design; and analysis or interpretation of data |
Study Funding
The authors report no targeted funding.
Disclosure
The authors report no relevant disclosures. Go to Neurology.org/N for full disclosures.
References
- 1.Kapur J, Elm J, Chamberlain JM, et al. Randomized trial of three anticonvulsant medications for status epilepticus. N Engl J Med. 2019;381(22):2103-2113. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Evans SR, Rubin D, Follmann D, et al. Desirability of outcome ranking (DOOR) and response adjusted for duration of antibiotic risk (RADAR). Clin Infect Dis. 2015;61(5):800-806. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Van Duin D, Lok JJ, Earley M, et al. Colistin vs. ceftazidime-avibactam in the treatment of infections due to carbapenem-resistant enterobacteriaceae. Clin Infect Dis. 2018;66(2):163-171. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Lodise TP Jr, Rosenkranz SL, Finnemeyer M, et al. The emperor's new clothes: prospective observational evaluation of the association between initial vancomycin exposure and failure rates among adult hospitalized patients with MRSA bloodstream infections (PROVIDE). Clin Infect Dis. 2020;70(8):1536-1545. doi. 10.1093/cid/ciz460 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.van Duin D, Arias CA, Komarow L, et al. Molecular and clinical epidemiology of carbapenem-resistant enterobacteriaceae in the United States: a prospective cohort study. Lancet Infect Dis. 2020;20(6):731-741. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Williams DJ, Creech CB, Walter EB, et al. Short course outpatient antibiotic therapy for community-acquired pneumonia in children. JAMA Pediatr. 2022;176(3):253-261. doi: 10.1001/jamapediatrics.2021.5547 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Study Protocol. ICECAP: Influence of Cooling Duration on Efficacy in Cardiac Arrest Patients. Accessed July 2, 2022. docs.google.com/document/d/1t88x-lwoCKIOuTvbm5brFIdGGhK--JZWQLCSPqfp5zg/edit#heading=h.3dy6vkm. [Google Scholar]
- 8.Barsan W. Brain Oxygen Optimization in Severe TBI, Phase 3 (BOOST 3). ClinicalTrials. gov Identifier: NCT03754114. US National Library of Medicine; 2021. [Google Scholar]
- 9.Study Protocol. Hyperbaric Oxygen Brain Injury Treatment (HOBIT) Trial: A Multicenter, Randomized, Prospective Phase II Adaptive Clinical Trial Evaluating the Most Effective Hyperbaric Oxygen Treatment Paradigm for Severe Traumatic Brain Injury. Accessed July 2, 2022. nett.umich.edu/sites/default/files/docs/hobit_protocol_version_2.pdf. [Google Scholar]
- 10.Weinstein MC, Torrance G, McGuire A. QALYs: the basics. Value Health. 2009;12:S5-S9. [DOI] [PubMed] [Google Scholar]
- 11.Howard-Anderson J, Hamasaki T, Dai W, et al. Improving traditional registrational trial endpoints: development and application of a desirability of outcome ranking endpoint for complicated urinary tract infection clinical trials. Clin Infect Dis. 2023;76(3):e1157-e1165. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Hunink MGM, Weinstein MC, Wittenberg E, et al. Decision Making in Health and Medicine: Integrating Evidence and Values. Cambridge University Press; 2014:447. [Google Scholar]
- 13.Vergel YB, Sculpher M. Quality-adjusted life years. Pract Neurol. 2008;8(3):175-182. [DOI] [PubMed] [Google Scholar]
- 14.Furlong WJ, Feeny DH, Torrance GW, Barr RD. The Health Utilities Index (HUI®) system for assessing health-related quality of life in clinical studies. Ann Med. 2001;33(5):375-384. [DOI] [PubMed] [Google Scholar]
- 15.Halperin M, Hamdy MI, Thall PF. Distribution-free confidence intervals for a parameter of Wilcoxon-Mann-Whitney type for ordered categories and progressive censoring. Biometrics. 1989;45:509-521. [PubMed] [Google Scholar]
- 16.Follmann DA. Regression analysis based on pairwise ordering of patients' clinical histories. Stat Med. 2002;21:335-367. [DOI] [PubMed] [Google Scholar]
- 17.Follmann D, Fay MP, Hamasaki T, Evans S. Analysis of ordered composite endpoints. Stat Med. 2020;39:602-616. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Evans SR, Follmann D. Using outcomes to analyze patients rather than patients to analyze outcomes: a step toward pragmatism in benefit:risk evaluation. Stat Biopharm Res. 2016;8(4):386-393. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Celestin AR, Odom SR, Angelidou K, et al. Novel method suggests global superiority of short-duration antibiotics for intra-abdominal infections. Clin Infect Dis. 2017;65(9):1577-1579. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Perez F, Colindres RV, Wilson BM, et al. Desirability of outcome ranking for the management of antimicrobial therapy (DOOR MAT) reveals improvements in the treatment of bloodstream infections caused by Escherichia coli and Klebsiella pneumoniae in patients from the Veterans Health Administration. Clin Infect Dis. 2021;73(7):1231-1238. [DOI] [PMC free article] [PubMed] [Google Scholar]