Applying a Risk-benefit Analysis to Outcomes in Tuberculosis Clinical Trials

Abstract

Although it is common to analyze efficacy and safety separately in clinical trials, this could yield a misleading study conclusion if an increase in efficacy is accompanied by a decrease in safety. A risk-benefit analysis is a systematic approach to examine safety and efficacy jointly. Both the “rank-based” and “partial-credit” methods described in this paper allow researchers to create a single, composite outcome incorporating efficacy, safety, and other factors. The first approach compares the distribution of rankings between arms. In the second approach, a score can be assigned to each outcome category, considering its severity and comparing the mean or median scores of arms. The methods were applied to the A5279/Brief Rifapentine-Isoniazid Efficacy for TB Prevention study, and design considerations for future clinical trials are discussed, including the challenge of arriving at a consensus on rankings/scorings. If well designed, a risk-benefit analysis may allow for a superiority comparison and, therefore, avoid setting a noninferiority margin.

Clinical Trials Registration. NCT01404312 (A5279).

A risk-benefit analysis is a flexible approach to examining efficacy and safety simultaneously. We describe 2 methods with scoring/ranking, applied to A5279/BRIEF-TB outcomes. If well designed, they may allow researchers to compare arms in a superiority, rather than noninferiority, setting.

In most tuberculosis (TB) treatment clinical trials, an efficacy outcome, such as the proportion of participants with an unfavorable TB outcome by the end of the study follow-up, is evaluated as the primary analysis, and a safety outcome, such as the proportion of participants experiencing 1 or more serious adverse events (SAEs), is assessed separately as a secondary analysis. Similarly, in TB prevention trials, the efficacy outcome of active TB is often analyzed separately from a safety outcome. Although this approach is commonly practiced, it might not capture the overall picture of study results, which influence the study conclusions. A risk-benefit analysis that utilizes a composite outcome is a systematic and comprehensive approach to examining safety and efficacy simultaneously.

Several methods for examining multiple outcomes simultaneously have been proposed in the statistical and clinical literature. Follmann [1] proposed ranking 1 participant relative to the other in each pair of participants across arms based on multiple outcomes, with logistic regression used on the rankings to compare treatment arms. Evans et al [2, 3] proposed ranking each participant using the desirability of outcome ranking and the response adjusted for the duration of antibiotic risk. Montepiedra et al [4] discussed the application of the desirability of outcome ranking approach to TB research.

Related methods have been applied to outcomes from clinical trials in cardiovascular disease [5] and Staphylococcus aureus bloodstream infections [6]; however, to our knowledge, they have not been formally applied to outcomes in TB clinical trials. In this paper, we discuss the motivation for conducting a unified risk-benefit approach in this context, describe 2 types of risk-benefit analysis methods that could be applied to a TB treatment-shortening or prevention trial, and apply them to the outcomes from the A5279 TB prevention trial [7] to illustrate how the methods work and how a superiority trial using the risk-benefit analysis can be an alternative to a noninferiority design that would analyze efficacy and safety separately.

MOTIVATION

Motivation #1: Examining Concordance Between Efficacy and Safety

Suppose a TB prevention trial to compare efficacy and safety between experimental and control arms showed that the proportion of participants who developed TB was 5% in both arms, and the proportion of participants who had at least 1 SAE was 10% in both arms. In this case, the study team might conclude that the 2 arms were equal. However, what if the breakdown of efficacy and safety rates were as shown in Table 1? In this extreme situation, all participants in the experimental arm who developed TB also experienced SAEs, and none of the participants in the control arm who developed TB experienced SAEs. The risk of developing TB in the experimental arm could be increased due to SAEs requiring the discontinuation of treatment. By examining efficacy and safety separately, the study results would not capture this strong concordance between the 2 endpoints, which could lead to an incomplete picture of the comparative safety/efficacy profiles of the 2 arms.

Table 1.

Example for Motivation #1: Breakdown of Efficacy and Safety Rates by Arm

Outcome	Experimental Arm, %	Control Arm, %
No TB/No SAE	90	85
No TB/SAE	5	10
TB/No SAE	0	5
TB/SAE	5	0

Abbreviations: SAE, serious adverse effect; TB, tuberculosis.

Motivation #2: Examining Balance Between Efficacy and Safety

Suppose a noninferiority clinical trial for TB treatment shortening was conducted and the primary analysis was to compare efficacy outcomes between experimental and control arms. The efficacy data showed that noninferiority was not demonstrated (Figure 1). However, the safety data showed that the experimental arm was safer than the control arm (Figure 2). In such a situation, a method that accounted for both efficacy and safety might be desired, allowing gains in 1 dimension to offset losses in another. A risk-benefit analysis allows researchers to predetermine the trade-off between these outcomes. In a noninferiority trial, hypothesized safety rates may be considered when specifying the margin, but not the actual observed rates [8].

Figure 1.

Example for Motivation #2: efficacy result from a noninferiority tuberculosis treatment shortening trial. Abbreviations: EXP, experimental arm; SOC, standard of care/control arm; NI, noninferiority margin.

Figure 2.

Example for Motivation #2: safety result from a noninferiority tuberculosis treatment shortening trial. Abbreviations: EXP, experimental arm; SOC, standard of care/control arm.

Motivation #3: Avoiding the Competing Risk Problem

Handling deaths in an efficacy analysis is seldom a simple task. If a participant dies from causes other than TB or TB treatment, or discontinues study participation before reaching the efficacy endpoint, the efficacy outcome for this participant is missing. This is known as a competing risk problem. If we simply exclude this participant from the final analysis, the result could be biased. A common approach to handling this problem is to create a binary, composite outcome of active TB and death. The interpretation of such a composite outcome can be misleading, especially when each component has an unequal impact on the participant’s well-being. Table 2 illustrates how a simple composite outcome of TB and death can yield a misleading result. It is clear that the control arm is better in this scenario, but a test based on the overall proportion of participants positive for the composite outcome would conclude that both arms were equal. A risk-benefit analysis enables different outcomes and combinations of components to be prioritized or weighted differently, taking severity into account.

Table 2.

Example for Motivation #3: Breakdown of Tuberculosis and Death Rates by Arm

Outcome	Experimental Arm, %	Control Arm, %
Active TB	1	4
Death due to TB or toxicity	5	1
Death from other causes	0	1
Overall	6	6

Abbreviation: TB, tuberculosis.

Risk-benefit Analysis Methods

We present 2 related methods of risk-benefit analyses that have previously been proposed, which we call the rank-based method and the partial-credit method. Common to both methods the assignment of a single, ordinal outcome category to each participant that takes into account multiple domains. For example, suppose in a TB treatment trial we want to consider efficacy (favorable, unfavorable) and safety (SAE, no SAE, death) simultaneously; then a reasonable ordering of the categories for combined efficacy and safety outcomes could be shown in Table 3.

Table 3.

Example Scoring Categories for the Rank-based and Partial-credit Methods

Outcome Category	Ordinal Score	Partial-credit Score
Favorable efficacy and no SAE	1	100
Favorable efficacy and SAE	2	80
Unfavorable efficacy and no SAE	3	20
Unfavorable efficacy and SAE	4	10
Death	5	0

Abbreviation: SAE, serious adverse effect.

For the rank-based method, after assigning each participant an ordinal score, the distribution of scores is compared between arms using a Wilcoxon rank-sum test, or using ordinal logistic regression if we need to adjust for baseline covariates. The partial-credit method further requires that we assign a score to each category. Possible partial-credit scores are shown in Table 3. This method allows us to weight the ordinal categories to account for the differing severities of the combinations of outcomes. After assigning each participant a partial score, we compare the mean scores between arms using a t-test, or using linear regression if we need to adjust for baseline covariates.

An appealing aspect of a risk-benefit analysis is that the ordinal categories can be extended easily to incorporate additional factors (tolerability, quality of life, etc) The selection of factors is likely to vary across disease models, types of trials, or study populations. For example, in pediatric TB trials, a study team may want to treat an irreversible adverse event (AE), such as hearing loss, differently compared to reversible or treatable AEs. Persistent lower-grade AEs can be considered similarly to a treatable higher-grade AE. If investigators wish to add granularity into the scoring, within the same category, an additional tie-breaker factor can be introduced. The grade of AE, total number of AEs, or duration of a specific AE can be used as a tie breaker. It is important to note that the ranking/scoring needs to be specified prior to beginning the trial.

The above example has a single death category; however, a death due to TB can be differentiated from a death with a cause that is clearly not related to TB or TB treatment; for example, a death due to an accident. Instead of giving a score of 0 for all deaths, different scores can be assigned to each subcategory of death. Alternatively, deaths not related to TB or TB treatment can be treated as missing data if they are considered to be missing at random, and standard missing data analysis methods, such as inverse probability weighting methods [9], can be applied.

Accounting for Loss to Follow-up

A limitation of the risk-benefit analysis methods described above is that they do not account for differential follow-up of participants. However, the methods can be adapted to do so. Instead of assigning each participant to a category and then conducting the Wilcoxon test or t-test, we can compare pairs of participants between arms up to their common follow-up time. Suppose we are comparing Arm A participant A₁ and Arm B participant B₁, and B₁ has an SAE and withdraws from the study at Month 1, while A₁ has no events until developing TB at Month 3. In this case, we would only compare the 2 participants up to Month 1, since we do not have information about B₁ after Month 1. With the rank-based method, the A₁B₁ comparison would equal 1 if A₁ had a better outcome than B₁, 0 if they had the same outcome, and -1 if A₁ had a worse outcome than B₁. Although A₁ developed TB, A₁ has a better outcome up to the common follow-up time, so this pairwise comparison would score 1. We would similarly compare A₁ with B₂, B_3, …, B_n, and then sum each of the pairwise scores. This sum would be equal to the rank for participant A_1. The same process would be repeated for each participant on both arms, and then a Wilcoxon rank-sum test would be performed on these ranks. This approach to deal with censored data has been proposed in the statistical literature [10, 11], and is most appropriate when the follow-up time distributions are similar between arms, which is common in clinical trials. It has also been demonstrated to provide valid tests when both arms have the same distribution of outcomes over time, even if the follow-up distributions are different in each arm [12, 13]. However, because the follow-up distributions influence the effect estimates, it may be difficult to compare estimates between different trials [14].

The same approach can be used with the partial-credit method, but instead of 1, 0 or -1 for the pairwise comparison scores, each pairwise comparison would be the difference in partial-credit scores between participants on arms A and B. The sum or average of the pairwise scores for each participant would constitute their overall partial-credit score. A t-test comparing these overall scores between arms, or a linear regression if we needed to adjust for baseline covariates, could then be performed. Alternatively, inverse probability of censoring weighted methods can be applied to account for losses to follow-up.

Design Considerations for Risk-benefit Trials

Prior to the decision to use a risk-benefit approach, the study team needs to be clear about the objectives and assumptions of the trial. In a TB treatment or prevention trial, is it anticipated that the experimental regimen will be more effective on safety and noninferior on efficacy, will be more effective on efficacy and noninferior on safety, or will improve both safety and efficacy? Each of these scenarios may have different implications in how the team defines the risk-benefit outcome, and the sample size needed to detect differences between arms with requisite power.

To estimate the power and sample size needed for a risk-benefit analysis, the study team needs to make assumptions about the proportion of participants in each arm expected to fall in each category, in order to characterize the distribution of the risk-benefit outcome in each arm. For a fixed sample size, the data can be simulated from this distribution, and the proportion of times the null hypothesis is rejected would provide an estimate of the power. The sample size can be adjusted until the desired power is achieved. Further, the team can compare the different methods of risk-benefit analyses in their design simulations, and select the method that would yield the smallest sample size for a given power. The method that is ultimately chosen should still be clinically relevant, even if it does not yield the smallest sample size.

The risk-benefit analysis will not necessarily improve efficiency in terms of sample size over a traditional analysis of a single outcome, but instead is a method of adequately capturing multiple components of a participant’s experience in a single measure. Nevertheless, it may be more efficient than traditional analyses in certain scenarios. If an experimental arm is similar on efficacy and superior on safety, a risk-benefit analysis will often have better power, as compared to having to show both noninferior efficacy and superior safety as co-primary objectives, but this will depend on the non-inferiority margin and magnitude of superiority on safety.

Application to Outcomes From the A5279/Brief Rifapentine-Isoniazid Efficacy for TB (BRIEF-TB) Prevention Study

A5279/BRIEF-TB was a Phase III clinical trial comparing a 4-week daily isoniazid and rifapentine regimen (1HP) to the standard 9-month daily isoniazid regimen (9H). The study randomized 3000 participants living with HIV who were without evidence of active TB at entry and followed them for at least 3 years. The study concluded that the 1HP regimen had non-inferior efficacy to the 9H regimen and had slightly improved safety.

Prior to applying the risk-benefit analysis to the study outcomes, a survey was created to facilitate the development of scores of composite outcome categories. For the sake of simplicity, 7 basic categories were created: (1) no TB/≤Grade 2 AE; (2) no TB/Grade 3 AE; (3) no TB/Grade 4 AE; (4) TB/≤ Grade 2 AE; (5) TB/Grade 3 AE; (6) TB/Grade 4 AE; and (7) death. Category 1 was given a fixed score of 100 to reflect the best outcome and Category 7 was given a score of 0 to reflect the worst outcome. A convenience sample of clinical TB investigators from the Acquired Immunodeficiency Syndrome (AIDS) Clinical Trials Group and International Maternal Pediatric Adolescent AIDS Clinical Trials Network networks were asked to assign any score between 0 and 100 to each of the 5 remaining categories. The results. based on 18 survey respondents, are summarized in Table 4.

Table 4.

Survey Results: Descriptive Statistics on Scoring of Ranking Categories

Category	Efficacy	Safety	Mean (SD)	Median
1	No TB	≤G2	100	100
2	No TB	G3	68.9 (18.7)	72.5
3	No TB	G4	50.6 (18.6)	50
4	TB	≤G2	33.1 (16.8)	30
5	TB	G3	19.7 (12.1)	20
6	TB	G4	11.1 (7.9)	10
7	G5 = death		0	0

N = 18.

Abbreviations: G, grade; SD, standard deviation; TB, tuberculosis.

We used the median score assigned for each category by survey respondents as the pre-specified score for that category. Each participant was assigned a score based on their experience on A5279/BRIEF-TB, and the mean scores were compared between arms by the t-test. The mean scores were 95.5 (standard deviation [SD] 16.6) and 94.7 (SD 18.0) for the 1HP and 9H regimens, respectively. Although the mean was slightly higher in the 1HP regimen, there was no significant difference between the regimens (P = .20). In addition, the frequency distribution of ordinal categories by arm (Table 5) was compared using the Wilcoxon rank-sum test. Since the majority of participants (>90%) fell under the best category, no significant difference between the study arms was detected (P = .18).

Table 5.

Frequency Distribution of Ranking Categories by A5279/BRIEF-TB Study Arm

Rank	Efficacy	Safety	1HP, n (%)	9H, n (%)
1	No TB	≤G2	1359 (91)	1347 (90)
2	No TB	G3	62 (4)	69 (5)
3	No TB	G4	22 (1)	31 (2)
4	TB	≤G2	21 (1)	20 (1)
5	TB	G3	1 (<1)	2 (<1)
6	TB	G4	2 (<1)	1 (<1)
7	G5 = death		21 (1)	28 (2)

Abbreviations: 1HP, 4-week daily isoniazid and rifapentine regimen; 9H, 9-month daily isoniazid regimen; BRIEF-TB, Brief Rifapentine-Isoniazid Efficacy for TB Prevention; G, grade; TB, tuberculosis.

Adding additional granularity, a post hoc analysis was performed on expanded outcome categories (Table 6), where Grade 2 AEs were separated from Grade ≤1 AEs. With the expanded categories, a statistically significant difference between the arms was detected (P = .03) using the Wilcoxon rank-sum test. While this was a post hoc analysis, this example illustrates the potential application of the risk-benefit approach, which could have shown that 1HP was superior to 9H in the A5279/BRIEF-TB study based on a more comprehensive assessment of participants’ experiences, despite both arms having very similar efficacy. If well designed, a superiority setting using the risk-benefit analysis can be an alternative to a noninferiority design. where efficacy and safety are analyzed separately and setting a noninferiority margin can be challenging.

Table 6.

Frequency Distribution of Ranking Categories by A5279/BRIEF-TB Study Arm

Rank	Efficacy	Safety	1HP, n (%)	9H, n (%)
1	No TB	≤G1	1269 (85)	1234 (82)
2	No TB	G2	90 (6)	113 (8)
3	No TB	G3	62 (4)	69 (5)
4	No TB	G4	22 (1)	31 (2)
5	TB	≤G1	20 (1)	17 (1)
6	TB	G2	1 (<1)	3 (<1)
7	TB	G3	1 (<1)	2 (<1)
8	TB	G4	2 (<1)	1 (<1)
9	G5 = death		21 (1)	28 (2)

Abbreviations: 1HP, 4-week daily isoniazid and rifapentine regimen; 9H, 9-month daily isoniazid regimen; BRIEF-TB, Brief Rifapentine-Isoniazid Efficacy for TB Prevention; G, grade; TB, tuberculosis.

DISCUSSION

The risk-benefit approach discussed in this paper allows researchers to assess efficacy, safety, and other factors simultaneously with a single-hypothesis test. This approach could incorporate input from study participants on their overall experiences with the study treatment (eg, quality of life), in addition to considering input from clinical investigators in the outcome. The comparison will be of superiority; thus, the challenge of prespecifying a noninferiority margin could be avoided.

The disadvantage of the risk-benefit approach is that the ranking/scoring of outcome categories can be subjective, and arriving at a consensus could be challenging. Especially in the partial-credit approach, there will be inter-stakeholder variation in the score assigned for each category. If a survey was administered, sensitivity analyses should be conducted by repeating the analysis using the scoring from different survey participants. Another limitation is that the interpretation of the risk-benefit analysis is less straightforward, compared to the interpretation of the marginal analyses. The risk-benefit analysis simply evaluates whether participants on the experimental arm have a higher chance of a better overall experience, compared to the control. We can interpret the risk-benefit analysis as a comparison between 2 competing treatments, based on a utility function that accounts for the overall patient experience with respect to efficacy and safety. For instance, in the partial-credit method, the treatment effect of interest could be seen as a weighted sum of the treatment differences in proportions of the categories; in the rank-based method, the treatment effect of interest could be seen as the probability that the experimental arm has a better overall outcome than the control arm. Finally, the standardization of ranking/scoring across multiple TB trials is challenging, and cross-trial comparisons may not be feasible, at least not this early in the use of risk-benefit approaches in TB trials.

We applied the risk-benefit approaches to outcomes from the A5279/BRIEF-TB study to illustrate the method and how a superiority comparison using the risk-benefit outcome could be used as an alternative study design. The example used the data from a TB prevention trial; however, the risk-benefit analysis can be applied to treatment-shortening or regulatory trials. The ranking/scoring of categories will need to be tailored to each type of trial, depending on the objectives of the trial, the treatment regimens, and the population. It is also important that the choice of categories and their ranks/scores should be prespecified prior to the initiation of the study.

Using a risk-benefit analysis in any clinical trial requires careful planning during protocol development, through discussion among stakeholders. Gaining acceptability of the use of composite ranking/scoring as the primary outcome may take time; however, it can be use as a primary analysis if the risk benefit analysis is the most relevant question of interest. A concern raised by Philips et al [15] was that a single, composite risk-benefit outcome could “obscure important differences in the clinical outcome.” The risk-benefit approach is not meant to supplant the analysis of marginal efficacy and safety data, but rather to be used in concert with the marginal analyses to provide a robust picture of the overall effect of each regimen. If it is used as a primary analysis, it is important to examine the marginal efficacy and safety data in secondary analyses to understand the overall results of the trial.

Notes

Acknowledgments. The authors thank Sanofi for donating the A5279 study medications.

Disclaimer. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Financial support. Research reported in this publication was supported by the National Institute of Allergy and Infectious Diseases (NIAID) of the National Institutes of Health (NIH) (award numbers UM1 AI068634, UM1 AI068636, and UM1 AI106701 to the AIDS Clinical Trials Group apply to authors S. M., R. R., S. S., and R. E. C.; award number UM1 AI068616 to the International Maternal Pediatric Adolescent AIDS Clinical Trials Network applies to authors A. G., S. K., and G. M.

Potential conflicts of interest. S. R. E. reports grants from the NIAID of the NIH (ARLG Network) and personal fees from Takeda/Millennium (Data Safety Monitoring Board [DSMB]), Pfizer (DSMB), Roche (DSMB), Novartis (DSMB), Achaogen (DSMB), Huntington’s Study Group (DSMB), Analgesic, Anesthetic, and Addiction Clinical Trial Translations, Innovations, Opportunities, and Networks (ACTTION) (Think Tank), Genentech (Think Tank), Amgen (Think Tank and DSMB), GSK (teaching), American Statistical Association (Board of Directors; teaching/travel), Food and Drug Administration (Advisory Committee/teaching), Osaka University (teaching), National Cerebral and Cardiovascular Center of Japan (teaching), the NIH (DSMB), the Society for Clinical Trials (Board of Directors; travel/speaking), Statistical Communications in Infectious Diseases (Editor-in-Chief), AstraZeneca (advisor), Teva (Think Tank), Austrian Breast & Colorectal Cancer Study Group/Breast International Group and the Alliance Foundation Trials (DSMB), Zeiss (Think Tank), Dexcom (Think Tank), the American Society for Microbiology (lecture), Taylor and Francis (book royalty), Claret Medical (Think Tank), Vir (Advisor), Arrevus (Advisor), Five Prime (DSMB), Shire (DSMB), Alexion (DSMB), Gilead (DSMB), Spark (Think Tank), the Clinical Trials Transformation Initiative (travel), Nuvelution (DSMB), Tracon (DSMB), the Deming Conference (keynote speaker), the Antimicrobial Resistance and Stewardship Conference (speaker), the World Antimicrobial Congress (speaker), WAVE (DSMB), Advantagene (DSMB), Braeburn (Think Tank), Cardinal Health (Think Tank), Lipocine (Think Tank), Microbiotix (advisor), and Stryker (Think Tank), outside of the submitted work. A. G. received grants from the NIH (UM1AI069465) for this work and grants from the National Institutes of Health, Centers for Disease Control and Prevention, Gilead Foundation, Wyncote Foundation, and UNITAID, outside of the submitted work. S. S. has received grants from ViiV Healthcare, outside of the submitted work. R. E. C. received grants from the National Institutes of Health and personal fees from Merck (spouse is shareholder; consulting), Otsuka (consulting), and Sanofi (consulting), outside of the submitted work. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.