A Clinical Prediction Model for Unsuccessful Pulmonary Tuberculosis Treatment Outcomes

Abstract

Background

Despite widespread availability of curative therapy, tuberculosis (TB) treatment outcomes remain suboptimal. Clinical prediction models can inform treatment strategies to improve outcomes. Using baseline clinical data, we developed a prediction model for unsuccessful TB treatment outcome and evaluated the incremental value of human immunodeficiency virus (HIV)–related severity and isoniazid acetylator status.

Methods

Data originated from the Regional Prospective Observational Research for Tuberculosis Brazil cohort, which enrolled newly diagnosed TB patients in Brazil from 2015 through 2019. This analysis included participants with culture-confirmed, drug-susceptible pulmonary TB who started first-line anti-TB therapy and had ≥12 months of follow-up. The end point was unsuccessful TB treatment: composite of death, treatment failure, regimen switch, incomplete treatment, or not evaluated. Missing predictors were imputed. Predictors were chosen via bootstrapped backward selection. Discrimination and calibration were evaluated with c-statistics and calibration plots, respectively. Bootstrap internal validation estimated overfitting, and a shrinkage factor was applied to improve out-of-sample prediction. Incremental value was evaluated with likelihood ratio–based measures.

Results

Of 944 participants, 191 (20%) had unsuccessful treatment outcomes. The final model included 7 baseline predictors: hemoglobin, HIV infection, drug use, diabetes, age, education, and tobacco use. The model demonstrated good discrimination (c-statistic = 0.77; 95% confidence interval, .73–.80) and was well calibrated (optimism-corrected intercept and slope, –0.12 and 0.89, respectively). HIV-related factors and isoniazid acetylation status did not improve prediction of the final model.

Conclusions

Using information readily available at treatment initiation, the prediction model performed well in this population. The findings may guide future work to allocate resources or inform targeted interventions for high-risk patients.

We detail the development and internal validation of a prognostic model, including 7 easily collected variables that accurately predict unsuccessful pulmonary tuberculosis treatment outcome. The model can be applied at the point of care with a nomogram or web application.

Tuberculosis (TB) is a leading cause of death worldwide [1]. Despite widespread availability of effective drugs, TB treatment outcomes remain suboptimal. The World Health Organization (WHO) estimated the global TB treatment success rate was 85% in 2018 [1]. A recent study in Brazil, a high-burden TB country, found that from 2015 through 2019, only 67% of TB patients reported to the Brazilian National TB Program Notifiable Disease Information System were successfully treated, whereas 20% were lost to follow-up or transferred, 9% died, and 4% failed treatment or relapsed [2]. This is well below the End TB Strategy goal of 90% treatment success by 2025 [1, 3].

To improve overall treatment outcomes, clinicians and researchers should swiftly identify patients most likely to have unsuccessful treatment, then augment treatment or intervention strategies to support them. One approach to identify high-risk patients is clinical prediction modeling, which estimates an individual’s risk of a specific end point within a defined time period [4]. In a recent systematic review, 33 studies presenting 37 prediction models for end-of-treatment TB outcomes were identified [5]. All models suffered bias due to poor reporting of the study population and data collection, exclusion of missing data, univariate analysis–based model selection procedures, lack of validation, or limited generalizability.

With data available at treatment initiation, we developed and internally validated a prediction model for unsuccessful pulmonary TB treatment outcomes among patients with culture-confirmed, drug-susceptible pulmonary TB who were treated with standard anti-TB therapy. Additionally, given the strong effect of human immunodeficiency virus (HIV) infection on TB treatment outcomes [6] and the importance of isoniazid metabolism for safety and efficacy of TB treatment [7], we evaluated the incremental value of HIV-related severity measures and isoniazid acetylator status to provide insight about the importance of collecting these data in routine care.

METHODS

Study Design and Population

In this study, we used data from the Regional Prospective Observational Research for Tuberculosis (RePORT) Brazil cohort, a prospective study of TB patients at 5 sites across 3 regions in Brazil: 3 in Rio de Janeiro (Instituto Nacional de Infectologia Evandro Chagas, Clínica de Saúde Rinaldo Delmare, Secretaria de Saúde de Duque de Caxias), 1 in Salvador (Instituto Brasileiro para Investigação da Tuberculose), and 1 in Manaus (Fundação Medicina Tropical Dr Heitor Vieira Dourado) [2]. Participants were consecutively enrolled at each site from June 2015 through June 2019 with active follow-up through June 2020. The population of RePORT-Brazil is broadly representative of TB cases in Brazil [2].

RePORT-Brazil participants with newly diagnosed, culture-confirmed, drug-susceptible pulmonary TB who were aged ≥18 years and started a standard first-line anti-TB therapy regimen within the last 7 days were included. Standard anti-TB therapy included isoniazid, rifampin or rifabutin, pyrazinamide, and ethambutol for 2 months, followed by isoniazid and rifampin for 4 months [8]. Participants who received anti-TB therapy for ≥7 days within 30 days of enrollment, received >7 days of fluoroquinolone therapy within 30 days of enrollment, were pregnant or breastfeeding, or did not plan to remain in the region during follow-up were excluded.

Standardized clinical, demographic, and outcome information was collected longitudinally at 3 clinical visits (TB treatment initiation [baseline], 2 months after initiating treatment, and end of TB treatment) and via telephone follow-up every 6 months until 24 months [9, 10]. Methods and results are reported according to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis Guidelines (Supplementary Materials 1) [11].

Outcomes and Predictors

Participants were assigned 1 of 6 mutually exclusive TB treatment outcomes based on new WHO guidelines (Table 1) [12]. The outcomes were cure, treatment completion, treatment failure, death due to any cause, treatment incomplete, and not evaluated. Cure and treatment completion were collectively considered successful TB treatment and were the referent outcome. Treatment failure, death due to any cause, treatment incomplete, and not evaluated were collectively considered unsuccessful TB treatment, which was the outcome of interest. Standard TB treatment typically lasts 6 months, but we allowed follow-up through 1 year to ensure completion of treatment. For a 20% outcome rate and 12–21 candidate predictors, we estimated that between 710 and 1242 participants were needed for sufficient precision of the prediction; thus, developing the model using available data was well justified [13].

Table 1.

World Health Organization Definitions of Tuberculosis Treatment Outcomes for Drug-Susceptible Patients on a First-Line Drug Regimen

Outcome	Definition
Cure	A patient with bacteriologically confirmed TB at the beginning of treatment who completed treatment as recommended by the national policy and who had evidence of bacteriologic response (a negative smear or culture at the end of TB treatment and on at least 1 previous occasion more than 7 days apart)
Treatment completion	A patient who completed treatment as recommended by national policy but whose outcome does not meet the definition for cure or failure, either because tests were not done or because results were unavailable
Treatment success	Composite of cured and treatment completed
Treatment failure	A patient whose regimen needed to be terminated due to lack of clinical response (sputum smear or culture is positive at month 5 or later during treatment) or whose regimen was permanently changed to a new regimen or treatment strategy
Death	A patient who died for any reason during the course of TB treatment
Treatment incomplete	A patient whose TB treatment was interrupted for 2 consecutive months or more
Not evaluated	A patient for whom no TB treatment outcome was assigned, which includes cases who “transferred out” to another treatment unit as well as cases for whom the treatment outcome was unknown to the reporting unit

Abbreviation: TB, tuberculosis.

Candidate predictors were selected a priori using previous TB prediction models [5] and clinical input from co-authors. Fifteen baseline candidate predictors were considered: age, sex, self-reported race, years of formal education, body mass index (BMI), previous TB, cavitation on chest radiograph, smear positive, HIV infection, diabetes (self-reported history of diabetes or glycated hemoglobin ≥6.5%) [14], hemoglobin, any other chronic disease comorbidity, tobacco use, drug use, and alcohol use. Full definitions are available in Supplementary Materials 2.

Model Development and Validation

Model development and validation are detailed in Figure 1. Missing values for predictors were multiply imputed over 10 iterations. Imputed values were averaged by taking the median of continuous variables and mode of categorical variables across all 10 imputed datasets and summarized in 1 complete dataset for primary analysis [15]. Sensitivity analyses were performed replicating all model selection and validation steps in complete case analysis and each imputed dataset (Supplementary Materials 3).

Figure 1.

Schematic representation of each model development step and assessment of model performance. First, missing data were imputed across 10 datasets and summarized into an analysis dataset. Next, redundancy analysis and linearity assessment were carried out to identify highly correlated sets of variables and variables with evidence of nonlinearity. Following that, 500 repetitions of bootstrapped backward selection were used to identify the most important predictors of unsuccessful tuberculosis treatment outcome based on those included in at least 70% of bootstrap samples. Finally, model performance was evaluated in the original sample (apparent performance) and averaged over 2000 bootstrap samples (internal bootstrap validation). *Sensitivity analyses were conducted repeating all steps following imputation of the original data with missing information (complete case analysis) and in each of the imputed datasets. ^Model performance measures included discrimination (evaluated with the c-statistic) and calibration (evaluated using a calibration plot and the calibration slope and intercept).

Candidate predictors were examined for collinearity with redundancy analysis based on Hoeffding D statistic [16]. Nonlinearity was evaluated with a chunk test using restricted cubic splines with 5 knots for continuous predictors (age, BMI, education, and hemoglobin) [16]. Age was the only nonlinear variable and was therefore categorized (18–24.9, 25–34.9, 35–44.9, 45–54.9, and ≥55 years) for primary analysis and modeled using a restricted cubic spline in sensitivity analyses (Supplementary Materials 4).

The primary model building process used logistic regression for a binary end point. Bootstrapped backward selection was used to identify the most important set of predictors for unsuccessful TB treatment [17]. Variables selected in ≥70% of the 500 bootstraps were included in the final model [15, 18]. Sensitivity analyses evaluated a 50% threshold.

Performance was evaluated with discrimination and calibration. Discrimination was quantified with the c-statistic [19]. Calibration was assessed using calibration plot, calibration intercept, calibration slope, and a Hosmer–Lemeshow goodness-of-fit test [20]. Overall model fit was evaluated with the Brier score. Internal validation with bootstrap resampling was used to estimate optimism-corrected performance measures [16]. Predictions from the final model account for shrinkage according to the heuristic shrinkage factor, estimated as χ ^2model—df / χ ²_model, where χ ²_model is the χ ² model and df is the degrees of freedom [21]. Coefficients and model performance for the main model building process were compared to model approximation [16] and least absolute shrinkage and selection operator (LASSO) [22] (Supplementary Materials 5).

We conducted decision curve analysis to evaluate the net benefit of using the prediction model to inform care decisions across a range of threshold probabilities [23, 24] (additional methods in Supplementary Materials 6). We suggested cutoffs, defined a priori, for 3 risk groups based on clinical relevance and potential utility in future studies, including sensitivity, specificity, positive predictive value, and negative predictive value.

Comparison With Existing TB Outcome Prediction Models

Of 37 TB outcome prediction models identified in a recent systematic review [5], only 1 could be externally validated. Definitions of predictors were matched to the original study (Supplementary Materials 7), and missing data were imputed and summarized as described above. Two methods of external validation were evaluated: original model coefficients were applied to our study cohort and coefficients were updated by fitting a new model with each of the included predictors [25]. Discrimination and calibration were estimated with c-statistics and calibration plots, respectively.

Added Value Analysis

An a priori decision was made to evaluate the added value of HIV-related disease severity and NAT2 acetylator status, given these data are not routinely collected but may be important for TB treatment outcome. NAT2 acetylator status (fast, intermediate, and slow) [26] and 3 HIV-related disease severity characteristics (CD4 T-cell count <200 cells/mm³, plasma HIV-1 RNA ≥200 copies/mL, and antiretroviral therapy [ART] experience [experienced vs naive)] were added to the final model (Supplementary Materials 2). Added value was quantified using likelihood-based measures, net-reclassification index, and integrated discrimination index [27, 28]. R version 4.9.1 was used for analysis. Code is available online at https://github.com/lspeetluk/report-tb-pm.

RESULTS

The study included 944 culture-confirmed, drug-susceptible pulmonary TB patients who started standard anti-TB therapy. Median age was 35 years (interquartile range [IQR], 25–49), 34% were female, 80% were non-White, and 19% were persons living with HIV (PLWH; Table 2).

Table 2.

Study Population Characteristics

Characteristic	No.	No. (%) or Median [Interquartile Range]
Age, years	944	35 [25–49]
Female sex	944	317 (34)
Non-White race	943	751 (80)
Education, years	943	9.0 (6.0–12.0)
Body mass index, kg/m2	944	20.2 [18.3–22.5]
Previous tuberculosis diagnosis	935	142 (15)
Cavitation on chest X ray	937	468 (50)
Smear positive	943	768 (81)
Diabetes	945	235 (25)
Human immunodeficiency virus infection	938	182 (19)
Antiretroviral therapy experienced	182	71 (39)
CD4 count, cells/mm 3	175	126 [54–274]
CD4 count <200	175	116 (66)
Viral load, copies/mL	169	32,183 [309–205,226]
Viral load ≥200	169	130 (77)
Hemoglobin	939	12.10 [10.7–13.3]
Any disease comorbiditya	944	143 (15)
Alcohol use	944
Current		426 (45)
Former		366 (39)
Never		152 (16)
Drug use	943
Current		117 (12)
Former		203 (22)
Never		623 (66)
Tobacco use	944
Current		216 (23)
Former		273 (29)
Never		455 (48)
NAT2 acetylator status	944
Rapid		78 (8)
Intermediate		382 (40)
Slow		484 (51)
Treatment outcome category	944
Cure		386 (41)
Treatment completion		367 (39)
Death		29 (3)
Treatment failure		43 (4)
Treatment incomplete		56 (6)
Not evaluated		63 (7)
Unsuccessful outcome		191 (20)

Italics indicate nonmutually exclusive groups, describing characteristics only among persons living with human immunodeficiency virus (HIV).

^aExcluding diabetes and HIV infection.

Of the 944 participants, 191 (20%) had an unsuccessful outcome. Median time to unsuccessful outcome was 107 days (IQR, 41–176) compared with median time to successful outcome of 186 days (IQR, 179–205). Overall, 914 (97%) of 944 participants had complete data for every predictor.

After bootstrapped backward selection, the most important predictors of unsuccessful treatment were hemoglobin, HIV positivity, drug use, diabetes, age, education, and tobacco use (Table 3). The model demonstrated good discrimination with a c-statistic of 0.77 (95% confidence interval [CI], .73–.80; Figure 2A) and good calibration with a near-diagonal calibration curve at predicted risks below 0.40 (Figure 2B); departure from diagonal above 0.40 is likely due to lack of very high-risk patients. The Brier score was 0.14, and Hosmer-Lemeshow goodness-of-fit test P value was 0.1. The model showed good internal validation with an optimism-corrected c-statistic of 0.75 (95% CI, .71–.78); the optimism-corrected calibration intercept and slope were –0.12 and 0.89, respectively.

Table 3.

Main results from final model^a based on boostrapped backwards selection_^b after imputation for missing values, N = 944

Characteristic	Bootstrap Inclusion Frequency (%)	Coefficient	Standard Error
Intercept	100.00	0.66	0.63
Hemoglobin	98.40	–0.18	0.05
Human immunodeficiency virus infection	97.00	0.71	0.22
Former drug use	96.80	0.50	0.24
Current drug use	96.80	1.19	0.28
Diabetes	95.60	0.65	0.21
Age group, years
25–34.9	88.60	–0.48	0.25
35–44.9	88.60	–0.71	0.27
45–54.9	88.60	–1.09	0.33
≥ 55	88.60	–0.46	0.32
Years of education	83.00	–0.06	0.02
Former smoker	82.40	0.63	0.23
Current smoker	82.40	0.56	0.25
Female sex	60.80	...	...
Body mass index	57.40	...	...
Smear positive	53.00	...	...
Non-White race	52.00	...	...
Other comorbidity	34.40	...	...
Previous tuberculosis	34.20	...	...
Former alcohol use	22.40	...	...
Current alcohol use	22.40	...	...
Chest X-ray cavitation	18.00	...	...

aFinal model is based on 70% inclusion frequency in the boostrapped backwards selection.

bBoostrapped backwards selection included 500 repetitions.

Figure 2.

A, The receiver operating characteristic (ROC) curve measures discrimination of the model, that is, how well the model can differentiate between those with and without an outcome. The grey error bars represent the 95% confidence intervals (CIs) for across the ROC curve, using 2000 stratified bootstrap samples. The area under the ROC curve, which is equivalent to the c-statistic, is 0.77 (95% CI, .73–.80). B, The calibration plot displays agreement between observed and predicted outcome probabilities across deciles of outcome risk. An ideal calibration curve has an intercept of 0 and a slope of 1 (dashed line). The apparent calibration (dotted line) is calibration of the model in the original data, and the bias-corrected line is corrected for overfitting using 200 bootstrap samples. The bias-corrected calibration intercept and slope were –0.12 and 0.89, respectively. The top of the plot displays a histogram of the distribution of predicted probabilities of unsuccessful outcome for the 944 culture-confirmed, drug-susceptible pulmonary tuberculosis patients included in the study.

Variable selection and model performance results were consistent in complete case analysis within each imputed dataset (Supplementary Materials 3) and sensitivity analyses using a spline for age (Supplementary Materials 4). Model performance was similar using a 50% inclusion threshold, model approximation, and LASSO (Supplementary Materials 5).

After applying the heuristic shrinkage factor of 0.91, predicted risks from the final model can be applied to new populations using a nomogram (Figure 3), web-based application (https://lauren-peetluk.shinyapps.io/tb_outcome_risk_calculator), or the following formula:

Figure 3.

The nomogram can be used in clinical settings to estimate individual risk of an unsuccessful tuberculosis outcome. For example, for an individual who is aged 50 years with diabetes as their only comorbidity, hemoglobin of 13 g/dL, 12 years of education, never drug use, and current tobacco use, their risk is calculated as: hemoglobin = 78 points, HIV-infection = 0 points, drug use = 0 points, diabetes = 12 points, age = 0 points, education = 7 points, and tobacco use = 10 points. Total points = 107, which equates to approximately 11% risk of an unsuccessful outcome. This is equivalent to what one would get when using the formula provided in the text: 1/(1 + exp (–Χβ*0.91)), where Χβ = 0.66 – 0.18*[13] + 0.71*[0] + 0.50*[0] + 1.19*[0] + 0.65*[1] – 0.48*[0] – 0.71*[0] – 1.09*[1] –0.46*[0 – 0.06*[12] + 0.63*[0] + 0.56*[1] = –2.28. Then risk = 1/(1 + exp(–(–2.28)*0.91) = 11%. This is also consistent with the calculation from the web app. Abbreviation: HIV, human immunodeficiency virus.

$${\displaystyle \begin{array}{l}{\displaystyle \begin{array}{lll}& & {\displaystyle \mathrm{P}\mathrm{r}\mathrm{o}\mathrm{b}\mathrm{a}\mathrm{b}\mathrm{i}\mathrm{l}\mathrm{i}\mathrm{t}\mathrm{y}(\mathrm{u}\mathrm{n}\mathrm{s}\mathrm{u}\mathrm{c}\mathrm{c}\mathrm{e}\mathrm{s}\mathrm{s}\mathrm{f}\mathrm{u}\mathrm{l}=1)=1/(1+\mathrm{e}\mathrm{x}\mathrm{p}(-\mathbf{X}\beta \ast 0.91)),}\\ & & {\displaystyle \mathrm{w}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{e}\mathbf{X}\beta =0.660.18\ast [\mathrm{h}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{g}\mathrm{l}\mathrm{o}\mathrm{b}\mathrm{i}\mathrm{n}]+0.71\ast [\mathrm{H}\mathrm{I}\mathrm{V}]+0.50\ast }\\ & & {\displaystyle [\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{d}\mathrm{r}\mathrm{u}\mathrm{g}\mathrm{u}\mathrm{s}\mathrm{e}]+1.19\ast [\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{d}\mathrm{r}\mathrm{u}\mathrm{g}\mathrm{u}\mathrm{s}\mathrm{e}]+0.65\ast [\mathrm{d}\mathrm{i}\mathrm{a}\mathrm{b}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{s}]}\\ & & {\displaystyle 0.48\ast [\mathrm{a}\mathrm{g}\mathrm{e}25--35]0.71\ast [\mathrm{a}\mathrm{g}\mathrm{e}35--45]1.99\ast \left[\mathrm{a}\mathrm{g}\mathrm{e}45--55\right]}\\ & & {\displaystyle 0.46\ast [\mathrm{a}\mathrm{g}\mathrm{e}55+]0.06\ast \left[\mathrm{y}\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{s}\mathrm{o}\mathrm{f}\mathrm{e}\mathrm{d}\mathrm{u}\mathrm{c}\mathrm{a}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\right]+}\\ & & {\displaystyle 0.63\ast \left[\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{e}\mathrm{r}\mathrm{s}\mathrm{m}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{r}\right]+0.56\ast \left[\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\mathrm{m}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{r}\right]}\end{array}}\end{array}}$$

Application of the model to inform risk-based interventions is recommended when the cost-benefit ratio of the intervention under consideration is between 1:9 and 2:3 or when the risk threshold at which intervention is considered is between 11% and 41% (Figure 4; Supplementary Materials 6). Sensitivity, specificity, positive predictive value, and negative predictive value across 3 potential risk groups are presented in Table 4.

Figure 4.

The decision curve plots the standardized net benefit (y-axis) across a variety of risk thresholds (x-axis) for 3 scenarios: intervene on all (All), intervene on none (None), or intervene based on predicted risk from the risk model (Risk Model). Standardized net benefit quantifies the total benefit (true-positive rate) minus the total harm (false-positive rate), assuming a population prevalence of unsuccessful outcome of 20% and standardized to a maximum benefit of 1 [24]. The lowest y-axis indicates the cost-benefit of intervention across risk thresholds. When an intervention has low perceived cost relative to high benefit, lower risk thresholds should be considered because the harms of unnecessary intervention are minimal compared with benefit of necessary intervention. Alternatively, as the cost-benefit of the intervention approaches 1:1, the risk threshold at which intervention should be considered increases because the costs or harms of unnecessary intervention start to balance out the benefit of necessary intervention. The 2 vertical lines bound the range (risk threshold, 11%–41%) where the lower 95% confidence interval estimate of using the risk model to inform treatment/intervention decisions has a higher standardized net benefit than treating/intervening on “All” patients and treating/intervening on “None”; however, the exact choice of the risk threshold should be selected based on cost-benefit considerations, which are intervention-specific. Use of the risk model to inform a novel treatment or intervention strategy is expected to have net benefit (true-positive rate outweighs false-positive rate, assuming outcome rate of 20%) when the cost-benefit ratio of the intervention is between 1:9 and 2:3.

Table 4.

Classification Measures for Low-, Medium-, and High-Risk Groups

Risk Group	Predicted Risk Thresholds, %	No. of Observed Unsuccessful Outcomes/No. in Group (%)	Sensitivity	Specificity	Positive Predictive Value	Negative Predictive Value
Low	<10	13/307 (4)	1.00	...	1.00	0.00
Medium	10–20	47/277 (17)	0.93	0.39	0.28	0.96
High	≥20	131/360 (36)	0.69	0.69	0.36	0.90

Classification measures are calculated considering true-positives as individuals who experienced an unsuccessful outcome at or above the selected risk group, true-negatives are individuals who experienced a successful outcome below the selected risk group, false-positives are individuals who experienced a successful outcome but are at or above the selected risk group, and false-negatives are individuals who experienced an unsuccessful outcome but are below the selected risk group. The intention of this table is to provide a range of estimates that can serve as clinical cut-points in future studies or clinical practice. For example, we could target novel adherence technologies to individuals in the high-risk group (n = 360), which comprises 36% of the population but 69% of all unsuccessful outcomes, or the medium- and high-risk groups, which comprise 53% of the population but 93% of all unsuccessful outcomes. More details on how to consider various risk thresholds from decision curve analysis are detailed in Supplementary Figure 6.

The prediction model developed in this study performed well compared with external validation of the Costa-Veiga model, which had, at best, a c-statistic of 0.68 (95% CI, .64–.71) [29] (Supplementary Materials 7). Of HIV-related and NAT2 acetylator status variables, only ART experience added notable value to the final prediction model with a net reclassification index of 0.24 and approximately 3% new information gained (Table 5).

Table 5.

Added Value of Human Immunodeficiency Virus–Related Disease Severity and NAT2 Acetylator Status

Metric	Antiretroviral Therapy Experienced	CD4 <200	Viral Load ≥200	NAT2
LR test P value	.03	.81	.47	.88
Net reclassification index	0.24	<0.01	0.16	0.13
Integrated discrimination improvement	0.01	<0.01	<0.01	<0.01
Fraction of new information based on comparison of model LRs	0.03	<0.01	<0.01	<0.01
Fraction of new information based on ratio of the variances explained by the models	0.03	<0.01	<0.01	<0.01
Fraction of new information based on variance of predicted risk to the sum of the variance of predicted risk and the average risk	0.03	<0.01	<0.01	<0.01

Abbreviation: LR, likelihood ratio.

DISCUSSION

In this analysis, we describe the development and internal validation of a prediction model for unsuccessful outcome in a prospective cohort of Brazilian patients with culture-confirmed, drug-susceptible pulmonary TB. The prediction model combined 7 variables, including demographics, clinical characteristics, and a single laboratory parameter, all of which are widely available in clinical settings at the time of TB treatment initiation. Results were robust to different methods of handling nonlinearity of age and missing data and were consistent across several variable selection techniques. Individual risk from the final model can be easily calculated in clinical settings with the provided nomogram, risk formula, or online calculator. The model performed well compared with an existing prediction model [29], with improved discrimination and calibration. Internal bootstrap validation indicated slight overfitting of the model, but external validation is necessary prior to model implementation in any new setting [16].

In the model, hemoglobin, HIV infection, drug use, and diabetes were the strongest predictors of unsuccessful outcome. These factors have been consistently reported as associated with unsuccessful TB outcome. Anemia is linked to worse prognosis and elevated mortality following TB diagnosis [30]. It has also been suggested as an alternative to CD4 T-cell counts in TB–HIV-coinfected populations, given its low cost, wide availability, and possibly similar predictive value [31].

There are clear clinical implications of TB–HIV coinfection, including the effect of HIV infection on TB outcomes. In Brazil, 11% of TB cases were coinfected with HIV, yet PLWH accounted for 28% of TB deaths [1]. Factors such as poverty, undernutrition, poor access to healthcare services, and crowded living conditions additionally interact with both TB and HIV infection to fuel worse treatment outcomes for both, and concomitant treatment of TB and HIV is difficult due to drug–drug interactions and increased pill burdens [32]. Despite these complexities, the only factor related to HIV infection that added minor predictive value in this cohort was ART experience, but the power to detect differences in the models may have been limited by sample size.

Research has additionally suggested a synergistic relationship between TB and diabetes: diabetes increases risk of TB and impacts TB treatment outcomes [33]. The definition of diabetes used in this study was based on self-reported history of diabetes and baseline hyperglycemia (glycated hemoglobin ≥6.5), which may have influenced observed results. Several recent studies have suggested that TB causes dysregulation in blood glucose levels, which can manifest as transient prediabetes or frank diabetes. Although this resolves with treatment in some patients, it may still confer increased risk of unsuccessful TB outcomes [34].

The impact of behavioral factors on TB treatment outcomes have long been explored, such as drug, alcohol, and tobacco use. Most studies suggest that drug use and alcohol use contribute to poor adherence, treatment discontinuation, and loss to follow-up [35]. However, mechanistic effects, such as compromising the immune system, may also play a role in treatment outcomes [36]. Because a composite end point was used, we could not assess whether drug use affects TB treatment outcomes by way of poor retention in care or due to biologic reasons; regardless, it was an important predictor of TB treatment outcome.

We found no association between NAT2 acetylator status and TB treatment outcome. Previous research suggests that slow acetylators have increased risk of drug-induced hepatotoxicity, whereas fast acetylators may not achieve target bactericidal activity, increasing risk for therapeutic failure or relapse [7, 26]. However, most studies of acetylator status have primarily focused on toxicity end points rather than end of treatment outcomes, and more research in this area is needed. Participants with treatment-associated toxicities may have undergone dose alterations without complete regimen modification or treatment failure and would not have been captured as unsuccessful outcomes in this study.

There were limitations to this study. First, the model used a composite end point. This reflects the commonly used TB treatment outcome definition recommended by the WHO and provides a general risk estimate for unsuccessful outcome. However, there may be heterogeneous predictors for biologic (death, treatment failure, regimen switch) and behavioral outcomes (incomplete treatment, not evaluated), and analytic methods that account for competing risks or multiple outcomes should be considered in future studies [37]. Second, rather than using Rubin’s rules for multiple imputation, data were averaged across imputations into a single dataset. There is debate about how to develop and validate prediction models with multiply imputed data; some recommended approaches suggest using stacked datasets or weighting methods, which were not explored in this study due to complexities in using them together with bootstrapped backward selection [21, 38]. However, given low amounts of missing data and consistency of the current approach with each individually imputed dataset and in complete case analysis, there is confidence in the current approach. Third, some of the variables included in the model may not be available in all settings, such as surveillance data, which do not typically include laboratory parameters. Fourth, only 1 existing prediction model could be externally validated with our data, though several others may apply to this setting. Fifth, external validation of the developed prediction model was not performed due to lack of data for a validation population; however, we conducted comprehensive internal validation with the bootstrap [16]. Regardless, it is not yet clear how generalizable the model will be in other settings. Notably, RePORT-Brazil belongs to the RePORT international consortia (reportinternational.org), including sites in India, South Africa, China, Indonesia, and the Philippines. All sites are expected to begin data harmonization efforts in the coming years, which will provide opportunities for future external validation [9].

Strengths of the study include that RePORT-Brazil is a moderately sized prospective cohort study of drug-susceptible TB patients in Brazil who are representative of cases throughout Brazil [2]. All variables included are easily collected and should be readily known at the time of TB treatment initiation. The steps of prediction model development adhered to best practice guidelines for developing prognostic models, and sensitivity analyses confirmed the robustness of results [4, 11]. The bootstrapped backward selection strategy is a preferred alternative to univariable stepwise modeling approaches [17, 18]. The proportion of times a variable was retained in the model provides information about the importance of that predictor; variables that were included more frequently likely had a stronger independent association with the outcome than variables included less frequently. Performance measures were corrected for overfitting using bootstrapped internal validation, and final model coefficients were multiplied using a heuristic shrinkage factor, which both aimed to correct for overfitting and improve performance in future datasets [16, 25]. Additionally, the model performed favorably compared with an existing prediction model with good discrimination, calibration, and performance.

Regarding implementation, this model may be useful to target interventions or modified treatment regimens, such as those of shorter duration but higher cost, to individuals at high risk of unsuccessful outcome. The practicality and acceptability of risk thresholds is context-specific and should be considered relative to the expected cost-benefit of the intervention/treatment strategy. Several recent systematic reviews and meta-analyses have evaluated the impact of interventions on TB treatment outcomes, including directly observed therapy (DOT), novel digital health technologies, patient education and counseling, and financial support (eg, cash transfer programs), all of which show potential to improve treatment outcomes and could be considered in future risk-based intervention studies [39].

CONCLUSIONS

In conclusion, we detail the development and internal validation of a straightforward and easy-to-use clinical prediction model that combines 7 routinely available predictors to estimate individual risk of an unsuccessful TB treatment outcome among drug-susceptible, pulmonary TB patients on standard therapy. The model can be implemented with pen and paper, a nomogram, or using a web application. Though the model requires external validation prior to widespread implementation in any new setting, the individual risks derived from the model may be useful in future studies to allocate resources or target interventions to patients at the highest risk of unsuccessful outcomes.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Author contributions. L. S. P. conceptualized the research question, conducted the analysis, and drafted the initial manuscript. D. L., V. R., and T. R. S. provided thorough feedback on the research design and analysis interpretation, supervised the analysis, and revised successive drafts of the manuscript. P. F. R. assisted with methodology conceptualization and analysis interpretation and revised successive manuscript drafts. D. H. conducted genetic analysis and revised successive manuscript drafts. F. R., B. A., M. C. S., A. K., B. D., S. C., and M. C. F. played pivotal roles in the conceptualization of the Regional Prospective Observational Research for Tuberculosis (RePORT)-Brazil cohort, project administration, and data and funding acquisition and revised successive drafts of the manuscript. All authors approved of the final version of the manuscript.

Acknowledgments. The authors are grateful for the support and dedication of study staff at each enrollment site who made the study possible, including all members of the RePORT-Brazil consortium: Renata Spener-Gomes, Alexandra Brito de Souza, Jaquelane Silva Jesus, Aline Benjamin, Flavia Marinho Sant’Anna, Francine Peixoto Ignácio, Maria Cristina Lourenço, Adriano Gomes-Silva, Jamile G. de Oliveira, Adriana S. R. Moreira, Anna Cristina Calçada Carvalho, Elisangela C. Silva, Mayla Mello, Michael S. Rocha, Betania Nogueira, Vanessa Nascimento, Saulo Nery, Alice M. S. Andrade, Hayna Malta-Santos, Jéssica Rebouças-Silva, André M.C. Ramos, Sayonara Melo, Juan M. Cubillos-Angulo, and Laise de Moraes.

Disclaimer. The contents presented here are solely the responsibility of the authors and do not necessarily represent the official views of the National Center for Advancing Translational Sciences or the National Institutes of Health.

Financial support. This work was supported by the Departamento de Ciência e Tecnologia–Secretaria de Ciência e Tecnologia–Ministério da Saúde, Brazil (25029.000507/2013-07 to V. C. R.), National Institute of Allergy and Infectious Diseases (NIAID) at the National Institutes of Health (U01 AI069923, R01 A1120790, K01 AI131895 to P. F. R.; F31 AI152614 to L. S. P.; AI077505, AI110527, AI120790, TR002243 to D. W. H.), and the National Center for Advancing Translational Sciences (CTSA; TL1 TR002244 to L. S. P.).

Potential conflicts of interest. V. C. R., T. S., S. C., M. C. S., M. C. F., D. W. H., F. M. R., A. K., B. D., and B. B. A. report grants U01 AI069923 (CCASAnet; RePORT-Brazil supplement)/R01 AI20790 from NIAID during the conduct of the study. L. P. reports CTSA award TL1 TR002244 training grant from National Center for Advancing Translational Science (NCATS) and U01 AI069923 (CCASAnet; RePORT-Brazil supplement), R01 AI20790, and F31 AI152614 fellowship award from NIAID during the conduct of the study. P. F. R. reports a career development award (K01 AI131895) during the conduct of the study, grant R21 AI145686 for registry linkage to improve mortality ascertainment among human immunodeficiency virus (HIV) cohorts in Latin America, and a single honorarium for participation on an expert panel on HIV care in 2020 from Gilead outside the submitted work. T. R. S. reports royalties from UpToDate for chapters on management of tuberculosis outside the submitted work. V. C. R. reports individual payment for a conference presentation from the Brazilian Society of Infectious Diseases and serving on Data Safety and Monitoring Board (DSMB) for the Division of AIDS outside the submitted work. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.