Abstract
Objective
To evaluate the change in categories of risk of death by adding D-dimer to conventional mortality risk factors.
Design
Cohort study
Methods
Data on HIV-infected participants receiving standard combination antiretroviral therapy in two clinical trials (ESPRIT and SMART), who had baseline D-dimer measured, were randomly split into two equal training and a validation datasets. A multivariable survival model was built using the training dataset and included only conventional mortality risk factors measured at baseline. D-dimer was added to create the comparison model. The level of reclassification of mortality risk, for those with at least 5-years of follow-up, was then assessed by tabulating mortality risk, defined as low (≤2% predicted rate), moderate (2–5%) or high (>5%). Reclassification analyses were then repeated on the validation dataset.
Results
The analysis population at baseline had a mean age of 43 years, median CD4+ count 535cells/mm3(IQR:420-712), and 83% had ≤500 HIV RNA copies/mL. In the training dataset (n=1946, 8939 person-years), there were 83 deaths at a rate of 0.93 per-100-person-years. Addition of D-dimer to the reference model resulted in 6% or fewer (p>0.05) being correctly reassigned, either up or down, to a new risk-category, in both, training and validation datasets. The integrated discrimination improvement (IDI) in training and validation datasets was 0.60% (p=0.084) and 0.45% (p=0.168), respectively.
Conclusion
In this relatively well population, at the given risk-cutoffs, D-dimer appeared to only modestly improve the discernment of risk. Risk-reclassification provides a method for assessing the clinical utility of biomarkers in HIV cohort studies.
Keywords: HIV, risk assessment, D-dimer, HAART, biomarker, prognosis, cohort analysis, nri
INTRODUCTION
With the availability of combination antiretroviral therapy (cART), mortality in HIV-infected individuals with higher CD4+ counts is remarkably low, and is predominantly due to causes other than AIDS.[1–4] While conventional HIV-related factors such as CD4+ T-cell count (CD4+ count) and HIV RNA Viral Load (VL) continue to be of prognostic significance, they are less strongly predictive of non-AIDS as compared to AIDS events.[1, 5–9] Novel biomarkers to improve the prediction of risk in such low-intermediate risk populations are therefore needed.[10] Recent literature suggests that markers of inflammation and coagulation are potential candidates for improving risk prediction.[11, 12] These markers, including C-reactive protein, interleukin-6, and D-dimer, are reported to be higher in untreated and treated HIV-infected individuals as compared to HIV-negative individuals,[13, 14] and are associated with the risk of all-cause mortality, independently of CD4+ count and VL.[11, 15, 16] Further, although these markers have been associated with the risk of cardiovascular disease or death in the general population, some such as D-dimer, appear to show a stronger association with these outcomes in HIV infected individuals.[11]
While a statistically significant association is an important signal in identifying relevant biomarkers, a new biomarker must meet several criteria before being deemed clinically useful, including an assessment of its ability to contribute to risk prediction beyond conventional factors.[17–19] This has traditionally been assessed through calculation of the area under the receiver-operating characteristic curve (AUROC) (or equivalently, Harrell’s concordance(C)-statistic), in addition to generalized R2 statistic. However the AUROC (C-statistic) method has been criticised for being insensitive to clinically relevant improvements in prediction models.[18, 20–23] The concept of net reclassification improvement (NRI), which measures the extent to which persons with and without events are appropriately reclassified into clinically accepted higher or lower risk categories following the addition of a new marker, has been proposed.[22, 24] The NRI provides a method of quantifying the enhancement in clinically useful risk estimation when a novel marker is added to a standard risk prediction model. This new approach has now been widely adopted in cardiovascular and cancer literature [25–27] and is being increasingly used in other areas of research.[28]
In this study, we evaluated the potential role of D-dimer in improving 5-year risk-prediction of death in a cohort of treated HIV-infected patients, using the NRI approach. The paper also aims to illustrate the application and interpretation of the NRI approach to the HIV/AIDS research community. We use data from the non-intervention arms of two international clinical trials, the Evaluation of Subcutaneous Proleukin in a Randomized International Trial (ESPRIT) and the Strategic Management of antiretroviral therapy (SMART) studies, which constitute a large heterogeneous group of treated HIV-infected patients with high baseline CD4+ counts, with regular assessments, low rates of loss to follow-up and well documented and validated endpoints.
METHODS
Study population
The design, methods, and results of ESPRIT and SMART have been reported.[29–32] Briefly, in ESPRIT, a total of 4111 participants with a CD4+ cell count of 300 cells/mm3 or more were randomized to interleukin-2 plus continuous cART or to continuous cART alone (control arm). In SMART, a total of 5472 participants with a CD4+ cell count of more than 350 cells/ mm3 were randomized to either CD4+ cell count guided episodic use of cART [Drug Conservation group] or to continuous use of cART [Viral Suppression (VS) group] respectively. For this analysis, we included participants in the control and VS arms of the two studies, respectively, who were receiving cART at baseline and had baseline plasma samples available for measuring D-dimer. Participants in each study were asked to consent to storing blood for future research, and only samples from consenting participants were used. Both studies, including the consent for stored specimens, were approved by the institutional review board at each participating site and at the University of Minnesota.
Measurement of D-dimer
Plasma D-dimer has high biologic and laboratory reproducibility.[33] In SMART participants, it was measured by the Laboratory for Clinical Biochemistry Research at the University of Vermont (Burlington), with immunoturbidometric methods on the Sta-R analyzer, Liatest D-DI (Diagnostica Stago, Parsippany, NJ). For the ESPRIT study, it was measured by the Clinical Serviced Program, SAIC Frederick (Frederick, MD) using an enzyme-linked fluorescent assay on a VIDAS instrument (bioMerieux Inc., Durham, North Carolina, USA). Lower limit of detection was 0.01 μg/mL for the former and 0.045 μg/mL for the latter. In a comparison of 20 samples, there was very good agreement between the two methodologies.
Other risk factors
Other baseline covariates considered in the analysis include: age (per 10 year increment), sex, race (Black/White/Other), mode of infection (Intravenous Drug User (IDU)/ other), Body Mass Index kg/m2 (BMI) (categorized as ≤18.5, 18.5–25, 25–30, and ≥30 kg/m2), receipt of blood pressure, lipid lowering and anti-diabetic medication (details on specific medication were not collected), hepatitis B (HBs Antigen positive) or C co-infection (HCV antibody positive), nadir CD4+cell count (categorized as ≤200, 200–350, and ≤350 cells/mm3), prior cardiovascular disease (CVD), prior serious hepatic event, prior AIDS, and CD4+ cell count (categorized as ≤350, 350–500, and ≥500 cells/mm3) and VL (categorized as ≤or ≥500 copies/mL), and cumulative duration of ART (months). All decisions regarding categorization were made a priori. Categorization was considered based on clinically relevant cut-offs, where known or available. For BMI, the WHO classification was used.[34] Similarly for CD4+ counts, clinically relevant cut-offs were chosen. For age, there is a well-known monotonic relationship with mortality (and also no clinical cut-offs exist). We therefore modeled age as a continuous variable. ‘Missing’ was used as a separate category for BMI, receipt of blood pressure and lipid lowering medications, hepatitis co-infections and prior hepatic events (and was not found to be associated with different risk of death in any of the variables).
Outcome
The endpoint for this study was all-cause mortality.
Statistical analysis
We randomly split the data in half stratified by region (categorized as North America, South America, Europe, Australasia, and Africa) and CD4+ cell count (categorized as ≤350, 350–500, and ≥500cells/mm3) to form a training and a validation dataset with equal probability.
Development of prediction model in training dataset
We examined generalized gamma, Weibull, log-logistic and exponential distributions for survival models. Based on the value of kappa and the Akaike’s Information Criterion (AIC), we chose to assume Weibull distribution. Although ‘baseline’ in our study does not have any true clinical relevance (it is an arbitrary point in time relating to enrollment in one of the trials), and therefore we could have also used Poisson regression for our models, we preferred to use the Weibull model over Poisson regression as it allows the hazard of mortality to change over time and had lower AIC value. Results from Poisson regression were nevertheless similar (not shown).
We developed a reference prediction model with backward elimination of standard risk variables (see ‘Other risk factors’) (cut-off P-value for inclusion in the model: 0.10) using all the follow-up information in the training dataset. The comparison model was developed by adding baseline D-dimer (categorized at quartiles) to the reference model. These models were used to estimate for each individual the 5-year predicted risk of death. We also calculated the C-statistic for each model.[35]
Risk reclassification
For this analysis, we included only those participants with complete 5-years of follow-up or those who died within 5 years, as these methods are contingent on knowledge of end-point status and do not allow for censored data.[22] Clinically relevant risk categories over which to examine reclassification are required for this analysis.[22–24] There are no established clinically relevant 5-year risk strata for the risk of death in treated HIV disease. We a priori decided to categorize the 5-year risk of death as ≤2% (low risk), 2 to 5% (intermediate risk) and > 5% (high risk) in order to balance between distribution of events in each category and making clinically meaningful risk cutoffs. We then compared the assigned categories for the pair of models using cross tabulation. The following metrics were then assessed:
Model calibration- a measure of how closely the predicted probabilities of risk using the model reflect the actual observed risk- was assessed by comparing the observed proportion of events in the margins of the reclassification table to the corresponding predicted risk.[23] The reclassification calibration statistic (based on Hosmer-Lemeshow calibration statistic) which compares the observed and predicted events within each cell (with at least 20 observations) of the reclassification table was also calculated.[22, 23] A P value of > 0.05 for this statistic implies adequate calibration.
The capacity of the model to stratify the population into high or low risk categories was then assessed. A good model classifies fewer people into the intermediate risk category, and more people into high or low risk categories. Further, for the reclassification to be correct, participants who died (cases) should move to the higher risk category, and those who were alive at the end of 5-year follow-up (controls) should move to a lower risk category. This is formally assessed by calculating reclassification improvement (RI). The RI(cases) is the difference in proportions of cases who moved up and down the risk categories; similarly, RI(controls) is the difference in proportions of participamts who survived (controls) who moved down and up the risk categories. The net RI (NRI) is the sum of RI(cases) and RI(controls.[22–24]
The Integrated Discrimination Improvement (IDI), which measures the ability of the new biomarker to improve the discrimination between cases and controls.[22, 24] IDI, unlike the AUROC, takes into account the magnitude of difference in predicted probabilities of death for cases and controls for each model.[22, 24] Thus, the larger the IDI, the better is the ability of the marker to improve discrimination.
Model validation
The models developed in the training dataset were used to predict 5-year risk of death in the validation dataset. Risk reclassification analysis was then repeated on the validation dataset.
Sensitivity analysis
We conducted the following sensitivity analysis: (i) For the sake of comparison, the metrics described above were also calculated for an established risk factor (age) in both, training and validation datasets. (ii) Re-classification methods which can account for censored data have recently been proposed.[36] We used these methods on all participants, regardless of duration of follow-up, to NRI for D-dimer.
Data were analyzed using STATA (StataCorp, College Station, Texas, USA) version 10 and SAS/STAT software, Version 9.2 of the SAS system for Windows.
RESULTS
Participant characteristics and follow-up
There were a total of 1946 participants in the training and 1935 participants in the validation datasets. The median follow-up times in training and validation datasets were 4.7 years (Inter-quartile range (IQR): 2.2–6.6; 8939 person-years) and 4.5 years (IQR: 2.2–6.6; 8835 person-years), respectively. Table-1 describes baseline characteristics of patients included in each dataset. In the training dataset, mean age at baseline was 43 years, 77% were male, median CD4+ count was 535cells/mm3, 83% had VL ≤500copies/mL and the median D-dimer (IQR) was 0.23μg/mL (0.15–0.36). The patients in the validation dataset had similar characteristics (Table-1).
Table-1.
Baseline Characteristics
| Characteristics | Training dataset N(%) | Validation dataset N(%) |
|---|---|---|
|
| ||
| No. of participants | 1946 | 1935 |
|
| ||
| Study | ||
| ESPRIT | 898(46.2) | 879(45.4) |
| SMART | 1048(53.8) | 1056(54.6) |
|
| ||
| Region | ||
| North America | 789(40.5) | 785(40.6) |
| South America | 268(13.8) | 265(13.7) |
| Europe | 705(36.2) | 704(36.4) |
| Australasia | 173(8.9) | 172(8.9) |
| Africa | 11(0.6) | 9(0.5) |
|
| ||
| Gender | ||
| Male | 1502(77.2) | 1514(78.2) |
|
| ||
| Age | ||
| Mean(SD) | 43.3(9.3) | 43.1(9.7) |
|
| ||
| Race/Ethnicity | ||
| White | 1331(68.4) | 1289(66.6) |
| Black | 351(18) | 383(19.8) |
| Other/Unknown | 264(13.6) | 263(13.6) |
|
| ||
| Body Mass Index kg/m2 | ||
| <=18.5 | 47(2.4) | 39(2) |
| >18.5–<=25 | 1075(55.2) | 1079(55.8) |
| >25–<=30 | 595(30.6) | 598(30.9) |
| >30 | 195(10) | 188(9.7) |
| Missing | 34(1.8) | 31(1.6) |
| Median (IQR) | 24(22–27) | 24(22–27) |
|
| ||
| CD4 count cells/mm3 | ||
| Median (IQR) | 535(420–712) | 532(416–715) |
| <350 | 176(9) | 173(9) |
| 350–500 | 680(35) | 675(35) |
| >500 | 1090(56) | 1087(56.2) |
|
| ||
| Nadir CD4 count cells/mm3 | ||
| Median(IQR) | 213(105–318) | 216(112–320) |
| <=200 | 916(47.1) | 902(46.6) |
| 200–350 | 652(33.5) | 668(34.5) |
| >350 | 378(19.4) | 365(19) |
|
| ||
| HIV RNA copies/mL | ||
| ≤500 | 1608(82.6) | 1604(83) |
| >500 | 338(17.4) | 331(17) |
|
| ||
| Prior AIDS at baseline | 508(26.1) | 527(27.2) |
|
| ||
| Prior CVD | ||
| Yes | 54(2.8) | 65(3.4) |
| No | 1888(97) | 1861(96.2) |
| Unknown | 4 (0.2) | 9(0.5) |
|
| ||
| Prior serious hepatic event | ||
| Yes | 24(1.2) | 12(0.6) |
| No | 1797(92.3) | 1797 (92.8) |
| Unknown | 125(6.4) | 126(6.5) |
|
| ||
| ART duration in months | ||
| Mean (SD) | 74(43) | 72.3(43.4) |
|
| ||
| ART class at baseline: | ||
| NNRTI | 996(51.2) | 928(48) |
| PI | 861(44.2) | 897(46.4) |
|
| ||
| Likely Infection mode | ||
| IDU | 192(9.9) | 187(9.7) |
| Other | 1754(90.1) | 1748(90.3) |
| Hepatitis B positive | ||
| Yes | 65(3.3) | 91(4.7) |
| No | 1790(92) | 1752(90.5) |
| Unknown | 91(4.7) | 94(4.9) |
|
| ||
| Hepatitis C positive | ||
| Yes | 262(13.5) | 267(13.8) |
| No | 1561(80.2) | 1562(80.6) |
| Unknown | 123(6.3) | 108(5.6) |
|
| ||
| Receipt of BP lowering drugs | ||
| Yes | 227(11.7) | 242(12.5) |
| No | 1629(83.7) | 1608(83.0) |
| Unknown | 90(4.6) | 87(4.5) |
|
| ||
| Receipt of lipid lowering drugs | ||
| Yes | 273(14) | 284(14.7) |
| No | 1583(81.4) | 1566(80.9) |
| Unknown | 90(4.6) | 87(4.5) |
|
| ||
| Receipt of anti-diabetic drugs | ||
| Yes | 93(4.8) | 99(5.1) |
| No | 1849(95) | 1829(94.4) |
| Unknown | 4(0.2) | 9(0.5) |
|
| ||
| Hemoglobin g/dl | ||
| Median (IQR) | 14.5(13.4–15.4) | 14.6(13.5–15.5) |
| Missing | 605(31) | 645(33) |
|
| ||
| D-dimer μg/mL | ||
| Median (IQR) | 0.23(0.15–0.36) | 0.22(0.15–0.35) |
NOTE: ART= Antiretroviral therapy, BP= Blood Pressure, CVD= cardiovascular disease, IDU= Intravenous drug use, IQR= Inter-quartile range, NNRTI= Non-Nucleoside Reverse Transcriptase Inhibitors, PI= Protease Inhibitors, SD= Standard Deviation. Hepatitis B infection was defined as HBs Antigen positivity and Hepatitis C (HCV) infection was defined as HCV Antibody positivity.
In the training dataset, there were 83 deaths (incidence rate: 0.93(95% CI: 0.75–1.15) per 100 person-years of follow-up) of which 6 (7%) were due to AIDS. In the validation dataset, there were 60 deaths (incidence rate: 0.70 (95% CI: 0.52–0.87) per 100 person-years of follow-up) of which 4 (6.5%) were due to AIDS Figure-1 illustrates distribution of causes of death in each cohort. The difference in risk between training and validation cohorts was not statistically significant.
Figure-1.
Distribution of causes of death in training and validation datasets
Distribution of causes of death in training (n=83) and validation (n=60) cohorts. Note: CVD= cardiovascular diseases.
Risk-prediction model
Variables identified to be independently associated with risk of death in the reference model included age, VL, exposure category, prior CVD and CD4+ count (Table-2). Addition of D-dimer was found to be significantly associated with the risk of mortality (Hazard ratio (HR) for 4th quartile vs. 1st quartile: 3.99 (1.40–11.36), P for trend=0.009). Co-efficients of most other variables decreased slightly (<10%) following addition of D-dimer (Table-2). The AUROCs in the training dataset for the 5-year risk of death for the models without and with D-dimer were 0.71 and 0.73, respectively. In the validation dataset, the AUROCs were 0.76 and 0.78, respectively.
Table-2.
Predictors of death in the training dataset**, without and with D-dimer
| N | Reference model without D-dimer | Comparison model with D-Dimer | |||
|---|---|---|---|---|---|
|
| |||||
| Covariate | HR ( 95% Conf. Interval) | P | HR ( 95% Conf. Interval) | P | |
| Age | |||||
| per 10 year increase | 1946 | 1.83 (1.46 to 2.30) | <0.001 | 1.72(1.37 to 2.17) | <0.001 |
|
| |||||
| HIV RNA copies/mL | |||||
| <=500* | 1608 | Reference | Reference | ||
| >500 | 338 | 2.38 (1.48 to 3.84) | <0.001 | 2.26 (1.40 to 3.65) | 0.001 |
|
| |||||
| Exposure Category | |||||
| IDU | 1754 | 3.42 (2.06 to 5.70) | <0.001 | 3.19 (1.91 to 5.32) | <0.001 |
| Other* | 192 | Reference | Reference | ||
|
| |||||
| BMI kg/m2 | |||||
| <=18.5 | 47 | 3.00 (1.21 to 7.47) | 0.018 | 2.96 (1.17 to 7.24) | 0.020 |
| >18.5–<=25* | 1075 | Reference | Reference | ||
| >25–<=30 | 595 | 0.80 (0.48 to 1.32) | 0.378 | 0.83 (0.50 to 1.38) | 0.474 |
| >30 | 195 | 1.08 (0.49 to 2.41) | 0.846 | 1.09(0.49 to 2.44) | 0.833 |
| missing | 34 | 0.34 (0.05 to 2.50) | 0.291 | 0.32 (0.04 to 2.34) | 0.262 |
|
| |||||
| Prior CVD | |||||
| No* | 1892 | Reference | Reference | ||
| Yes | 54 | 2.50 (1.07 to 5.86) | 0 .035 | 2.46 (1.05 to 5.78) | 0.039 |
|
| |||||
| CD4 count cells/mm3 | |||||
| <350 | 176 | 1.95 (1.08 to 3.52) | 0.027 | 1.77 (0.97 to 3.21) | 0.061 |
| 350–500 | 680 | 1.11 (0.67 to 1.83) | 0.680 | 1.02 (0.62 to 1.68) | 0.942 |
| >500 | 1090 | Reference | Reference | ||
| P for trend | 0.051 | 0.113 | |||
|
| |||||
| Quartiles of D-Dimer μg/mL | |||||
| <0.15* | 520 | Reference | |||
| 0.15–0.22 | 419 | 2.89 (0.96 to 8.69) | 0.059 | ||
| 0.22– 0.35 | 485 | 2.92 (1.00 to 8.48) | 0.049 | ||
| >0.35 | 522 | 3.99 (1.40 to 11.36) | 0.010 | ||
| P for trend | 0.009 | ||||
|
| |||||
| AUROC *** | 0.71 | 0.73 | |||
NOTE: BMI=Body Mass Index, CVD= cardiovascular disease, HR=Hazard ratio, IDU= Intravenous drug use.
Reference category.
Models adjusted for all the variables. Models developed on 1946 participants (with 83 events) in the training dataset.
Area under receiver operating characteristic curve calculated for risk of death at 5 years.
P-value for difference in AUROC: 0.161.
Risk Reclassification in training dataset
For the risk reclassification analysis, participants with complete 5-years of follow-up (n=987 with 55 deaths) were included. Table-3 is a cross-tabulation of the predicted risk (categorized as low, intermediate and high) in the training dataset from the models without and with D-dimer. The margins of the table can be first examined for calibration.[23] For the reference model without D-dimer, the proportions of observed events (the observed risk) within each risk category lies within the bounds of the risk category (observed risk, category bounds: ≤2.0%, 2.0%; 4.5%, >2.0–≤5%; 11.3%, >5.0%) (Table-3). Similarly, the observed risk falls within the range of the risk categories predicted by the comparison model with D-dimer (observed risk, category bounds: 1.9%, ≤2.0%; 4.0%, >2.0–≤5.0%; 12%, >5.0%). The P-value for reclassification-calibration statistic for the models without and with D-dimer was 0.079 and 0.226, respectively. Thus, both models calibrated well.
Table-3.
Risk-Reclassification from the reference model without D-dimer to the comparison model with D-dimer for risk of death in 5 years in the Training dataset.
| Comparison model with D-dimer* | ||||
|---|---|---|---|---|
|
| ||||
| Reference model without D-dimer* | Low risk (≤2%) | Moderate risk (>2 -≤5%) | High risk (>5%) | Total |
|
| ||||
| Low risk (≤2%) | ||||
| Persons included, n (%) | 247(25) | 51(5.2) | 0(0) | 298(30.2) |
| Cases, n (%)** | 5(9.1) | 1(1.8) | 0(0) | 6(10.9) |
| Observed risk, % | 2.0 | 1.9 | - | 2.0 |
|
| ||||
| Moderate risk (>2 – ≤5%) | ||||
| Persons included, n (%) | 60(6.1) | 324(32.8) | 39(4) | 423(42.9) |
| Cases, (%)** | 1(1.8) | 13(23.6) | 5(9.1) | 19(34.5) |
| Observed risk, % | 1.7 | 4.0 | 12.8 | 4.5 |
|
| ||||
| High risk (>5%) | ||||
| Persons included, n (%) | 7(0.7) | 24(2.4) | 235(23.8) | 266(26.9) |
| Cases, (%)** | 0(0) | 2(3.6) | 28(50.9) | 30(54.5) |
| Observed risk, % | 0.0 | 8.0 | 12.0 | 11.3 |
|
| ||||
| Total | ||||
| Persons included, n (%) | 314(31.8) | 399(40.4) | 274(27.8) | 987(100) |
| Cases, n (%)** | 6(10.9) | 16(29.1) | 33(60) | 55(100) |
| Observed risk, % | 1.9 | 4.0 | 12.0 | |
NOTE: Dark-gray shading indicates increase in risk category, and light-gray shading indicates decrease in risk category. Total reclassified= 181 (18%).
Models adjusted for age, HIV RNA, CD4+ count, exposure category, Body Mass Index, and prior cardiovascular disease.
Participants who died within 5 years of follow-up.
We next examined the risk-stratification. The reference model without D-dimer puts 423 (42.9%) participants in the intermediate risk-category (Table-3), while the comparison model with D-dimer classifies 399(40.4%) participants in the intermediate risk-category (Table-3). Thus, the addition of information about D-dimer allows us to classify an additional 2.5% of individuals to the high- or low- risk category.
Overall, 181 (18.0%) participants were reclassified to a different risk category in the comparison model including D-dimer. As seen in Table-3, 6 of 55 (11.0%) cases (those who died) moved to higher risk category, whereas 3 of 55 (5.5%) moved to lower risk category, giving a reclassification improvement (RI) for cases as 5.5% (i.e. 11.0%-5.5%, p= 0.317). Similarly, 88 of 932 (9.4%) controls (alive persons) moved to the lower risk category and 84 of 932 (9.0%) controls moved to higher risk category, giving RI for controls as 0.4% (p=0.760). The Net RI (RI for cases + RI for controls) was 6.0% (p=0.296). This means that compared to controls, participants who died were almost 6% more likely to be classified to a higher risk category.
Finally, the IDI was 0.60% (p=0.084), suggesting only modest (non-significant) improvement in the discrimination of the model with the addition of D-dimer.
Risk Reclassification in validation dataset
In the validation dataset, reclassification was assessed on 944 participants (with complete 5-years of follow-up) experiencing 48 deaths. Table-4 is a cross-tabulation of the predicted risk in the validation dataset from the reference model without and the comparison model with D-dimer. Both models calibrated well. The P-values for reclassification-calibration statistic for the models without and with D-dimer were 0.251 and 0.110, respectively. The model with D-dimer, compared to the model without D-dimer, classified about 3.0% fewer individuals to the intermediate risk category.
Table-4.
Risk-Reclassification from the reference model without D-dimer to the comparison model with D-dimer for risk of death in 5 years in the Validation dataset.
| Comparison model with D-dimer* | ||||
|---|---|---|---|---|
|
| ||||
| Reference model without D- dimer* | Low risk (≤2%) | Moderate risk (>2 -≤5%) | High risk (>5%) | Total |
|
| ||||
| Low risk (≤2%) | ||||
| Persons included, n (%) | 269(28.5) | 44(4.7) | 0(0) | 313(33.2) |
| Cases, n (%)** | 2(4.2) | 0(0) | 0(0) | 2(4.2) |
| Observed risk, % | 0.7 | 0.0 | - | 0.6 |
| Moderate risk (>2 -≤5%) | ||||
|
| ||||
| Persons included, n (%) | 50(5.3) | 284(30.1) | 39(4.1) | 373(39.5) |
| Cases, (%)** | 2(4.2) | 11(22.9) | 2(4.2) | 15(31.2) |
| Observed risk, % | 4.0 | 3.9 | 5.1 | 4.0 |
|
| ||||
| High risk (>5%) | ||||
| Persons included, n (%) | 8(0.9) | 16(1.7) | 234(24.8) | 258(27.3) |
| Cases, (%)** | 0(0) | 0(0) | 31(64.6) | 31(64.6) |
| Observed risk, % | 0.0 | 0.0 | 13.2 | 12.0 |
|
| ||||
| Total | ||||
| Persons included, n (%) | 327(34.6) | 344(36.4) | 273(28.9) | 944(100) |
| Cases, n (%)** | 4(8.3) | 11(22.9) | 33(68.7) | 48(100) |
| Observed risk, % | 1.2 | 3.2 | 12.1 | |
NOTE: Dark-gray shading indicates increase in risk category, and light-gray shading indicates decrease in risk category. Total reclassified= 157 (17%).
Models developed in training dataset and adjusted for age, HIV RNA, CD4+ count, exposure category, Body Mass Index, and prior cardiovascular disease.
Participants who died within 5 years of follow-up.
Overall, 157 (17.0%) participants were reclassified to a different risk category with the comparison model including D-dimer. An equal number of cases shifted to a higher or lower risk category, giving RI(cases)= 0% (p=0.999). Similarly, 9.0% controls moved to higher risk category and 8.0% moved to lower risk category, giving RI(controls)=-1.0%(0.467). The NRI was therefore -1.0% (p=0.818). Lastly, the IDI was 0.45% (p=0.168).
Sensitivity analysis
The results from the sensitivity analyses are as follows:
Firstly we assessed the effect of age on risk discrimination in the models without D-dimer. The NRI and IDI for age in the training dataset were 17.5% (p=0.015) and 2.3% (p=0.007), respectively. Similarly, in the validation dataset, NRI for age was 41% (p<0.001) and IDI was 4.9% (p<0.001).
Secondly we assessed the effect of including participants with censored data. We included all the eligible participants (n=1946 with 83 deaths in the training dataset, and n=1935 with 60 deaths in validation dataset) and calculated the reclassification for the models without and with D-dimer using the NRI approach for the censored data.[36] The RI(cases) in training dataset was similar at 6% and in validation dataset, it was -1.9%. The RI(controls) changed from 0.4% to 9.4% in training and from -1% to 9.3% in the validation datasets, giving the NRI of 15.4% and 7.4%, respectively. Thus, the change in NRI was largely driven by change in RI(controls), i.e. the reclassification in individuals who did not experience an event to the lower risk category. These results are likely to be more accurate and generalizable, as they include information on censored individuals up to their last follow-up. However, there is no clear consensus on the best approach for calculating P-values (or confidence intervals) using these methods.[36, 37]
DISCUSSION
Using the risk-reclassification approach, we evaluated the incremental value of D-dimer, a fibrin degradation product that reflects ongoing activation of blood coagulation and fibrinolytic systems, over the conventional factors in predicting mortality in a cohort of treated HIV-infected individuals with high baseline CD4+ counts. In the multivariable model, D-dimer was statistically significantly associated with the risk of death. More relevant to clinical care, however, is whether knowledge of D-dimer level results in reclassification of individuals to a different clinical risk-category. Our data indicate that the addition of D-dimer classified only 3% fewer individuals to the intermediate risk category. Further, a relatively small proportion of individuals were correctly assigned the new risk-category — 6% or fewer when both upward and downward risk-category movement were considered. In the sensitivity analysis, the improvement in NRI was largely driven by down-classification of individuals who did not die, without any significant change in the reclassification of individuals who experienced the event. Overall, addition of D-dimer was associated with only modest, statistically non-significant, improvement in the reclassification and discrimination capacity of the prognostic models.
The study has several strengths. Firstly, it was prospective in nature and incorporated a large heterogeneous group of HIV-infected individuals receiving cART in diverse settings across the globe. Also, the clinical endpoints were well documented and validated and the D-dimer was measured at a central laboratory in each cohort, thus reducing the likelihood of measurement error. Second, our methodological approach was able to evaluate the clinical utility of a new marker (D-dimer) by focusing on the key purpose of the risk-prediction model, which is to accurately stratify individuals into clinically relevant risk categories. This approach is considered as the significant advance over the traditional AUROC approach.[22, 23] Lastly, the reasonably large sample in our study allowed us to validate our findings in an independent population.
A few important points should be considered while interpreting our results and the reclassification approach in general. First, the choice of ‘clinically meaningful’ risk cut-offs and the number of categories could influence the results to the variable extent (except for IDI, which provides a measure of gain in discrimination independent of the chosen risk cut-offs).[22, 23, 36] The risk-cut-offs are ideally chosen on the basis of the severity and rate of the event, and the cost/benefit ratio of the available interventions for those at high-risk (for example, <6%, 6–20% and >20% for 10-year risk of CVD).[22, 23, 36] As there are no established clinical cut-offs for the 5-year risk of mortality in HIV-infected individuals, we arbitrarily generated these cut-offs based on the mortality rate in the cohort.
Such a classification is likely to be informative when determining an individual’s likely prognosis and alerting clinicians to those at high-risk, for example, so that they may undergo more detailed clinical evaluation. Also, knowledge of an individual’s risk status may be useful when identifying potential candidates for therapies such as statins or anti-inflammatory agents that are currently under evaluation.[38–40] Second, the incremental value of a new biomarker is ideally evaluated over a previously established risk-prediction model.[17, 22] We did not use previously established models such as those from the ART–Cohort Collaboration (ART-CC) or the Veterans Aging Cohort Study (VACS) as the population characteristics and variables collected in these studies were different to our own.[41, 42] In particular, a higher proportion of individuals in these studies had CD4+ counts <300 cells/mm3 and VL > 500 copies/mL and, in the case of ART-CC, were ART-naïve at baseline. These studies did not take account of prior CVD or other non-AIDS events. Further, we did not have all the laboratory variables that are included in the VACS index (i.e. hemoglobin, serum creatinine, platelet count and hepatic enzymes). We therefore developed our own risk-prediction model on the training cohort. It should be noted that our overall aim was not to develop a new risk-prediction model, but rather to assess if knowledge of D-dimer improves prognostic modeling, in general. Our results are, nevertheless, largely consistent with the recent VACS study evaluating the value of adding D-dimer to their model.[42]
Third, we did not calculate the power or sample size requirements since the study sample was a convenience sample (i.e. D-dimer was measured on the all available baseline plasma samples) and the methodological approach was pilot in nature. We repeated the model building on the entire dataset (therefore larger sample) which resulted in same variables remaining in the model with similar coefficients. Also, our ability to detect a significant NRI for an established risk-factor (age) suggests that our study was well powered to detect a very strong risk-factor such as age. However, it is possible that we were underpowered to detect the impact of D-dimer. Further, we were not able to evaluate changes in short-term risk prediction because of a low number of early events. Early risk may be clinically more relevant, given the recommended routine frequent clinical monitoring of HIV-infected individuals.[43] Future studies in larger cohorts may be better powered to evaluate short term risk prediction. Fourth, the population in our study was a healthier group with high baseline CD4+ count and had a low overall risk of death. The findings therefore may not be generalizable to high-risk HIV-positive populations. Also, our analysis included participants from clinical trials and therefore results may have limited generalizability to the routine clinical cohort population. Lastly, although we accounted for several important risk factors, a few important variables (such as smoking, prior renal events or cancers, other biomarkers) were not consistently available for both cohorts and were therefore not included. Similarly, hemoglobin (Hb) was missing in about 59% of SMART participants. Of note, a recent analysis on entire SMART cohort suggested that current Hb value, but not the one measured at randomization (baseline), was associated with risk of death.[44] Availability of these variables, however, is unlikely to change our overall conclusion regarding the predictive utility of D-dimer as their addition would only make the reference model better, thereby reducing the likelihood of further improvement with the addition of D-dimer.
In summary, we found that in this cohort of treated HIV-infected individuals, knowledge of baseline D-dimer only modestly improved the reclassification or discrimination in the 5-year risk of death, over conventional HIV-related factors. Our study also demonstrates the risk reclassification approach in evaluating the incremental value of a biomarker in an HIV cohort. This approach is increasingly gaining popularity because of its ability to assess more clinically relevant change in risk categories of the individuals with the additional knowledge of the biomarker. Recent studies have suggested that the knowledge of multiple bio-markers of disease patho-physiology could add to substantial prognostic information and risk stratification, as opposed to using a single bio-marker. [27]Future studies could use the risk reclassification approach in evaluating the predictive value of single or a combination of several biomarkers (for example markers of inflammation, coagulation and immune-activation- processes thought to be responsible for various non-AIDS events) for specific endpoints in HIV cohorts.
Acknowledgments
The study was funded by the National Institute of Allergy and Infectious Diseases, National Institutes of Health [grant numbers: U01AI46957 and U01AI068641 (ESPRIT); U01AI042170 and U01AI46362 (SMART)]. We would like to acknowledge the and ESPRIT and SMART participants, the ESPRIT study team (see [29] for list of investigators), the SMART study team [[31, 32]for list of investigators], the INSIGHT Executive Committee, and Adam Rupert (SAIC-Frederick) for running the D-dimer assays for the ESPRIT study. We thank Prof. James D. Neaton for his advice with the analysis; Jacqueline Neuhaus and Deborah Wentworth for help with data queries.
Role of authors
A.C.A contributed to the design of the study, performed statistical analysis and made the first draft of the manuscript. J.A., M.G.L., S.E. and D.A.C. contributed to design of the study, review of the results and critical review of the manuscript. M.G.L over sighted the project and analysis. C.S., H.C., D.D, L.H.K., and J.A.K. each contributed to the review of the results and provided critical revision to the manuscript.
References
- 1.Hoffmann CJ, Fielding KL, Johnston V, Charalambous S, Innes C, Moore RD, et al. Changing Predictors of Mortality Over Time From cART Start: Implications for Care. J Acquir Immune Defic Syndr. 2011;58:269–276. doi: 10.1097/QAI.0b013e31823219d1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Krentz HB, Kliewer G, Gill MJ. Changing mortality rates and causes of death for HIV-infected individuals living in Southern Alberta, Canada from 1984 to 2003. HIV Med. 2005;6:99–106. doi: 10.1111/j.1468-1293.2005.00271.x. [DOI] [PubMed] [Google Scholar]
- 3.Mocroft A, Reiss P, Gasiorowski J, Ledergerber B, Kowalska J, Chiesi A, et al. Serious Fatal and Nonfatal Non-AIDS-Defining Illnesses in Europe. J Acquir Immune Defic Syndr. 2010;55:262–270. doi: 10.1097/QAI.0b013e3181e9be6b. [DOI] [PubMed] [Google Scholar]
- 4.Palella FJJ, Baker RK, Moorman AC, Chmiel JS, Wood KC, Brooks JT, et al. Mortality in the Highly Active Antiretroviral Therapy Era: Changing Causes of Death and Disease in the HIV Outpatient Study. J Acquir Immune Defic Syndr. 2006;43:27–34. doi: 10.1097/01.qai.0000233310.90484.16. [DOI] [PubMed] [Google Scholar]
- 5.Achhra AC, Amin J, Law MG, Emery S, Gerstoft J, Gordin FM, et al. Immunodeficiency and the risk of serious clinical endpoints in a well studied cohort of treated HIV-infected patients. AIDS. 2010;24:1877–1886. doi: 10.1097/QAD.0b013e32833b1b26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Baker JV, Peng G, Rapkin J, Abrams DI, Silverberg MJ, MacArthur RD, et al. CD4+ count and risk of non-AIDS diseases following initial treatment for HIV infection. AIDS. 2008;22:841–848. doi: 10.1097/QAD.0b013e3282f7cb76. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Friis-Møller N, Thiébaut R, Reiss P, Weber R, D'Arminio Monforte A, De Wit S, et al. Predicting the risk of cardiovascular disease in HIV-infected patients: the Data collection on Adverse Effects of Anti-HIV Drugs Study. Eur J Cardiovasc Prev Rehabil. 2010;17:491–501. doi: 10.1097/HJR.0b013e328336a150. [DOI] [PubMed] [Google Scholar]
- 8.Lichtenstein KA, Armon C, Buchacz K, Chmiel JS, Buckner K, Tedaldi EM, et al. Low CD4+ T Cell Count Is a Risk Factor for Cardiovascular Disease Events in the HIV Outpatient Study. Clin Infect Dis. 2010;51:435–447. doi: 10.1086/655144. [DOI] [PubMed] [Google Scholar]
- 9.Reekie J, Gatell JM, Yust I, Bakowska E, Rakhmanova A, Losso M, et al. Fatal and nonfatal AIDS and non-AIDS events in HIV-1-positive individuals with high CD4 cell counts according to viral load strata. AIDS. 2011;25:2259–2268. doi: 10.1097/QAD.0b013e32834cdb4b. [DOI] [PubMed] [Google Scholar]
- 10.Justice AC, McGinnis KA, Skanderson M, Chang CC, Gibert CL, Goetz MB, et al. Towards a combined prognostic index for survival in HIV infection: the role of ‘non-HIV’ biomarkers. HIV Med. 2010;11:143–151. doi: 10.1111/j.1468-1293.2009.00757.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Neaton JD, Neuhaus J, Emery S. Soluble biomarkers and morbidity and mortality among people infected with HIV: summary of published reports from 1997 to 2010. Curr Opin HIV AIDS. 2010;5:480–490. doi: 10.1097/COH.0b013e32833ed75d. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Worm SW, Hsue P. Role of biomarkers in predicting CVD risk in the setting of HIV infection? Curr Opin HIV AIDS. 2010;5:467–472. doi: 10.1097/COH.0b013e32833f2ea6. [DOI] [PubMed] [Google Scholar]
- 13.Baker J, Ayenew W, Quick H, Hullsiek KH, Tracy R, Henry K, et al. High-Density Lipoprotein Particles and Markers of Inflammation and Thrombotic Activity in Patients with Untreated HIV Infection. J Infect Dis. 2010;201:285–292. doi: 10.1086/649560. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Neuhaus J, Jacobs J, David R, Baker Jason V, Calmy A, Duprez D, La Rosa A, et al. Markers of Inflammation, Coagulation, and Renal Function Are Elevated in Adults with HIV Infection. J Infect Dis. 2010;201:1788–1795. doi: 10.1086/652749. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Kuller LH, Tracy R, Belloso W, De Wit S, Drummond F, Lane HC, et al. Inflammatory and coagulation biomarkers and mortality in patients with HIV infection. PLoS Med. 2008;5:e203. doi: 10.1371/journal.pmed.0050203. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Tien PC, Choi AI, Zolopa AR, Benson C, Tracy R, Scherzer R, et al. Inflammation and Mortality in HIV-Infected Adults: Analysis of the FRAM Study Cohort. J Acquir Immune Defic Syndr. 2010;55:316–322. doi: 10.1097/QAI.0b013e3181e66216. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Ioannidis JPA, Tzoulaki I. What Makes a Good Predictor? The Evidence Applied to Coronary Artery Calcium Score. JAMA. 2010;303:1646–1647. doi: 10.1001/jama.2010.503. [DOI] [PubMed] [Google Scholar]
- 18.Grund B, Sabin C. Analysis of biomarker data: logs, odds ratios, and receiver operating characteristic curves. Curr Opin HIV AIDS. 2010;5:473–479. doi: 10.1097/COH.0b013e32833ed742. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Hlatky MA, Greenland P, Arnett DK, Ballantyne CM, Criqui MH, Elkind MSV, et al. Criteria for Evaluation of Novel Markers of Cardiovascular Risk. Circulation. 2009;119:2408–2416. doi: 10.1161/CIRCULATIONAHA.109.192278. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Cook NR. Use and Misuse of the Receiver Operating Characteristic Curve in Risk Prediction. Circulation. 2007;115:928–935. doi: 10.1161/CIRCULATIONAHA.106.672402. [DOI] [PubMed] [Google Scholar]
- 21.Cook NR. Statistical Evaluation of Prognostic versus Diagnostic Models: Beyond the ROC Curve. Clin Chem. 2008;54:17–23. doi: 10.1373/clinchem.2007.096529. [DOI] [PubMed] [Google Scholar]
- 22.Cook NR, Ridker PM. Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann Intern Med. 2009;150:795–802. doi: 10.7326/0003-4819-150-11-200906020-00007. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Janes H, Pepe MS, Gu W. Assessing the Value of Risk Predictions by Using Risk Stratification Tables. Ann Intern Med. 2008;149:751–760. doi: 10.7326/0003-4819-149-10-200811180-00009. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Pencina MJ, D' Agostino RB, Vasan RS. Evaluating the added predictive ability of a new marker: From area under the ROC curve to reclassification and beyond. Stat Med. 2008;27:157–172. doi: 10.1002/sim.2929. [DOI] [PubMed] [Google Scholar]
- 25.Melander O, Newton-Cheh C, Almgren P, Hedblad B, Berglund G, Engström G, et al. Novel and Conventional Biomarkers for Prediction of Incident Cardiovascular Events in the Community. JAMA. 2009;302:49–57. doi: 10.1001/jama.2009.943. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Tice JA, Cummings SR, Smith-Bindman R, Ichikawa L, Barlow WE, Kerlikowske K. Using Clinical Factors and Mammographic Breast Density to Estimate Breast Cancer Risk: Development and Validation of a New Predictive Model. Ann Intern Med. 2008;148:337–347. doi: 10.7326/0003-4819-148-5-200803040-00004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Zethelius B, Berglund L, Sundström J, Ingelsson E, Basu S, Larsson A, et al. Use of Multiple Biomarkers to Improve the Prediction of Death from Cardiovascular Causes. N Engl J Med. 2008;358:2107–2116. doi: 10.1056/NEJMoa0707064. [DOI] [PubMed] [Google Scholar]
- 28.Tzoulaki I, Liberopoulos G, Ioannidis JPA. Use of reclassification for assessment of improved prediction: an empirical evaluation. Int J Epidemiol. 2011;40:1094–1105. doi: 10.1093/ije/dyr013. [DOI] [PubMed] [Google Scholar]
- 29.Abrams D, Levy Y, Losso MH, Babiker A, Collins G, Cooper DA, et al. Interleukin-2 therapy in patients with HIV infection. N Engl J Med. 2009;361:1548–1559. doi: 10.1056/NEJMoa0903175. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 30.Emery S, Abrams DI, Cooper DA, Darbyshire JH, Lane HC, Lundgren JD, et al. The evaluation of subcutaneous proleukin (interleukin-2) in a randomized international trial: rationale, design, and methods of ESPRIT. Control Clin Trials. 2002;23:198–220. doi: 10.1016/s0197-2456(01)00179-9. [DOI] [PubMed] [Google Scholar]
- 31.El-Sadr WM, Lundgren JD, Neaton JD, Gordin F, Abrams D, Arduino RC, et al. CD4+ count-guided interruption of antiretroviral treatment. N Engl J Med. 2006;355:2283–2296. doi: 10.1056/NEJMoa062360. [DOI] [PubMed] [Google Scholar]
- 32.The SMART Study Group. Risk for Opportunistic Disease and Death after Reinitiating Continuous Antiretroviral Therapy in Patients with HIV Previously Receiving Episodic Therapy. Ann Intern Med. 2008;149:289–299. doi: 10.7326/0003-4819-149-5-200809020-00003. [DOI] [PubMed] [Google Scholar]
- 33.Sakkinen PA, Macy EM, Callas PW, Cornell ES, Hayes TE, Kuller LH, et al. Analytical and Biologic Variability in Measures of Hemostasis, Fibrinolysis, and Inflammation: Assessment and Implications for Epidemiology. Am J Epidemiol. 1999;149:261–267. doi: 10.1093/oxfordjournals.aje.a009801. [DOI] [PubMed] [Google Scholar]
- 34.World Health Organization (WHO) BMI Classification. 2012. [Google Scholar]
- 35.Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology. 1982;143:29–36. doi: 10.1148/radiology.143.1.7063747. [DOI] [PubMed] [Google Scholar]
- 36.Pencina MJ, D'Agostino RB, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30:11–21. doi: 10.1002/sim.4085. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 37.Chambless LE, Cummiskey CP, Cui G. Several methods to assess improvement in risk prediction models: extension to survival analysis. Stat Med. 2011;30:22–38. doi: 10.1002/sim.4026. [DOI] [PubMed] [Google Scholar]
- 38.Safety and Efficacy of Salsalate to Treat Endothelial Dysfunction in HIV–infected Adults. [Accessed on 01/12/2012];ClinicalTrials.gov identifier: NCT01046682. http://clinicaltrials.gov/ct2/show/NCT01046682?term=HIV+and+inflammation&rank=3.
- 39.Statin Therapy to Improve Atherosclerosis in HIV Patients. [Accessed on 01/12/2012];ClinicalTrials.gov identifier: NCT00965185. http://clinicaltrials.gov/ct2/show/NCT00965185?term=HIV+and+statins&rank=1.
- 40.Atorvastatin on Biomarkers of Inflammation, Coagulopathy, Angiogenesis & T-cells. [Accessed on 01/12/2012];ClinicalTrials.gov identifier: NCT01351025. http://clinicaltrials.gov/ct2/show/NCT01351025?term=HIV+and+statins&rank=21.
- 41.The Antiretroviral Therapy Cohort Collaboration. Prognosis of HIV-1-infected patients up to 5 years after initiation of HAART: collaborative analysis of prospective studies. AIDS. 2007;21:1185–1197. doi: 10.1097/QAD.0b013e328133f285. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42.Justice AC, Freiberg MS, Tracy R, Kuller L, Tate JP, Goetz MB, et al. Does an index composed of clinical data reflect effects of inflammation, coagulation, and monocyte activation on mortality among those aging with HIV? Clin Infect Dis. 2012;54:984–994. doi: 10.1093/cid/cir989. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 43.Thompson MA, Aberg JA, Cahn P, Montaner JSG, Rizzardini G, Telenti A, et al. Antiretroviral Treatment of Adult HIV Infection. JAMA. 2010;304:321–333. doi: 10.1001/jama.2010.1004. [DOI] [PubMed] [Google Scholar]
- 44.Mocroft A, Lifson AR, Touloumi G, Neuhaus J, Fox Z, Palfreeman A, et al. Haemoglobin and anaemia in the SMART study. Antivir Ther. 2011;16:329–337. doi: 10.3851/IMP1746. [DOI] [PMC free article] [PubMed] [Google Scholar]