Skip to main content
NCBI home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2018 Jun 1.
Published in final edited form as: J Acquir Immune Defic Syndr. 2017 Jun 1;75(2):156–163. doi: 10.1097/QAI.0000000000001335

Tuberculosis Treatment Outcomes among HIV/TB Co-Infected Children in the International Epidemiology Databases to Evaluate AIDS (IeDEA) Network

James G Carlucci 1,2, Meridith Blevins 1,3, Aaron M Kipp 1,4,5, Mary Lou Lindegren 1,6, Quy Tuan Du 7, Lorna Renner 8, Gary Reubenson 9, John Ssali 10, Marcel Yotebieng 11, Anna M Mandalakas 12, Mary-Ann Davies 13, Marie Ballif 14, Lukas Fenner 14,15,16, April C Pettit 4,17; for the International Epidemiology Databases to Evaluate AIDS (IeDEA) Network
PMCID: PMC5429189  NIHMSID: NIHMS851815  PMID: 28234689

Abstract

Introduction

Management of tuberculosis (TB) is challenging in HIV/TB co-infected children. The World Health Organization (WHO) recommends nucleic acid amplification tests for TB diagnosis, a four-drug regimen including ethambutol during intensive phase of treatment (IP), and initiation of antiretroviral therapy (ART) within eight weeks of TB diagnosis. We investigated TB treatment outcomes by diagnostic modality, IP regimen, and ART status.

Methods

We conducted a retrospective cohort study among HIV/TB co-infected children enrolled at International Epidemiology Databases to Evaluate AIDS treatment sites from 2012-2014. We modeled TB outcome using multivariable logistic regression including diagnostic modality, IP regimen, and ART status.

Results

Among 386 HIV-infected children diagnosed with TB, 20% had microbiological confirmation of TB, and 20% had unfavorable TB outcomes. During IP, 78% were treated with a four-drug regimen. Thirty-one percent were receiving ART at the time of TB diagnosis, and 32% were started on ART within eight weeks of TB diagnosis. Incidence of ART initiation within eight weeks of TB diagnosis was higher for those with favorable TB outcomes (64%) compared to those with unfavorable outcomes (40%) (p=0.04). Neither diagnostic modality (OR 1.77; 95%CI 0.86-3.65) nor IP regimen (OR 0.88; 95%CI 0.43-1.80) were associated with TB outcome.

Discussion

In this multinational study of HIV/TB co-infected children, many were not managed per WHO guidelines. Children with favorable TB outcomes initiated ART sooner than children with unfavorable outcomes. These findings highlight the importance of early ART for children with HIV/TB co-infection, and reinforce the need for implementation research to improve pediatric TB management.

Keywords: Tuberculosis, HIV, Pediatrics, Developing Countries, Treatment Outcome

Introduction

Tuberculosis (TB) remains a major cause of childhood morbidity and mortality in low- and middle-income countries (LMIC). Human Immunodeficiency Virus (HIV) has amplified the TB epidemic, as evidenced by the disproportionate impact of TB in areas with high HIV prevalence, especially Sub-Saharan Africa and Southeast Asia.1-3 Among children in 2014, there were an estimated 1 million incident TB cases, 140,000 deaths attributable to TB, and approximately 40% of TB deaths were among those with HIV/TB co-infection.1

The World Health Organization (WHO) defines TB treatment success as documented cure or completion of anti-TB therapy (ATT); unsuccessful TB treatment outcomes include death, treatment failure, default from care/loss to follow-up (LTFU), or unknown outcome.4 Compared to HIV-negative individuals, HIV/TB co-infected individuals are less likely to have successful TB treatment outcomes.1

Despite limited data on pediatric TB outcomes,3,5-11 WHO has developed guidelines for the management of HIV/TB co-infected children in LMIC.6 These recommendations include: nucleic acid amplification testing (NAAT) as the initial TB diagnostic test for children with HIV-associated TB;6,12 using a four-drug regimen including ethambutol during the intensive phase (IP; first two months) of ATT in TB endemic areas with background isoniazid resistance;6,13,14 antiretroviral therapy (ART) initiation within eight weeks of starting ATT, for ART naïve patients;6 and, for those already receiving ART at the time of TB diagnosis, to optimize ART regimens to avoid drug-drug interactions.6,15

However, it is unclear to what extent these recommendations have been implemented in pediatric ART programs in LMIC. Limited availability of diagnostics, challenges with obtaining sputum, and paucibacillary disease which may be more pronounced in HIV/TB co-infected children may result in many children being diagnosed with TB on clinical criteria alone (without microbiologic confirmation with NAAT, acid-fast bacilli [AFB] culture, and/or AFB smear).12,16-19 Despite studies demonstrating low risk for ethambutol associated ocular toxicity,13,14 some practitioners still opt to prescribe a three-drug IP regimen lacking ethambutol for preverbal children. Similarly, some providers delay ART initiation beyond the recommended eight weeks after ATT initiation to avoid immune reconstitution inflammatory syndrome (IRIS), which is uncommon among children.20,21 And, in the setting of medication stock-outs,22-24 limited availability of fixed-dose combination drug options,25 and adherence challenges with pediatric formulations of ART and ATT medications,26,27 optimizing regimens for potential ART-ATT interactions can be challenging.15,28

We set out to investigate these issues using data collected from sites participating in the International Epidemiology Databases to Evaluate AIDS (IeDEA).29 The objectives of this research are to: (1) provide recent patient-level data on the care and treatment of HIV/TB co-infected children; (2) assess whether TB treatment outcomes differ based on TB diagnostic modality, IP regimen, or ART status; and, (3) describe ART regimens and modifications made to ART regimens in the setting of ATT.

Methods

Study Design and Population

We conducted a retrospective cohort study among HIV/TB co-infected children aged 0 through 15 years. Children were enrolled from 14 participating ART programs in 11 LMIC, representing five IeDEA regions (Figure 1).29 Previously treated TB cases were included (n=8), so a patient could contribute more than one TB episode.

Figure 1.

Figure 1

Open in a new tab

HIV/TB co-infected children included in the study, by IeDEA Network region and country. Numbers in parentheses indicate the number of children contributed by each country.

Data Collection

Local TB registries were utilized to identify TB cases diagnosed at participating IeDEA sites from January 2012 through December 2014. Patient demographics, laboratory data, TB treatment regimens, and TB outcomes were collected retrospectively using a standardized electronic case report form (CRF) developed in Research Electronic Data Capture (REDCap).30 The CRF was piloted prior to implementation, and was made available in English and French. The French version was translated by an IeDEA co-investigator who is a native French speaker and engaged in HIV/TB research. Local IeDEA site investigators completed CRFs for TB cases following electronic and/or hard copy medical record review. Data entry took place from January 2012 through January 2016. Information on ART initiation dates and ART regimens were obtained from IeDEA regional HIV care and treatment data repositories for all patients with a completed CRF. Throughout the period of data collection, routine audits were made to ensure data quality. After database closure the data was further verified, and any suspected data quality issues were referred back to local sites for investigation, clarification, and revision when necessary.

Definitions

Any diagnosis of new or previously treated active TB while enrolled at one of the participating ART sites was considered a unique TB episode. Utilizing WHO definitions,4 we grouped TB treatment outcomes into favorable and unfavorable categories. Documented cure or completion of ATT were considered favorable TB treatment outcomes. Death, treatment failure, default from care/LTFU, or unknown outcome were considered unfavorable TB treatment outcomes – the primary outcome of interest. Children still on ATT at database closure or who transferred care to another facility were excluded from the analysis of TB treatment outcomes.

Our primary exposure variables of interest were diagnostic modality, IP regimen, and ART status. A microbiologic diagnosis was considered to be any diagnosis of TB that was confirmed by at least one laboratory test (AFB smear, mycobacterial culture, or NAAT). A clinical diagnosis was any TB diagnosis that was made without a positive laboratory test result; children with clinical diagnoses could have had negative diagnostic test result(s) or no diagnostic testing performed. All TB diagnoses included a decision to start ATT.

IP regimen was dichotomized by those who received the recommended four-drug IP regimen (isoniazid [H], rifampin [R], pyrazinamide [Z], ethambutol [E]; HRZE) and those who received a three-drug IP regimen lacking ethambutol (HRZ).

ART status at the time of TB diagnosis was categorized as ART naïve or previously on ART. Among those who were ART naïve at the time of TB diagnosis, we also calculated time to ART initiation from TB diagnosis by subtracting the date of ATT initiation from ART initiation date. ART regimens were classified into the following categories based on WHO recommendations:31 three nucleoside reverse transcriptase inhibitors (NRTI); two NRTI plus a non-nucleoside reverse-transcriptase inhibitor (NNRTI); two NRTI plus a boosted protease inhibitor (PI), and non-standard (“other”) regimens. ART regimen modifications made within eight weeks of TB diagnosis date were included as ART modifications made in the setting of ATT initiation.

Key covariates at TB diagnosis included age, sex, weight, CD4 count (and CD4 percentage), and IeDEA region (Asia-Pacific, Central Africa, Eastern Africa, Southern Africa, and Western Africa; Figure 1). The date of TB diagnosis was defined as the date of the first positive microbiologic test or clinical diagnosis of TB. CD4 count at TB diagnosis was defined as the first available value 180 days before or up to 30 days after the date of TB diagnosis. HIV viral loads were not routinely collected or reported at some treatment sites participating in this study. Weight-for-age Z-scores (WAZ) at TB diagnosis were calculated as a marker of nutritional status using United States Centers for Disease Control (CDC) standards.32 WHO standards are more commonly used to calculate WAZ in LMIC since the reference population from which they are derived is more diverse than the CDC reference population; however, the WHO standard only extends through 10 years of age, while the CDC standard extends through 18 years. Therefore, to maintain a consistent metric for our population of children 0-15 years of age, we elected to use the CDC standard. Regardless of the reference population, WAZ is a relative measure and lower WAZ scores indicate a greater degree of undernutrition.

Statistical Analyses

Descriptive statistics were used to summarize patient characteristics, TB diagnostics, TB treatment, TB outcomes, ART status, ART regimens, and modifications made to ART regimens in the setting of TB diagnosis and treatment. For continuous variables, median and interquartile ranges (IQR) are reported. Frequency and percentages are reported for categorical variables.

Multivariable logistic regression was used to estimate the odds ratio (OR) and 95% confidence interval (CI) of unfavorable TB treatment outcome associated with mode of diagnosis, IP regimen, and ART status while adjusting for age, sex, WAZ, CD4 count/percentage, and IeDEA region. Multiple imputation was used to account for missing data. Continuous variables were modeled as linear terms as there was no evidence for non-linearity. Acknowledging that there may be important differences between diagnostic and treatment strategies for younger children, an a priori decision was made to perform secondary regression analyses for the sub-groups of children younger than five years of age and children aged 5-15 years. The same covariates were included in main and secondary models, except CD4 percentage was used instead of CD4 count for those younger than five years. Sensitivity analyses excluding the small number of previously treated TB cases were also performed.

Among those who were ART naïve at TB diagnosis, we compared ART initiation across TB outcomes. Anticipating that LTFU during the eight weeks after TB diagnosis might make it difficult to investigate the effect of ART initiation during the first eight weeks, we plotted the cumulative incidence of ART initiation among groups with unfavorable and favorable TB treatment outcome restricted to those that were alive and in care at eight weeks after TB diagnosis, and tested the association using a log-rank test.

R version 3.2.5 (www.r-project.org) was used for all analysis and code is available online (http://biostat.mc.vanderbilt.edu/ArchivedAnalyses).

Ethical Considerations

Local institutional review board or ethics committee approval was obtained from all local study sites as well as Vanderbilt University. Procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation (institutional and national) and with the Helsinki Declaration of 1975, as revised in 2013.

Results

Patient Characteristics

There were 400 children under the age of 16 diagnosed with TB and with a TB CRF entered into REDCap during the study period. Fourteen (3.5%) were excluded; one was still on ATT and 13 had transferred to another facility. Among 386 children included, the median age was 5.7 years (IQR 2.0-9.5) and 47% were female. Median WAZ was -2.69 (IQR -3.82, -1.48; 17% missing data). Median CD4 count among those ≥ 5 years was 144 cells/mm3 (IQR 28-403 cells/mm3; 14% missing data). Median CD4 percentage among those < 5 years was 16% (IQR 8-23%; 34% missing data) (Table 1). The children were recruited from 14 ART programs in 11 LMIC (Burundi, Côte d'Ivoire, Ghana, Indonesia, Kenya, République Démocratique du Congo, Rwanda, South Africa, Tanzania, Uganda, and Vietnam), each contributing between one and 91 children (Figure 1).

Table 1.

Characteristics of HIV/TB co-infected children at time of TB diagnosis, stratified by TB treatment outcome, 2012-2014.

Favorable Outcome Unfavorable Outcome Combined
Completed ATT (n=240) Cure (n=68) All Favorable (n=308) Death (n=30) Treatment Failure (n=1) LTFU (n=33) Unknown (n=14) All Unfavorable (n=78) (N=386)
Age, median (IQR) 6 (2.1 - 9.8) 5.5 (2.7 - 9.4) 5.9 (2.4 - 9.6) 5.4 (2 - 9.8) 10.2 (10.2 - 10.2) 3.6 (1 - 6.7) 8.1 (6.6 - 9.6) 5.5 (1.1 - 8.8) 5.7 (2.0 - 9.5)
Age group, n (%)
 0 – 4.9 years 100 (42%) 29 (43%) 129 (42%) 13 (43%) 0 (0%) 19 (58%) 3 (21%) 35 (45%) 164 (43%)
 5 – 9.9 years 80 (33%) 25 (37%) 105 (34%) 9 (30%) 0 (0%) 10 (30%) 8 (57%) 27 (35%) 132 (34%)
 10 – 15 years 60 (25%) 14 (21%) 74 (24%) 8 (27%) 1 (100%) 4 (12%) 3 (21%) 16 (21%) 90 (23%)
Female, n (%) 121 (50%) 28 (41%) 149 (48%) 13 (43%) 1 (100%) 11 (33%) 7 (50%) 32 (41%) 181 (47%)
WAZ, median (IQR) -2.33 (-3.57 - -1.11) -2.91 (-3.76 - -1.98) -2.45 (-3.66 - -1.36) -4.36 (-5.13 - -2.45) -4.25 (-4.25 - -4.25) -2.64 (-3.54 - -1.81) -3.75 (-4.89 - -2.94) -3.26 (-4.84 - -2.18) -2.69 (-3.82 - -1.48)
 Missing, n (%) 45 (19%) 6 (9%) 51 (17%) 5 (17%) 0 (0%) 7 (21%) 4 (29%) 16 (21%) 67 (17%)
CD4 count*, median (IQR) 262 (70 - 665) 117 (16 - 531) 229 (54-638) 145 (63 - 324) 92 (92 - 92) 670 (367 - 890) 85 (20 - 268) 267 (81-670) 247 (60-668)
 Missing, n (%) 55 (23%) 2 (3%) 57 (19%) 8 (27%) 0 (0%) 10 (30%) 3 (21%) 21 (27%) 78 (20%)
CD4 %†, median (IQR) 12 (4 - 22) 5 (1 - 18) 10 (3 - 22) 7 (2 - 13) 11 (11 - 11) 17 (10 - 22) 7 (4 - 10) 10 (5 - 17) 10 (3 - 20)
 Missing, n (%) 75 (31%) 7 (10%) 82 (27%) 8 (27%) 0 (0%) 11 (33%) 3 (21%) 22 (28%) 104 (27%)
IeDEA region, n (%)
 Asia-Pacific 17 (7%) 53 (78%) 70 (23%) 13 (43%) 0 (0%) 1 (3%) 0 (0%) 14 (18%) 84 (21%)
 Central Africa 52 (22%) 1 (1%) 53 (17%) 3 (10%) 0 (0%) 4 (12%) 0 (0%) 7 (9%) 60 (15%)
 Eastern Africa 60 (25%) 9 (13%) 69 (22%) 10 (33%) 0 (0%) 3 (9%) 12 (86%) 25 (32%) 94 (26%)
 Southern Africa 76 (32%) 1 (1%) 77 (25%) 1 (3%) 0 (0%) 8 (24%) 2 (14%) 11 (14%) 88 (23%)
 Western Africa 35 (15%) 4 (6%) 39 (13%) 3 (10%) 1 (100%) 17 (52%) 0 (0%) 21 (27%) 60 (16%)
Open in a new tab

ATT = anti-TB therapy; IQR = interquartile range; LTFU = loss to follow-up; WAZ = weight-for-age Z-score

*

Median CD4 count among those ≥ 5 years was 144 cells/mm3 (IQR 28-403 cells/mm3; 14% missing data).

†

Median CD4 percentage among those < 5 years was 16% (IQR 8-23%; 34% missing data).

TB Diagnosis

Among the 386 HIV/TB co-infected children, 79 (20%) had microbiological confirmation of TB, while 307 (80%) were diagnosed clinically. Of the 79 with at least one positive diagnostic test, 36 (46%) had a positive AFB smear, 33 (42%) had a positive mycobacterial culture, and 48 (61%) had a positive NAAT (Table 2). In terms of diagnostic yield, 59% (229/386) of cases were tested with AFB smear, but only 16% (36/229) of those tests were positive; 32% (123/386) of cases were tested with AFB culture, and 27% (33/123) of those tests were positive; while only 22% (84/386) of cases were tested with NAAT, but 57% (48/84) of those tests were positive. Of the 307 cases diagnosed clinically, 155 (50%) had at least one test but no positive result, and 152 (50%) did not have a laboratory diagnostic test performed. Overall, 287 (74%) of cases were classified as pulmonary TB (PTB), 73 (19%) were classified as extra-pulmonary (EPTB), and 25 (6%) were classified as having both PTB and EPTB. Among those with EPTB, the most common sites were abdominal (9%) and lymphatic (7%) (Table 2).

Table 2.

Summary of TB diagnostics, sites of disease, treatment, and ART status for HIV/TB co-infected children, stratified by TB treatment outcome category.

Favorable outcome (n=308) Unfavorable outcome (n=78) Combined (N=386)
Test Result, n (%)
 Microbiologic diagnosis* 61 (20%) 18 (23%) 79 (20%)
AFB smear positive 25 (8%) 11 (14%) 36 (9%)
AFB culture positive 28 (9%) 5 (6%) 33 (9%)
NAAT positive 37 (12%) 11 (15%) 48 (12%)
 Clinical diagnosis 247 (80%) 60 (77%) 307 (80%)
Negative test result(s) 127 (41%) 28 (36%) 155 (40%)
No test performed 120 (39%) 32 (41%) 152 (39%)
Site of disease, n (%)
 Pulmonary 229 (74%) 58 (74%) 287 (74%)
 Extra-pulmonary* 60 (19%) 13 (17%) 73 (19%)
Abdominal 30 (10%) 5 (6%) 35 (9%)
CNS/Meningeal 3 (1%) 1 (1%) 4 (1%)
Lymphatic 16 (5%) 9 (12%) 25 (7%)
Miliary 4 (1%) 2 (3%) 6 (2%)
Osteoarticular 2 (1%) 0 (0%) 2 (<1%)
Pericardial 1 (<1%) 1 (1%) 2 (<1%)
Pleural 4 (1%) 3 (4%) 7 (2%)
Unknown 11 (4%) 1 (1%) 12 (3%)
 Both 18 (6%) 7 (9%) 25 (6%)
Intensive phase regimen, n (%)†
 3-drug HRZ 62 (20%) 22 (28%) 84 (22%)
 4-drug HRZE 246 (80%) 56 (72%) 302 (78%)
ART status, n (%)
 ART prior to TB diagnosis 101 (33%) 18 (23%) 119 (31%)
 ART ≤8 weeks after TB diagnosis 109 (34%) 15 (19%) 124 (32%)
 ART >8 weeks after TB diagnosis 45 (15%) 8 (10%) 53 (14%)
 No ART 25 (8%) 25 (32%) 50 (13%)
 Missing 28 (9%) 12 (15%) 40 (10%)
Open in a new tab
*

More than one can apply, thus numbers do not add up to 100%

†

A 3-drug regimen includes isoniazid (H), rifampin (R), and pyrazinamide (Z). A 4-drug regimen includes HRZ and ethambutol (E).

AFB = acid fast bacilli; NAAT = nucleic acid amplification test; ART = antiretroviral therapy

Only three cases (<1%) were found to have isoniazid or rifampin resistance, one of which was in the Asia-Pacific region, and the other two were in the Southern Africa region. These regions were also the only two regions that were routinely utilizing NAAT for TB diagnosis, and were the primary regions where gastric aspirates and induced sputum were utilized to obtain diagnostic specimens.

TB Treatment Regimens

As for IP regimens, 84 (22%) cases were treated with a non-standard 3-drug regimen lacking ethambutol, while 302 (78%) were treated with a 4-drug regimen including ethambutol (Table 2). The practice of omitting ethambutol was common in Eastern Africa, Central Africa, and Western Africa regions where such IP regimens were prescribed for 47%, 36%, and 27% of cases, respectively. IP regimens lacking ethambutol were prescribed to children less than five years of age in 32% of cases, and to children aged 5-15 years in 15% of cases.

ART Status

One hundred and nineteen (31%) of 386 HIV/TB co-infected children were receiving ART prior to their TB diagnosis. Of the 267 (69%) not on ART at the time of starting ATT, 124 (46%) initiated ART within eight weeks of starting ATT, and 53 (20%) more initiated ART more than eight weeks after starting ATT. Fifty (13%) of the HIV/TB co-infected children did not initiate ART by the end of the study period. ART status data was missing for 40 (10%) children (Table 2).

Many ART regimens were modified when starting ATT. Among the 114 patients active on ART eight weeks prior to TB diagnosis, 64% of the ART regimens were comprised of two NRTI plus a NNRTI (32% efavirenz, 32% nevirapine), 16% were comprised of two NRTI plus a boosted PI, 2% were comprised of three NRTI, and 18% were non-standard regimens. The majority of children receiving an efavirenz-based (97%), boosted PI-based (94%), or three NRTI (100%) regimen remained on the same ART regimen after TB diagnosis. Sixty percent of children on a nevirapine-based regimen remained on the same regimen after TB diagnosis, 32% switched to an efavirenz-based regimen, 3% switched to a boosted PI-based regimen, and 3% switched to three NRTI. Two percent experienced an interruption in ART after TB diagnosis. Children initiating ART within eight weeks of TB diagnosis were started on the following regimens: two NRTI plus a NNRTI (78%; 66% efavirenz, 12% nevirapine), two NRTI plus a boosted PI (8%), three NRTI (2%), and other non-standard regimens (12%) (supplemental figure accessible at: http://biostat.mc.vanderbilt.edu/MeridithBlevins/pediatric-tb-hiv-sunburst.html). Dose adjustments for ART and/or ATT also may have occurred, but dosing data was not captured.

TB Treatment Outcomes

Among the 386 HIV/TB co-infected children, 78 (20%) had unfavorable TB treatment outcomes and 308 (80%) had favorable outcomes. Among the 78 with unfavorable outcomes, 30 (38%) died, 1 (1%) had treatment failure, 33 (42%) were LTFU, and 14 (18%) had unknown outcomes. The proportion of unfavorable outcomes attributable to LTFU was 1% (1/84) in the Asia-Pacific region, 7% (4/60) in the Central Africa region, 3% (3/94) in the Eastern Africa region, 9% (8/88) in the Southern Africa region, and 28% (17/60) in the Western Africa region. Among the 308 with favorable TB treatment outcomes, 68 (22%) had documented cure, and 240 (78%) completed therapy (Table 1).

Multivariable logistic regression was performed to identify factors independently associated with an unfavorable TB treatment outcome. In the model including all 386 HIV/TB co-infected children, neither mode of diagnosis (OR 1.77; 95%CI 0.86-3.65), IP regimen (OR 0.88; 95%CI 0.43-1.80), nor ART status (OR 0.71; 95%CI 0.38-1.31) were significantly associated with TB treatment outcome. Better nutritional status/higher WAZ was protective against unfavorable TB treatment outcomes (OR 0.35; 95% CI 0.67-0.94). Children in the Asia-Pacific (OR 0.35; 95% CI 0.14-0.89) and Southern Africa (0.34; 95% CI 0.14-0.84) IeDEA regions had significantly lower odds of having an unfavorable outcome compared with the Eastern Africa region. In the subgroup analysis restricted to children younger than five years, those with microbiological confirmation of TB had higher odds of unfavorable TB outcome (OR 3.8; 95%CI 1.12-12.94) (Table 3). When previously treated TB cases were excluded (n=8), results were similar (data not shown).

Table 3.

Logistic regression models to identify factors associated with an unfavorable TB treatment outcome.

All Pediatric Patients Odds Ratio (95% CI)1 Children under 5 Odds Ratio (95% CI)2 Children aged 5-15 Odds Ratio (95% CI)3
Microbiologic diagnosis (vs. Clinical diagnosis) 1.77 (0.86, 3.65) 3.80 (1.12, 12.90) 1.45 (0.53, 4.00)
3-drug HRZ regimen (vs. 4-drug HRZE)* 0.88 (0.43, 1.80) 0.84 (0.23, 2.99) 0.91 (0.33, 2.50)
On ART at TB diagnosis (vs. not on ART) 0.71 (0.38, 1.31) 1.04 (0.40, 2.67) 0.48 (0.21, 1.14)
Age (per 1 year) 0.96 (0.89, 1.03) 0.83 (0.62, 1.13) 1.01 (0.88, 1.15)
Weight-for-age Z-score (per 1 standard deviation) 0.80 (0.67, 0.94) 0.80 (0.63, 1.02) 0.78 (0.61, 1.01)
CD4 count (per 10 cells) 1.00 (1.00, 1.01) omitted† 1.00 (0.99, 1.02)
CD4 percentage (per 1 unit) omitted† 1.00 (0.96, 1.05) omitted†
Female (vs. Male) 0.77 (0.45, 1.32) 0.93 (0.41, 2.11) 0.64 (0.30, 1.40)
Region
 Eastern Africa (reference) 1 1 1
 Asia-Pacific 0.35 (0.14, 0.89) 0.28 (0.05, 1.43) 0.41 (0.12, 1.40)
 Central Africa 0.47 (0.18, 1.23) 0.88 (0.22, 3.52) 0.23 (0.04, 1.21)
 Southern Africa 0.34 (0.14, 0.84) 0.91 (0.23, 3.59) 0.08 (0.02, 0.45)
 Western Africa 1.22 (0.57, 2.62) 2.81 (0.69, 11.53) 0.78 (0.31, 1.99)
Open in a new tab
1

There are 386 patients included in this model; 78 had an unfavorable outcome.

2

There are 164 patients <5 years old included in this model; 35 had an unfavorable outcome.

3

There are 222 patients 5-15 years old included in this model; 43 had an unfavorable outcome.

*

A 3-drug regimen includes isoniazid (H), rifampin (R), and pyrazinamide (Z). A 4-drug regimen includes HRZ and ethambutol (E).

†

CD4 count and CD4 percentage are correlated, so to save degrees of freedom only one or the other was used in each regression model. Consistent with their clinical application, CD4 percentage was used in the model for those <5 years old, while CD4 count was used for the models including children 5-15 years old.

Among HIV/TB co-infected children who were ART naïve at the time of TB diagnosis and were alive and retained in care eight weeks later, ART initiation within eight weeks of TB diagnosis was higher for those with favorable TB outcomes (64%) compared to those with unfavorable TB outcomes (40%) (p=0.04) (Figure 2).

Figure 2.

Figure 2

Open in a new tab

Cumulative incidence of antiretroviral therapy initiation within eight weeks after TB diagnosis, among HIV/TB co-infected children who were previously ART naïve and who were alive and retained in care at eight weeks after TB diagnosis.

Discussion

In this large multinational study of HIV/TB co-infected children, many were not diagnosed or treated per WHO guidelines. Eighty percent of diagnoses were clinical (without microbiologic confirmation), and NAAT use was infrequent. Nearly one-quarter of patients did not receive a four-drug IP regimen including ethambutol. Only half of ART naïve children were started on ART within eight weeks of TB diagnosis, and 13% of patients were not started on ART during the study period. Overall, 20% of HIV/TB co-infected children in this cohort had unfavorable TB treatment outcomes. Those who initiated ART earlier were more likely to have favorable TB treatment outcomes. Among those younger than five years of age, those with microbiologic confirmation of TB had higher odds of unfavorable TB treatment outcomes.

While at first the finding that microbiologic confirmation of TB is associated with unfavorable TB outcomes among the youngest patients seems counterintuitive, this might be explained in several ways. First, there is likely over-diagnosis of TB among clinical cases (false-positives) and therefore better outcomes in this group. Second, more extensive disease, and therefore higher bacterial burden, could both facilitate microbiologic confirmation of TB and be associated with poorer outcomes. Third, awaiting culture results in the absence of NAAT confirmation may have delayed TB treatment and possibly ART initiation, and therefore resulted in a greater likelihood of unfavorable TB treatment outcomes.

This study highlights an urgent need to improve diagnosis of TB among HIV/TB co-infected children in LMIC. Only 20% of children diagnosed with TB had microbiologic confirmation of disease, and only 22% of cases were tested with NAAT. Similarly, in 2012 we conducted a site-level survey of pediatric ART programs in six IeDEA regions and found that NAAT was available at about one-third of sites; and, among 146 children who developed TB, AFB smear was utilized for 52%, mycobacterial culture for 17%, and NAAT for only 8%.19 Among the many impediments to TB diagnosis in children is obtaining diagnostic respiratory specimens. Performing early morning gastric aspirates can improve diagnostic yield, but requires hospitalization, which is often not practical in LMIC. Capacity to perform sputum induction in the outpatient setting is growing, and can improve diagnostic yield in children unable to spontaneously expectorate sputum. Even when a good specimen is obtained, access to diagnostics may be limited in LMIC. Furthermore, the yield of diagnostics is low due to paucibacillary disease in children, particularly those with compromised immune systems.12,18 Strategies to improve pediatric TB diagnosis should be multifaceted, and might include: improving the accuracy of clinical diagnosis, enhancing the implementation and utilization of current and emerging NAAT platforms,1 and developing non-sputum based diagnostics.33-35

WHO makes a “strong recommendation” that a four-drug IP regimen including ethambutol be used for children from settings where the HIV prevalence is greater than 1% or there is high prevalence (as defined by national treatment programs) of isoniazid resistance, in order to minimize the risk of developing or transmitting drug-resistant TB.6,14 IP regimen did not have an impact on TB treatment outcomes in this cohort, despite about one-quarter of HIV/TB co-infected children not receiving ethambutol.

This study adds to the literature that supports early ART as an important part of TB treatment for HIV/TB co-infected persons,36-39 and as such highlights the concerning observation that only half of ART naïve patients in this cohort were started on ART within eight weeks of TB diagnosis and that 13% of patients were not started on ART during the study period. Other studies have also reported delays in ART initiation,40 and there is some evidence that failing to implement WHO guidelines is associated with substantial mortality.41 Further implementation research aimed at improving access to ART for HIV/TB co-infected children is essential.

Regional differences in outcomes are also important to acknowledge. Patients in the Asia-Pacific and Southern Africa regions were less likely to have unfavorable TB treatment outcomes. These two regions also had the highest utilization of sputum induction, gastric aspirates, and NAAT. There were especially high LTFU rates in the Western Africa region, resulting in this region having the highest proportion of unfavorable TB treatment outcomes. It is possible that those who were LTFU may have had different characteristics than those who were classified as having unfavorable outcomes for some other reason. However, due to the relatively small number of unfavorable outcomes, it would be difficult to draw conclusions from sensitivity analyses disaggregating these unfavorable outcomes. Furthermore, other reports have demonstrated significant mortality among those who are LTFU from HIV care and treatment programs.42-44 Regardless, improving access to diagnostics and interventions aimed at reducing attrition from care could lead to improved outcomes. Additionally, better understanding of the factors that contribute to heterogeneity between sites and regions could result in improved implementation of diagnostic and treatment services.

We were unable to fully assess the appropriateness of ART regimens and modifications to ART regimens in the context of TB treatment, due to the absence of data on weight-based dosing of ART and ATT. The antiretrovirals nevirapine (a NNRTI) and lopinavir/ritonavir (a boosted PI) significantly interact with the anti-TB medication rifampin and require dose adjustment or changing to an alternative antiretroviral regimen. For those who were treated with nevirapine or lopinavir/ritonavir, it would be important to know whether ART dose adjustments were made. A second limitation was incomplete data for CD4 count/percentage, HIV viral load (75% missing), WAZ, and ART status. We accounted for incomplete data in our logistic regression models using multiple imputation.

In conclusion, in this large multinational population of HIV/TB co-infected children, many were not managed per WHO guidelines. Children with favorable TB outcomes initiated ART sooner than children with unfavorable outcomes. These findings highlight the importance of early ART for children with HIV/TB co-infection, and reinforce the need for implementation research to improve pediatric TB management.

Acknowledgments

The authors would like to thank the local and regional IeDEA site staff for their assistance in data collection and quality assurance. We would also like to thank Charlotte Lewden for French translations of case report forms and responses. Sites participating in this project included: Sanglah Hospital, Udayana University, Bali, Indonesia; Children's Hospital 1, Ho Chi Minh City, Vietnam; Central University Hospital of Kamenge, Burundi; Kalembelembe Pediatric Hospital, République Démocratique du Congo; Rwanda Military Hospital, Rwanda; Moi University, Moi Teaching and Referral Hospital, Academic Model Providing Access to Healthcare (AMPATH), Eldoret, Kenya; National AIDS Control Programme, Tumbi Regional Hospital, Tanzania; National Institute for Medical Research, Mwanza Research Centre-Kisesa Clinic, Mwanza, Tanzania; Masaka Regional Hospital, Masaka, Uganda; Rahima Moosa Mother and Child Hospital, South Africa; Centre Intégré de Recherches Biocliniques d'Abidjan, Côte d'Ivoire; Centre Hospitalier Universitaire de Cocody, Côte d'Ivoire; and Korle-Bu Teaching Hospital, Accra, Ghana. The authors would also like to acknowledge the IeDEA TB working group members.

Sources of Funding: This research was supported by the National Institutes of Health (NIH) under award numbers: K08 AI104352 (Pettit), T32 HD060554-06A1 (Carlucci), U01 AI096299 (IeDEA Central Africa), U01 AI069919 (IeDEA Western Africa), U01 AI069924 (IeDEA Southern Africa), U01 AI069911 (IeDEA Eastern Africa), U01 AI069907 (IeDEA Asia-Pacific), U01 AI096186 (IeDEA Network Coordinating Center), P30 AI110527 (Tennessee Center for AIDS Research), and UL1 TR000445 (Vanderbilt Institute for Clinical and Translational Research). It has also been supported by the President's Emergency Plan for AIDS Relief (PEPFAR) through the United States Agency for International Development (USAID) under the terms of Cooperative Agreement No. AID-623-A-12-0001. The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the views of the NIH, PEPFAR, or USAID.

Footnotes

Conflicts of Interest: None declared

Previous Presentation: A preliminary version of this report was presented as a poster at the Conference on Retroviruses and Opportunistic Infections in Boston, MA on February 23, 2016; Abstract #16-1151. The manuscript is not being considered for publication elsewhere.

References

  • 1.World Health Organization. Global tuberculosis report. [Accessed June 29, 2016];2015 http://apps.who.int/iris/bitstream/10665/191102/1/9789241565059_eng.pdf?ua=1.
  • 2.Corbett EL, Watt CJ, Walker N, et al. The growing burden of tuberculosis: global trends and interactions with the HIV epidemic. Archives of internal medicine. 2003;163(9):1009–1021. doi: 10.1001/archinte.163.9.1009. [DOI] [PubMed] [Google Scholar]
  • 3.Fenner L, Brinkhof MW, Keiser O, et al. Early mortality and loss to follow-up in HIV-infected children starting antiretroviral therapy in Southern Africa. J Acquir Immune Defic Syndr. 2010;54(5):524–532. doi: 10.1097/QAI.0b013e3181e0c4cf. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 4.World Health Organization. Definitions and reporting framework for tuberculosis – 2013 revision. [Accessed July 6, 2016]; http://apps.who.int/iris/bitstream/10665/79199/1/9789241505345_eng.pdf.
  • 5.Turkova A, Chappell E, Chalermpantmetagul S, Della-Negra M, et al. Tuberculosis in HIV-infected children in Europe, Thailand and Brazil: paediatric TB:HIV EuroCoord study. Poster presented at 20th International Workshop on HIV and Hepatitis Observational Databases; April, 2016; Budapest, Hungary. [DOI] [PubMed] [Google Scholar]
  • 6.World Health Organization. Guidance for national tuberculosis programmes on the management of tuberculosis in children. [Accessed June 29, 2016];2014 http://apps.who.int/medicinedocs/documents/s21535en/s21535en.pdf. [PubMed]
  • 7.Donald PR, Maher D, Qazi S. A research agenda to promote the management of childhood tuberculosis within national tuberculosis programmes. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease. 2007;11(4):370–380. [PubMed] [Google Scholar]
  • 8.Geoghagen M, Farr JA, Hambleton I, Pierre R, Christie CD. Tuberculosis and HIV co-infections in Jamaican children. West Indian Med J. 2004;53(5):339–345. [PubMed] [Google Scholar]
  • 9.Graham SM, Gie RP, Schaaf HS, Coulter JB, Espinal MA, Beyers N. Childhood tuberculosis: clinical research needs. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease. 2004;8(5):648–657. [PubMed] [Google Scholar]
  • 10.Hicks RM, Padayatchi N, Shah NS, et al. Malnutrition associated with unfavorable outcome and death among South African MDR-TB and HIV co-infected children. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease. 2014;18(9):1074–1083. doi: 10.5588/ijtld.14.0231. [DOI] [PubMed] [Google Scholar]
  • 11.Jeena PM, Pillay P, Pillay T, Coovadia HM. Impact of HIV-1 co-infection on presentation and hospital-related mortality in children with culture proven pulmonary tuberculosis in Durban, South Africa. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease. 2002;6(8):672–678. [PubMed] [Google Scholar]
  • 12.Detjen AK, DiNardo AR, Leyden J, et al. Xpert MTB/RIF assay for the diagnosis of pulmonary tuberculosis in children: a systematic review and meta-analysis. Lancet Respir Med. 2015;3(6):451–461. doi: 10.1016/S2213-2600(15)00095-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 13.Thee S, Detjen A, Quarcoo D, Wahn U, Magdorf K. Ethambutol in paediatric tuberculosis: aspects of ethambutol serum concentration, efficacy and toxicity in children. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease. 2007;11(9):965–971. [PubMed] [Google Scholar]
  • 14.World Health Organization. Ethambutol efficacy and toxicity: literature review and recommendations for daily and intermittent dosage in children. [Accessed July 13, 2016];2006 http://apps.who.int/iris/bitstream/10665/69366/1/WHO_HTM_TB_2006.365_eng.pdf.
  • 15.Cotton MF, Rabie H, van Zyl GU. Antiretroviral therapy in children with tuberculosis: progress toward defining the issues. The Journal of infectious diseases. 2010;201(8):1113–1114. doi: 10.1086/651455. [DOI] [PubMed] [Google Scholar]
  • 16.Marais BJ, Gie RP, Hesseling AC, et al. A refined symptom-based approach to diagnose pulmonary tuberculosis in children. Pediatrics. 2006;118(5):e1350–1359. doi: 10.1542/peds.2006-0519. [DOI] [PubMed] [Google Scholar]
  • 17.Marais BJ, Graham SM, Cotton MF, Beyers N. Diagnostic and management challenges for childhood tuberculosis in the era of HIV. The Journal of infectious diseases. 2007;196(1):S76–85. doi: 10.1086/518659. [DOI] [PubMed] [Google Scholar]
  • 18.Newton SM, Brent AJ, Anderson S, Whittaker E, Kampmann B. Paediatric tuberculosis. Lancet Infect Dis. 2008;8(8):498–510. doi: 10.1016/S1473-3099(08)70182-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 19.Ballif M, Renner L, Claude Dusingize J, et al. Tuberculosis in Pediatric Antiretroviral Therapy Programs in Low- and Middle-Income Countries: Diagnosis and Screening Practices. J Pediatric Infect Dis Soc. 2015;4(1):30–38. doi: 10.1093/jpids/piu020. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 20.Link-Gelles R, Moultrie H, Sawry S, Murdoch D, Van Rie A. Tuberculosis Immune Reconstitution Inflammatory Syndrome in children initiating Antiretroviral Therapy for HIV infection: A systematic literature review. The Pediatric infectious disease journal. 2014;33(5):499–503. doi: 10.1097/INF.0000000000000142. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 21.Van Rie A, Sawry S, Link-Gelles R, et al. Paradoxical tuberculosis-associated immune reconstitution inflammatory syndrome in children. Pediatr Pulmonol. 2016;51(2):157–164. doi: 10.1002/ppul.23221. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 22.Mori AT, Owenya J. Stock-outs of antiretroviral drugs and coping strategies used to prevent changes in treatment regimens in Kinondoni District, Tanzania: a cross-sectional study. J Pharm Policy Pract. 2014;7:3. doi: 10.1186/2052-3211-7-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 23.Pasquet A, Messou E, Gabillard D, et al. Impact of drug stock-outs on death and retention to care among HIV-infected patients on combination antiretroviral therapy in Abidjan, Cote d'Ivoire. PloS one. 2010;5(10):e13414. doi: 10.1371/journal.pone.0013414. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 24.Schouten EJ, Jahn A, Ben-Smith A, et al. Antiretroviral drug supply challenges in the era of scaling up ART in Malawi. J Int AIDS Soc. 2011;14(1):S4. doi: 10.1186/1758-2652-14-S1-S4. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 25.Bouazza N, Foissac F, Fauchet F, et al. Lopinavir/ritonavir plus lamivudine and abacavir or zidovudine dose ratios for paediatric fixed-dose combinations. Antivir Ther. 2015;20(2):225–233. doi: 10.3851/IMP2876. [DOI] [PubMed] [Google Scholar]
  • 26.Howard LM, Tique JA, Gaveta S, et al. Health literacy predicts pediatric dosing accuracy for liquid zidovudine. Aids. 2014;28(7):1041–1048. doi: 10.1097/QAD.0000000000000197. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 27.Taneja R, Garcia-Prats AJ, Furin J, Maheshwari HK. Paediatric formulations of second-line anti-tuberculosis medications: challenges and considerations. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease. 2015;19(1):61–68. doi: 10.5588/ijtld.15.0435. [DOI] [PubMed] [Google Scholar]
  • 28.Vitoria M, Ford N, Doherty M, Flexner C. Simplification of antiretroviral therapy: a necessary step in the public health response to HIV/AIDS in resource-limited settings. Antivir Ther. 2014;19(3):31–37. doi: 10.3851/IMP2898. [DOI] [PubMed] [Google Scholar]
  • 29.Egger M, Ekouevi DK, Williams C, et al. Cohort Profile: the international epidemiological databases to evaluate AIDS (IeDEA) in sub-Saharan Africa. Int J Epidemiol. 2012;41(5):1256–1264. doi: 10.1093/ije/dyr080. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 30.Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377–381. doi: 10.1016/j.jbi.2008.08.010. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 31.World Health Organization. Consolidated guidelines on the use of antiretroviral drugs for treating and preventing HIV infection: recommendations for a public health approach. [Accessed July 6, 2016];2016 http://www.who.int/entity/hiv/pub/arv/arv-2016/en/index.html. [PubMed]
  • 32.Flegal KM, Cole TJ. Construction of LMS parameters for the Centers for Disease Control and Prevention 2000 growth charts. Natl Health Stat Report. 2013;(63):1–3. [PubMed] [Google Scholar]
  • 33.Mishra AK, Driessen NN, Appelmelk BJ, Besra GS. Lipoarabinomannan and related glycoconjugates: structure, biogenesis and role in Mycobacterium tuberculosis physiology and host-pathogen interaction. FEMS Microbiol Rev. 2011;35(6):1126–1157. doi: 10.1111/j.1574-6976.2011.00276.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 34.Pai M, Schito M. Tuberculosis diagnostics in 2015: landscape, priorities, needs, and prospects. The Journal of infectious diseases. 2015;211(2):S21–28. doi: 10.1093/infdis/jiu803. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 35.World Health Organization. The use of lateral flow urine lipoarabinomannan assay (LF-LAM) for the diagnosis and screening of active tuberculosis in people living with HIV. [Accessed July 1, 2016];2015 http://www.who.int/tb/publications/use-of-lf-lam-tb-hiv/en/.
  • 36.Abay SM, Deribe K, Reda AA, et al. The Effect of Early Initiation of Antiretroviral Therapy in TB/HIV-Coinfected Patients: A Systematic Review and Meta-Analysis. J Int Assoc Provid AIDS Care. 2015;14(6):560–570. doi: 10.1177/2325957415599210. [DOI] [PubMed] [Google Scholar]
  • 37.Manosuthi W, Wiboonchutikul S, Sungkanuparph S. Integrated therapy for HIV and tuberculosis. AIDS Res Ther. 2016;13:22. doi: 10.1186/s12981-016-0106-y. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 38.Yan S, Chen L, Wu W, et al. Early versus Delayed Antiretroviral Therapy for HIV and Tuberculosis Co-Infected Patients: A Systematic Review and Meta-Analysis of Randomized Controlled Trials. PloS one. 2015;10(5):e0127645. doi: 10.1371/journal.pone.0127645. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 39.Yotebieng M, Van Rie A, Moultrie H, et al. Effect on mortality and virological response of delaying antiretroviral therapy initiation in children receiving tuberculosis treatment. Aids. 2010;24(9):1341–1349. doi: 10.1097/QAD.0b013e328339e576. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 40.Patel MR, Nana M, Yotebieng M, Tabala M, Behets F, Van Rie A. Delayed antiretroviral therapy despite integrated treatment for tuberculosis and HIV infection. The international journal of tuberculosis and lung disease : the official journal of the International Union against Tuberculosis and Lung Disease. 2014;18(6):694–699. doi: 10.5588/ijtld.13.0807. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 41.Patel MR, Westreich D, Yotebieng M, et al. The Impact of Implementation Fidelity on Mortality Under a CD4-Stratified Timing Strategy for Antiretroviral Therapy in Patients With Tuberculosis. Am J Epidemiol. 2015;181(9):714–722. doi: 10.1093/aje/kwu338. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 42.Cornell M, Lessells R, Fox MP, et al. Mortality among adults transferred and lost to follow-up from antiretroviral therapy programmes in South Africa: a multicenter cohort study. J Acquir Immune Defic Syndr. 2014;67(2):e67–75. doi: 10.1097/QAI.0000000000000269. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 43.Geng EH, Odeny TA, Lyamuya RE, et al. Estimation of mortality among HIV-infected people on antiretroviral treatment in East Africa: a sampling based approach in an observational, multisite, cohort study. Lancet HIV. 2015;2(3):e107–116. doi: 10.1016/S2352-3018(15)00002-8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  • 44.Schoni-Affolter F, Keiser O, Mwango A, et al. Estimating loss to follow-up in HIV-infected patients on antiretroviral therapy: the effect of the competing risk of death in Zambia and Switzerland. PloS one. 2011;6(12):e27919. doi: 10.1371/journal.pone.0027919. [DOI] [PMC free article] [PubMed] [Google Scholar]

RESOURCES