Skip to main content
PMC home page
NIHPA Author Manuscripts logoLink to NIHPA Author Manuscripts
. Author manuscript; available in PMC: 2013 Sep 26.
Published in final edited form as: Trop Med Int Health. 2011 Aug 11;16(11):1367–1371. doi: 10.1111/j.1365-3156.2011.02854.x

Accuracy of immunological criteria for identifying virological failure in children on antiretroviral therapy - The IeDEA Southern Africa Collaboration

Mary-Ann Davies 1,*, Andrew Boulle 1, Brian Eley 2, Harry Moultrie 3, Karl Technau 4, Helena Rabie 5, Gilles van Cutsem 1,6, Janet Giddy 7, Robin Wood 8, Matthias Egger 9, Olivia Keiser 9, for the International epidemiologic Databases to Evaluate AIDS Southern Africa (IeDEA-SA) Collaboration
PMCID: PMC3783841  NIHMSID: NIHMS312058  PMID: 21834797

Summary

Objectives

To determine the diagnostic accuracy of World Health Organization (WHO) 2010 and 2006 as well as United States Department of Health and Human Services (DHHS) 2008 definitions of immunological failure for identifying virological failure in children on antiretroviral therapy (ART).

Methods

Analysis of data from children (<16 years at ART initiation) at South African ART sites at which CD4 count/percent and HIV-RNA monitoring are performed 6-monthly. Incomplete virological suppression (IVS) was defined as failure to achieve ≥1 HIV-RNA ≤400 copies/mL between 6 and 15 months on ART and viral rebound (VR) as confirmed HIV-RNA ≥5000 copies/mL in a child on ART for ≥18 months who had achieved suppression during the first year on treatment.

Results

Among 3115 children (median (IQR) age 48 (20-84) months at ART initiation) on treatment for ≥1 year, sensitivity of immunological criteria for IVS was 10%, 6% and 26% for WHO 2006 2010 and DHHS 2008 criteria respectively. The corresponding positive predictive values (PPV) were 31% 20% and 20%. Diagnostic accuracy for VR was determined in 2513 children with ≥18 months of follow-up and virological suppression during the first year on ART with sensitivity of 5% (WHO 2006/2010) and 27% (DHHS 2008). PPV results were 42% (WHO 2010), 43% (WHO 2006) and 20% (DHHS 2008).

Conclusion

Current immunological criteria are unable to correctly identify children failing ART virologically. Improved access to viral load testing is needed to reliably identify virological failure in children.

Keywords: children; antiretroviral therapy; immunological criteria; sensitivity, specificity; virological failure

Introduction

Poor access to viral load testing in resource-limited settings results in reliance on immunological criteria to identify treatment failure in patients on antiretroviral therapy (ART). Studies in adults have shown limited value of immunological criteria to detect virological failure (VF) (Keiser et al. 2009, Mee et al. 2008, Badri et al. 2008). While low sensitivity of immunological criteria for identifying VF could result in delayed switching to second-line treatment with accumulation of resistance mutations, low positive predictive value (PPV) may incorrectly identify patients as needing second-line treatment when they are virologically suppressed. The few paediatric studies of diagnostic accuracy of immunological criteria for VF are limited by small cohort size and/or VF definition based on single elevated HIV-RNA measurements (Ruel et al. 2010, Jittamala et al. 2009, Emmett et al. 2010). We analysed data from routine pediatric ART clinics to determine the diagnostic accuracy of WHO (2006, 2010) and United States Department of Health and Human Services (DHHS) 2008 (National Institutes of Health 2008) criteria for immunological failure for identifying (i) incomplete viral suppression during the first year on ART and (ii) confirmed viral rebound.

Methods

Data was collected prospectively from ART-naïve children (<16 years at ART start) initiating ≥3 antiretrovirals at South African sites participating in IeDEA-Southern Africa (IeDEA-SA, see www.iedea-sa.org), all of which had CD4 and HIV-RNA measurements performed 6-monthly. The characteristics of these sites have been described previously (Davies et al. 2009). Each site has institutional ethical approval to contribute data to IeDEA analyses.

HIV-RNA was measured using Amplicor 1.5 (Roche Diagnostics) or NucliSens EasyQ assays (bioMerieux), with good comparability (Stevens et al. 2005). CD4 measurements were performed using standard dual platform flow cytometry or the single platform PanLeucogated method (Glencross et al. 2008). We defined incomplete virological suppression (IVS) as failure to achieve ≥1 HIV-RNA ≤400 copies/mL between 6 and 15 months on ART and viral rebound (VR) as confirmed HIV-RNA ≥5000 copies/mL in a child on ART for ≥18 months whose HIV-RNA had suppressed during the first year on treatment. We examined the following immunological criteria: WHO 2010: No definition for children <2 years; CD4%<10% or CD4<200 cells/mm3 (age 2-4 years); CD4<100 cells/mm3 (age ≥5 years) (WHO 2010). WHO 2006: No definition for children <1 year; CD4%<15% (age 1-2 years); CD4%<10% (age 3-4 years); CD4<100 cells/mm3 (age ≥5 years) (WHO 2006). DHHS: During first year on ART: <5 percentage point CD4% increase from baseline (CD4%<15 and age <5 years at baseline) or <50 cells/mm3 CD4 increase from baseline (CD4<200 cells/mm3 and age ≥5 years at baseline); decline of CD4% by 5 percentage points from previous value, confirmed at subsequent measurement (<365 days after first low value) (any age); return of CD4 count ≤baseline value (age ≥5 years at baseline) (National Institutes of Health 2008).

For IVS, we assessed the diagnostic accuracy of never achieving a CD4 above the immunological thresholds between 6 and 15 months on ART for identifying a child who never had HIV-RNA ≤400 copies/mL during the same period. Only children followed up for ≥1 year with ≥1 HIV-RNA measurement between 6 and 15 months were included. For VR, CD4 and HIV-RNA measurements were carried forward for up to 3 months where tests were performed asynchronously. For each unique paired CD4 and HIV-RNA measurement, we determined the diagnostic value of CD4-based criteria for VR, using robust standard errors to account for multiple measures per patient. Immunological results for which there was still no concurrent virological diagnosis after carrying forward results, were not evaluated. Further, the last immunological result before the end of follow-up/data base closure was excluded to ensure that there was sufficient follow-up to for a confirmatory low viral load measurement to have been done. Separate sensitivity analyses were performed requiring either consecutive (within 365 days) CD4 measurements meeting immunological criteria or using an HIV-RNA threshold of 1000 copies/mL to define confirmed VR.

Results

Of 3640 children with ≥1 year of follow-up, 3115 (86%) had ≥1 HIV-RNA measurement between 6 and 15 months on ART. At ART initiation, median (interquartile range (IQR)) age was 48 (20-84) months. Most children were severely ill at ART initiation: 81% had WHO-defined severe immune suppression (n=2911) and 68% of children had WHO Clinical Stage 3/4 disease (n=2294) (WHO 2006). In keeping with South African guidelines recommending protease inhibitor- (PI-) based first-line therapy in children <3 years old (National Department of Health South Africa 2005), 35% of children were on PI-based first-line treatment with non-nucleoside reverse transcriptase-based therapy in the remainder. IVS occurred in 12.6% of children and sensitivity of immunological criteria for identifying IVS ranged from 6% (WHO 2010) to 26% (DHHS) and PPV from 20% (WHO 2010 and DHHS) to 31% (WHO 2006) (Table 1 (a)).

Table 1(a).

Diagnostic value of immunological criteria for identifying children with incomplete viral suppression during first year on ART

Comparison of immunological failure diagnosis
(based on highest CD4 between 6 and 15 months on ART)
with failure of viral suppression in same period
 Comparison of immunological failure diagnosis
(based on highest CD4 between 6 and 15 months on ART
and confirmed on a subsequent measurement)
with failure of viral suppression in same period
WHO 2006† WHO 2010§ USA WHO 2006† WHO 2010§ USA
Number of children (%) 90/2714 (3.3) 61/2380 (2.6) 355/2585 (13.7) 13/2502 (0.5) 9/2217 (0.4) 72/2277 (3.2)
Number of children with
paired data* 		2581 2256 2470 2405 2126 2190
Sensitivity (%) 10 (7 - 14) 6 (3 - 10) 26 (21 - 31) 2 (0 - 4) 2 (0 - 4) 9 (5 - 12)
Specificity (%) 97 (97 - 98) 98 (97 - 98) 87 (86 - 89) 99 ( 99 - 100) 99 (99 - 100) 97 (97 - 98)
PPV (%) 31 (22 - 41) 20 (10 - 30) 20 (16 - 24) 38 (8 - 69) 33 (0 - 72) 27 (16 - 38)
NPV (%) 90 (89 - 91) 92 (91 - 93) 91 (89 - 92) 90 (89 - 91) 92 (91 - 93) 90 (89 - 92)
Number true positives 28 12 69 5 3 19
Number true negatives 2242 2017 1928 2151 1945 1916
Number false negatives 250 179 197 241 172 204
Number false positives 61 48 276 8 6 51
LR + 3.8 2.7 2.07 5.49 5.57 3.29
LR − 0.92 0.95 0.85 0.98 0.99 0.94
Area under ROC curve 0.537 0.520 0.567 0.508 0.507 0.530
Open in a new tab
*

Number of children with paired data (both CD4 and HIV-RNA measures available and CD4 criteria evaluable in terms of immunological criteria) differs for different definitions because of different age and data requirements for each definition

†

Only determined for children>12 months of age

§

Only determined for children>24 months of age

PPV = positive predictive value; NPV = negative predictive value; LR + = likelihood ratio of a positive test; LR − = likelihood ratio of a negative test; ROC = receiver operating characteristic

The accuracy of immunological criteria for identifying VR was assessed in 2513 children with at least ≥18 months follow-up on ART whose HIV-RNA had suppressed during the first year on treatment. The cumulative probability of viral rebound in the following 2 years (by 42 months since ART start) was 5.5% (95% CI: 4.2-7.1). Requiring consecutive CD4 counts to meet immunological criteria increased PPV only slightly at the expense of sensitivity, without any improvement in the area under the receiver operating characteristic curve (Table 1 (b)).Sensitivity of immunological criteria for identifying VR ranged from 5% (WHO 2006/2010) to 27% (DHHS) and PPV from 20% (DHHS) to 43% (WHO 2006) (Table 1(b)). Lowering the VR threshold to 1000 copies/mL and using confirmed CD4 values to define immunological criteria yielded sensitivity (95%CI) of 2% (0-5%) (WHO 2006); 2% (0-4%) (WHO 2010) and 14% (8-19%) (DHHS); and PPV (95% CI) of 90% (72-100%) (WHO 2006); 89% (68-100%) (WHO 2010) and 38% (26-50%) (DHHS).

Table 1(b).

Diagnostic value of immunological criteria for identifying children with viral rebound

Comparison of immunological failure diagnosis
(based on single CD4 measure meeting criteria)
with confirmed HIV-RNA >5,000 copies/ml
 Comparison of immunological failure diagnosis
(based on consecutive CD4 measure meeting criteria
with confirmed HIV-RNA >5,000 copies/ml
WHO 2006 WHO 2010§ USA WHO 2006 WHO 2010§ USA
Cumulative probability by
2 years		 2.2% (1.5 - 3.2) 2.3% (1.5 - 3.3) 28.1% (25.2 - 31.3) 0.4% (0.2 - 1.0) 0.3% (0.1 - 0.9) 10.4% (8.6 -12.6)
Number of evaluable
pairs of data* 		2499 2593 2191 2395 2507 2110
Sensitivity (%) 5 (2 - 9) 5 (2 - 9) 27 (19 -35) 2 (0 - 5) 2 (0 - 4) 13 (6 - 21)
Specificity (%) 99 (99 - 100) 99 (99 - 100) 88 (86 - 90) 100 (99 - 100) 100 (99 - 100) 95 (94 - 96)
PPV (%) 43 (23 - 63) 42 (22-61) 20 (13 - 26) 50 (19 - 81) 44 (14 - 75) 21 (10 - 32)
NPV (%) 91 (89 - 93) 91 (89 - 93) 92 (90 - 94) 91 (89 - 93) 91 (89 - 93) 91 (89 - 93)
Number true positives 13 13 58 5 4 26
Number true negatives 2245 2327 1740 2168 2269 1816
Number false negatives 224 235 157 217 229 172
Number false positives 17 18 236 5 5 96
LR + 7.3 6.83 2.26 9.79 7.81 2.62
LR − 0.95 0.95 0.83 0.98 0.98 0.91
Area under ROC curve 0.524 0.522 0.575 0.51 0.508 0.541
Open in a new tab
*

Paired data created from carrying forward asynchronously measured CD4 and HIV-RNA results for up to 3 months; Number of children with paired data definitions because of different age and data requirements for each definition

§

Only determined for children>24 months of age

PPV = positive predictive value; NPV = negative predictive value; LR + = likelihood ratio of a positive test; LR − = likelihood ratio of a negative test; ROC = re characteristic

Discussion

In this large longitudinal study we found that sensitivity and PPV were low for identifying VF using WHO and DHHS immunological criteria. For example, using confirmed HIV-RNA ≥5000 copies/mL after initial virological suppression to define VR, sensitivity was only 4% using either WHO 2006 or 2010 criteria. Fewer than half of children meeting WHO 2006/2010 criteria would have HIV-RNA ≥5000 copies/mL.

These results concur with those of previous small studies from resource-limited settings. Among 116 children in Uganda 20 (17%) had sustained viraemia (≥400 copies/mL) beyond 24 weeks on ART (Ruel et al. 2010). Only 2 of these ever met WHO 2006 immunological criteria, and did so after >550 days of viraemia, while none met WHO 2010 criteria (Ruel et al. 2010). Similarly, in a cross-sectional study of 206 children in Tanzania on ART for a median duration of 2.4 years, 32% had a single HIV-RNA ≥400 copies/mL (Emmett et al. 2010). WHO 2006 clinical and immunological failure criteria combined had a PPV of 100% but identified only 3.5% of children with HIV-RNA ≥400 copies/mL (Emmett et al. 2010). In Thailand, the sensitivity and PPV of DHHS immunological criteria for identifying children with a single HIV-RNA >1000 copies/mL was 15% and 16% respectively (Jittamala et al. 2009).

The poor sensitivity of immunological criteria for identifying VF is disappointing, however perhaps an even greater concern is preventing the incorrect diagnosis of treatment failure in a virologically suppressed child, with unnecessary switch to second-line treatment, hence the importance of PPV. While PPV reached 89% for identifying VR ≥1000 copies/mL using a confirmed CD4 value meeting WHO 2010 criteria, given that the WHO 2010 guidelines HIV-RNA threshold for switching to second-line is 5000 copies/ml, in most instances the number of false positives using immunological criteria is unacceptably high. Further, absence of immunological failure on ART provides no assurance that a child is virologically suppressed and not accumulating resistance mutations, and even high CD4 thresholds (e.g. DHHS criteria) have low sensitivity for VF (PENPACT1 Study Team 2011). Improved access to VL monitoring both to assess adherence and identify VF is therefore needed (Ford & Calmy 2010, Wilson et al. 2009).

There are several limitations to this analysis of routinely collected data. Missing baseline CD4 values limited evaluation of DHHS criteria. Baseline CD4 values are often unavailable to clinicians if children initiate ART on clinical criteria, records are lost or a child changes treatment site after ART initiation, highlighting the value of simple criteria using current or recent measurements. The accuracy of WHO 2006 and 2010 criteria could not be evaluated in children <1 and <2 years old respectively. Work-up bias may occur if either the reference (HIV-RNA) or index (CD4) tests are not applied consistently (Whiting et al. 2004). In particular, rigorous confirmation of low CD4 values is seldom done in South Africa, due to access to HIV-RNA measurements to diagnose treatment failure. Exclusion of intercurrent illness as a cause of low CD4, as advised in WHO guidelines (WHO 2010), was not possible due to limited data on episodes of clinical illness. Immunological criteria may have performed better if only low CD4 counts not explained by intercurrent illness had been considered. PPV is dependent on the VF incidence in different programs, however the incidence in this cohort was similar to that of other studies (Davies et al. 2011, Jittamala et al. 2009, Kamya et al. 2007).

In summary, our results suggest that current immunological criteria are unable to correctly identify children failing ART virologically. Improved access to viral load testing appears to be the only feasible approach at this stage for reliably identifying VF in children on ART. There are several existing technologies for the measurement of viral load (Stevens and Marshall 2010) and expanded access should be supported by governments and donors.

Acknowledgements

This study was supported by the National Institute of Allergy and Infectious Diseases and the Eunice Kennedy Shriver National Institute of Child Health and Human Development (grant 1 U01 AI069924-01). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript. We thank all the children whose data was used in this analysis, as well as their caregivers. We also thank all staff at participating sites for providing clinical patient care and preparation of data contributed to the IeDEA Southern Africa collaboration. Many thanks to Nicola Maxwell for preparing the combined data for analysis, and to Morna Cornell and Claire Graber for project management.

References

  1. Badri M, Lawn SD, Wood R. Utility of CD4 cell counts for early prediction of virological failure during antiretroviral therapy in a resource-limited setting. BMC.Infect.Dis. 2008;8:89. doi: 10.1186/1471-2334-8-89. 89. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Davies M, Keiser O, Technau K, et al. Outcomes of the South African National Antiretroviral Treatment (ART) programme for children - The IeDEA Southern Africa Collaboration. S Afr Med J. 2009;99:730–737. [PMC free article] [PubMed] [Google Scholar]
  3. Davies M, Moultrie H, Eley B, et al. Virologic failure and second-line antiretroviral therapy in children in South Africa: The IeDEA Southern Africa collaboration. J Acquir Immune Defic Syndr. 2011;56:270–278. doi: 10.1097/QAI.0b013e3182060610. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Emmett SD, Cunningham C, Mmbaga BT, et al. Predicting virologic failure among HIV-1 infected children receiving antiretroviral therapy in Tanzania: a cross-sectional study. J Acquir Immune Defic Syndr. 2010;54:368–375. doi: 10.1097/QAI.0b013e3181cf4882. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Ford N, Calmy A. Improving first-line antiretroviral therapy in resource-limited settings. Curr Opin HIV AIDS. 2010;5:38–47. doi: 10.1097/COH.0b013e3283339b41. [DOI] [PubMed] [Google Scholar]
  6. Glencross DK, Janossy G, Coetzee LM, et al. Large-scale affordable PanLeucogated CD4+ testing with proactive internal and external quality assessment: in support of the South African national comprehensive care, treatment and management programme for HIV and AIDS. Cytometry B Clin Cytom. 2008;74:S131–S140. doi: 10.1002/cyto.b.20384. [DOI] [PubMed] [Google Scholar]
  7. Jittamala P, Puthanakit T, Chaiinseeard S, Sirisanthana V. Predictors of virologic failure and genotypic resistance mutation patterns in Thai children receiving non0nucleoside reverse transcriptase inhibitor-based antiretroviral therapy. Pediatr Infect Dis J. 2009;28:826–830. doi: 10.1097/INF.0b013e3181a458f9. [DOI] [PubMed] [Google Scholar]
  8. Kamya MR, Mayanja-Kizza H, Kambugu A, et al. Predictors of long-term viral failure among ugandan children and adults treated with antiretroviral therapy. J Acquir Immune Defic Syndr. 2007;46:187–193. doi: 10.1097/QAI.0b013e31814278c0. [DOI] [PubMed] [Google Scholar]
  9. Keiser O, MacPhail P, Boulle A, et al. Accuracy of WHO CD4 cell count criteria for virological failure of antiretroviral therapy. Trop Med Int Health. 2009;14:1220–5. doi: 10.1111/j.1365-3156.2009.02338.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
  10. Mee P, Fielding KL, Charalambous S, Churchyard GJ, Grant AD. Evaluation of the WHO criteria for antiretroviral treatment failure among adults in South Africa. AIDS. 2008;22:1971–1977. doi: 10.1097/QAD.0b013e32830e4cd8. [DOI] [PubMed] [Google Scholar]
  11. National Department of Health South Africa . Guidelines for the management of HIV-infected children in South Africa. Jacana; 2005. [Google Scholar]
  12. National Institutes of Health [Accessed 2008/17/12];Guidelines for the use of antiretroviral agents in Pediatric HIV infection (Online) 2008 NIH. Available: http://AIDSinfo.nih.gov.
  13. PENPACT1 Study Team First-line antiretroviral therapy with a protease inhibitor versus non-nucleoside reverse transcriptase inhibitor and switch at higher versus low viral load in HIV-infected children: an open-label, randomised phase 2/3 trial. Lancet Infect Dis. 2011;11:273–283. doi: 10.1016/S1473-3099(10)70313-3. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Ruel TD, Achan J, Charlebois E, Havlir D, Kamya M. Sustained viremia is common among HIV-infected Ugandan children receiving antiretroviral therapy and not detected by WHO CD4 criteria. 18th International AIDS Conference; Vienna Austria. 2010. [Google Scholar]
  15. Stevens W, Wiggill T, Horsfield P, Coetzee L, Scott LE. Evaluation of the NucliSensEasyQ assay in HIV-1-infected individuals in South Africa. Journal of Virological Methods. 2005;124:105–110. doi: 10.1016/j.jviromet.2004.11.009. [DOI] [PubMed] [Google Scholar]
  16. Stevens WS, Marshall TM. Challenges in implementing HIV load testing in South Africa. J Infect Dis. 2010;201(Suppl 1):S78–84. doi: 10.1086/650383. [DOI] [PubMed] [Google Scholar]
  17. Whiting P, Rutjes AW, Reitsma JB, Glas AS, Bossuyt PM, Kleijnen J. Sources of variation and bias in studies of diagnostic accuracy: a systematic review. Ann Intern Med. 2004;140:189–202. doi: 10.7326/0003-4819-140-3-200402030-00010. [DOI] [PubMed] [Google Scholar]
  18. WHO . Antiretroviral therapy of HIV infection in infants and children: towards universal access (Online) WHO; Switzerland: [Accessed 2007/09/02]. 2006. Available: http://www.who.int/hiv/pub/paediatric/infants/en/index.html. [Google Scholar]
  19. WHO [Accessed 2010/19/10 2010];Antiretroviral therapy for HIV infection in infants and children: Towards universal access. Recommendations for a public health approach: 2010 revision. 2010 (Online). Available: http://www.who.int/hiv/pub/paediatric/infants2010/en/index.html. [PubMed]
  20. Wilson D, Keiluhu AK, Kogrum S, et al. HIV-1 viral load monitoring: an opportunity to reinforce treatment adherence in a resource-limited setting in Thailand. Trans R Soc Trop Med Hyg. 2009;103:601–6. doi: 10.1016/j.trstmh.2008.11.007. [DOI] [PubMed] [Google Scholar]

RESOURCES