A comparison of tuberculosis diagnostic systems in a retrospective cohort of HIV-infected children in Rio de Janeiro, Brazil

Abstract

Objectives:

The diagnosis of pediatric tuberculosis (TB) presents many challenges, and is further complicated in HIV-infected patients. While many diagnostic systems have been proposed, there is no pediatric TB diagnosis gold standard. The outcomes of four TB diagnostic systems in HIV-infected children were compared in this study.

Methods:

A retrospective cohort study was conducted at a TB/HIV reference hospital in Rio de Janeiro. HIV-infected pediatric patients evaluated for TB from 1998 to 2010 were reassessed using four diagnostic systems: Kenneth Jones, 1969; Tidjani, 1986; Ben Marais, 2006; Brazilian Ministry of Health, 2010. Results were compared to standardized diagnoses made by an expert panel of physicians.

Results:

Of the 121 patients in the study cohort, the expert panel diagnosed 64 as TB and 57 as not TB cases. The Tidjani system showed the highest diagnostic accuracy, with and without the inclusion of microbiological data. The Tidjani and Kenneth Jones systems produced fewer false-positives, and the Ben Marais and Ministry of Health fewer false-negatives. Across systems, there was little agreement between TB diagnoses.

Conclusions:

In HIV-infected pediatric patients, the Ben Marais and Ministry of Health systems are useful for TB diagnostic screening, whereas the Tidjani and Kenneth Jones systems are best used in a reference center setting.

Introduction

According to the World Health Organization (WHO), there were one million pediatric tuberculosis (TB) cases and 136 000 TB deaths in children in 2014.¹ Children co-infected with HIV and Mycobacterium tuberculosis have a greater risk of developing TB,² as well as worse treatment outcomes and higher rates of relapse.³ While children account for just 7% of the total deaths in HIV-uninfected cases, in HIV-infected TB cases, 14% of all deaths are in children.¹ In Brazil, an estimated 15% of TB cases occur in children aged 15 years and under,⁴ and 6–7% of these cases are also infected with HIV.⁵

The standard diagnosis of pediatric TB is based on history of contact, clinical signs and symptoms, chest radiography, tuberculin skin testing (TST), and microbiological examination. However, TB diagnosis in children is complicated by challenges in a number of these diagnostic measures, and there is no gold standard by which pediatric TB cases are diagnosed. In addition to presenting clinical signs and chest X-rays that are often non-specific, respiratory specimens are difficult to collect and the bacteriological yield is low in pediatric patients, greatly reducing rates of bacteriological confirmation.¹ The diagnosis of TB is even more difficult in children infected with HIV, due to atypical presentation of signs and symptoms of TB, low sensitivity of the TST, and pulmonary diseases that mimic TB.^6–9 Since 2013, the WHO has recommended the use of the Xpert MTB/RIF molecular diagnostic system in children.¹⁰ However, a recent review of the Xpert assay in children showed this system to have high variability and, on average, only a 66% sensitivity compared to culture.¹¹ Additionally, even in settings where Xpert MTB/RIF is available, high levels of empirical treatment persist despite a negative Xpert result.^11,12

As such, there remains a critical need for a standardized system that can be used to diagnose pediatric TB, particularly in HIV-infected children. In recent decades, various scoring and categorical systems have been proposed to improve TB diagnosis in pediatric patients. While some studies suggest that these systems may be useful in immunocompetent children,^13–21 others have shown certain systems to have lower accuracy in HIV-infected children,^22–25 and to date no single system has been adequately validated in HIV-infected or uninfected children.^22,26,27

The present study aimed to analyze and compare the accuracy of different diagnostic systems in children and adolescents infected with HIV at a referral hospital in Rio de Janeiro. Specifically, the following systems were evaluated: the Kenneth Jones (KJ) system,²⁸ which is one of the oldest systems and has been employed for many years throughout the world;²⁹ the Tidjani system,³⁰ which became the system recommended by the WHO and has previously been evaluated in HIV-uninfected pediatric patients in Brazil;³¹ the Brazilian Ministry of Health (MoH) system,³² validated in HIV-infected and HIV-uninfected children^18,33 and adopted in 2002 by the MoH in the national guidelines for pediatric TB diagnosis; and the Ben Marais system (BM), the most recently developed system, which has also been evaluated previously in HIV-infected and uninfected children¹³ and has been modified for use as a pediatric diagnostic protocol in South Africa.³⁴

Study population and methods

Population

A retrospective cohort was established using the medical records of pediatric patients attended to at the Hospital Municipal Jesus between January 1998 and December 2010. Hospital Municipal Jesus is a TB and HIV pediatric reference hospital estimated to receive 15–20% of all pediatric TB cases in the Rio de Janeiro metropolitan area.³⁵ The medical records of HIV-infected patients under 15 years of age who were evaluated for active pulmonary and extrapulmonary TB during this time period were analyzed for inclusion in the study cohort. Data on epidemiological, clinical, laboratory, radiological, and treatment outcomes were extracted from the medical records using a standardized questionnaire. None of the patients had previously been evaluated for TB infection. Patients were excluded if their medical records could not be recovered or contained less than 70% of the data required for analysis, as well as if they had transferred to another service or had been lost to follow-up before initiation of the suggested treatment.

Because there is no gold standard for the diagnosis of pediatric TB, as a way to create consistent diagnostic groups for this study, a panel of experts evaluated each patient’s data and diagnosed them as a TB case (group A) or non-TB case (group B). This panel of experts consisted of an infectious diseases specialist and a pulmonologist, both with over 15 years of experience. The experts were blinded to the diagnosis made by the attending physician at the time of the case and the results of the scoring systems. Patients classified by the experts as confirmed, probable, and possible TB according to standardized pediatric clinical case definitions³⁶ were included in group A; patients classified as unlikely or not TB were included in group B. In cases where there was disagreement in the diagnosis provided by the two-person panel, a third expert—a pulmonologist specializing in pediatric TB—issued a final decision.

Evaluation of the diagnostic systems

Patients in group A (TB-positive/HIV-positive) and group B (TB-negative/HIV-positive) were evaluated and categorized or scored according to the rules proposed by the systems under evaluation – BM, Tidjani, KJ, and MoH (Supplementary Material, Table S1–S4). The MoH system is intended for the diagnosis of pulmonary TB, so it was not applied to patients presenting isolated forms of extrapulmonary TB. Additionally, unlike the BM, Tidjani, and KJ diagnostic systems, the MoH does not consider the results of microbiological examinations of M. tuberculosis. Because of this, the BM, Tidjani, and KJ systems were evaluated with and without the inclusion of microbiological examinations.

Classification of diagnostic system outcomes

In the KJ system, a score of ≥7 indicates unquestionable TB, 5–6 points probable TB, 3–4 points possible TB, and 1–2 points not likely TB. Here, subjects with ≥5 points were considered TB cases. Under the Tidjani system, a TB diagnosis is given for scores ≥6, and thus subjects with a score ≥6 were considered cases here. According to the MoH system, a score of 40 or more is a very likely case of TB, 30 or 35 indicates a possible TB case, and 25 or less is considered unlikely to be a TB case. For this study, the results of the system were evaluated using a TB case cut-off of 30 (including ‘possible’ and ‘very likely’ TB classifications) and a cut-off of 40 (only ‘very likely TB’ subjects). In the BM system and the present study, patients categorized as bacteriologically confirmed TB, radiologically certain TB, probable TB, and highly possible TB were considered TB cases.

Statistical analysis

Analyses were performed using IBM SPSS Statistics version 21.0 (IBM Corp., Armonk, NY, USA). The positive and negative predictive values (PPV and NPV) of each diagnostic system were simulated assuming different prevalence rates. To assess the degree of agreement between systems, a kappa statistic was calculated and interpreted according to the criteria of Landis and Koch, in which <0.00 = poor agreement; 0.00–0.20 = slight agreement; 0.21–0.40 = fair agreement; 0.41–0.60 = moderate agreement; 0.61–0.80 = substantial agreement; 0.81 to 1 = almost perfect agreement.³⁷

Results

Study cohort

Between January 1998 and December 2010, 447 HIV-infected pediatric patients were attended to at Hospital Municipal Jesus, 175 (39.1%) of whom were evaluated for TB. Forty-four (25.1%) patients were excluded for the following reasons: medical records could not be recovered (n = 22), parental request for transfer to another health center (n = 7), incomplete medical records (n = 6), loss to follow-up before treatment initiation (n = 5), and change in diagnosis mid-treatment (n = 4). Of the remaining 131 patients, the attending physicians diagnosed 83 (63.3%) as TB cases.

After reviewing data extracted from these 131 records, the panel of experts grouped subjects into TB cases (group A) and not TB cases (group B). Group A diagnoses agreed with 77.1% (64/83) of the original TB diagnoses; 10 of the patients originally diagnosed as TB cases did not have sufficient medical record data for the panel to make a diagnosis and were thus excluded from the study and nine subjects were classified as not TB cases (group B) by the panel. Panel diagnoses agreed with 100% (48/48) of the previous not TB diagnoses. Thus, according to the gold standard employed in this study, 64 subjects were included in group A and 57 in group B (Figure 1).

Figure 1.

Flowchart of the steps for inclusion in the study groups: 175 HIV-infected pediatric patients evaluated for TB were initially considered for the study. Forty-four were excluded due to loss to follow-up, incomplete or missing records, or change in diagnosis. Of the remaining 131 patients, 83 had been diagnosed as TB cases and 48 as not TB cases by the patients’ attending physicians. After review, the panel of experts excluded 10 cases and came to a different diagnostic conclusion for nine patients, resulting in 64 subjects in group A and 57 in group B.

In group A, 53.1% (34/64) of cases were bacteriologically confirmed; 47 (73.4%) were pulmonary TB cases, 13 (20.3%) were pulmonary and extrapulmonary disseminated TB cases, and four (6.25%) were isolated extrapulmonary TB cases. Subjects in group A were 54.7% male (35/64) and ranged in age from 6 months to 12 years (median 4.8 years), while subjects in group B were 57.9% female (33/57) and ranged in age from 4 months to 14 years (median 5.4 years). Antiretrovirals were being used by 35 (48.4%) subjects in group A and 25 (43.9%) subjects in group B at the time of TB investigation. Clinical and radiological characteristics of the subjects in each group are described in Table S5 of the Supplementary Material.

Sensitivity, specificity, and accuracy of diagnostic systems

The results of each diagnostic system were compared to the diagnoses made by the study expert panel. As the MoH scoring system is intended only for the diagnosis of pulmonary TB, the four patients in group A and six patients in group B under evaluation for extrapulmonary TB were excluded from the analysis.

The BM system had the highest sensitivity (93.8%) when microbiological data were included in the analysis, whereas the MoH system with a 30-point cut-off had the highest sensitivity (85.0%) of the systems excluding microbiological data. The Tidjani system had the highest specificity (93.0%) and accuracy (86.7% and 67.8%), regardless of the inclusion or exclusion of microbiological data. While the sensitivity of the BM system decreased only slightly in the absence of microbiological data (from 93.8% to 84.4%), the sensitivity of the other systems was reduced by about half after excluding microbiological data or increasing the cut-off point, in the case of the MoH system. Conversely, the exclusion of microbiological data did not impact the specificity of any system, and increasing the cut-off point in the MoH system almost doubled the specificity (from 39.2% to 76.5%). All systems had reduced accuracy after excluding microbiological data or increasing the cut-off point, although the reduction was greatest in the KJ and Tidjani systems (Table 1).

Table 1.

Comparison of the sensitivity, specificity, and accuracy of the diagnostic systems^a

Diagnostic system (n)	Group A n = 64	Group B n = 57	Sensitivity (95% CI)	Specificity (95% CI)	Accuracy %
Kenneth Jones
With microbiological data	43	9	67.2 (54.3–78.4)	84.2 (72.1–92.5)	75.2
Without microbiological data	18	9	28.1 (17.6–40.8)	84.2 (72.1–92.5)	54.5
Tidjani
With microbiological data	53	4	82.8 (71.3–91.1)	93.0 (83.0–98.1)	87.6
Without microbiological data	29	4	45.3 (32.8–58.3)	93.0 (83.0–98.1)	67.8
MoHb
With 30 point cut-off	51	31	85.0 (73.4–92.9)	39.2 (25.8–53.9)	64.0
With 40 point cut-off	29	12	48.3 (35.2–61.6)	76.5 (62.5–87.2)	61.3
Ben Marais
With microbiological data	60	40	93.8 (84.8–98.3)	29.8 (18.4–43.4)	63.6
Without microbiological data	54	40	84.4 (73.1–92.2)	29.8 (18.4–43.4)	58.7

CI, confidence interval; MoH, Brazilian Ministry of Health.

^aThe number of cases and percentage sensitivity, specificity, and accuracy for each diagnostic system, with and without the inclusion of microbiological data, are shown. For the MoH system, values were calculated considering a 30-point and 40-point TB case cut-off.

^bMoH: 60 TB cases and 51 not TB cases.

Comparison of false-positives and false-negatives

Among the diagnostic systems including microbiological data, the Tidjani system was associated with the lowest proportion of false-positives, while the BM system was associated with the least false-negatives. In the absence of microbiological data, the lowest proportion of false-negative results occurred with the MoH system (30-point cut-off), while the lowest proportion false-positive results occurred with the Tidjani system (Table 2). As expected, the exclusion of microbiological data did not alter the number of false-positives. However, the exclusion of microbiological data more than doubled the number of false-negatives in each system.

Table 2.

Comparison of false-positive and false-negative diagnoses^a

Diagnostic system	False-positive n (%)	False-negative n (%)
Kenneth Jones
With microbiological data	9 (15.8%)	21 (32.8%)
Without microbiological data	9 (15.8%)	46 (71.9%)
Tidjani
With microbiological data	4 (7.0%)	11 (17.2%)
Without microbiological data	4 (7.0%)	35 (54.7%)
MoH
With 30 point cut-off	31 (60.8%)	9 (15.0%)
With 40 point cut-off	12 (23.5%)	31 (51.7%)
Ben Marais
With microbiological data	40 (70.2%)	4 (6.3%)
Without microbiological data	40 (70.2%)	10 (15.6%)

MoH, Brazilian Ministry of Health.

^aThe number and percentage of false-positives and false-negatives for each diagnostic system, with and without the inclusion of microbiological data, are shown. For the MoH system, values were calculated considering a 30-point and 40-point TB case cut-off.

Positive and negative predictive values

Because PPV and NPV calculations depend on the TB prevalence in the site population, PPV and NPV were calculated using the prevalence observed in this study (52%), as well as using the simulated prevalence found at other health service locations (5%, 10%, and 15%). Considering 52% prevalence, the Tidjani system had the highest PPV and NPV when microbiological data were included. Without microbiological data, the Tidjani system had the highest PPV, while the MoH system with a 30-point cut-off had the highest NPV. Among the other prevalence simulations, with the inclusion of microbiological data, the highest PPV and NPV were observed with the Tidjani system. In the absence of microbiological data, the Tidjani system had the highest PPV, while the MoH system with a 30-point cut-off had the highest NPV across prevalence simulations (Table 3).

Table 3.

Positive predictive value (PPV) and negative predictive value (NPV) in the study cohort^a

Diagnostic system	52% prevalence		5% prevalence		10% prevalence		15% prevalence
	PPV (%)	NPV (%)	PPV (%)	NPV (%)	PPV (%)	NPV (%)	PPV (%)	NPV (%)
KJ with	82.2	70.3	18.3	98.0	32.1	95.9	42.9	93.6
KJ without	65.8	51.9	8.6	95.7	16.5	91.3	23.8	86.9
Tidjani with	92.7	83.3	38.4	99.0	56.8	98.0	67.6	96.8
Tidjani without	87.5	61.1	25.4	97.0	41.8	93.9	53.3	90.6
MoH 30 points	60.2	70.7	6.9	98.0	13.4	95.9	19.8	93.7
MoH 40 points	69.0	57.7	9.7	96.6	18.5	93.0	26.7	89.3
BM with	59.1	81.6	6.6	98.9	12.9	97.5	19.1	96.5
BM without	56.5	63.8	5.9	97.3	11.8	94.5	17.5	91.5

KJ, Kenneth Jones; MoH, Brazilian Ministry of Health; BM, Ben Marais.

^aCalculations for each were made considering (with) or not considering (without) microbiological data. For the MoH system, values were calculated considering a 30-point or 40-point TB case cut-off. The PPV and NPV of each system was calculated using the prevalence in the study cohort (52%), as well as prevalence rates commonly observed at healthcare units (5%, 10%, 15%).

Agreement among diagnostic systems

Finally, kappa statistics were calculated to compare diagnosis outcomes (Table 4). Low agreement between the systems was generally observed, with the highest agreement occurring between the BM and MoH systems and the lowest between the BM and Tidjani systems. The MoH and KJ, the MoH and the Tidjani, and the BM and Tidjani systems showed only slight agreement; the Tidjani and KJ, the BM and KJ, and the BM and MoH systems were found to have fair agreement in their classification of cases.

Table 4.

Comparison of diagnostic outcomes in each system^a

Diagnostic system	Tidjani	MoH	BM
KJ	0.22	0.17	0.24
Tidjani	X	0.11	0.06
MoH	X	X	0.28

MoH, Brazilian Ministry of Health; BM, Ben Marais; KJ, Kenneth Jones.

^aKappa values, shown here, were calculated to assess the agreement in diagnostic categorization between each of the systems.

Discussion

In this study an expert panel standardized diagnosis was used to directly compare the results of four diagnostic systems in HIV-infected children. Despite the continued need for a gold standard method to diagnose pediatric TB, there has been little research evaluating the various scoring systems that have been developed. While other studies have compared scoring systems in HIV-uninfected and mixed infection status populations,^22,31 very little has been done to evaluate individual systems in HIV-infected children and, to the authors’ knowledge, no study has directly compared scoring systems only in HIV-positive pediatric patients.

Within the study cohort, it was found that the MoH and BM systems showed high sensitivity and, in turn, a low proportion of false-negative results, whereas the Tidjani and KJ systems had high specificity and, thus, a low proportion of false-positives. As bacteriological confirmation of TB in pediatric patients is particularly challenging and is not included in the MoH system, it was sought to re-evaluate the validity of all systems in the absence of microbiological data. After the exclusion of microbiological data, the Tidjani and KJ systems still had the least false-positives, and the MoH and BM the least false-negatives.

The different strengths of each system indicate that they can be most effectively applied in different diagnostic settings. Owing to their higher sensitivity and lower specificity, both in the presence and absence of bacteriological confirmation, the MoH and BM systems may be most useful for diagnostic screening among HIV-infected children at primary care facilities. Conversely, as they have higher specificity and lower sensitivity, the Tidjani and KJ systems could be best applied for final treatment decisions in reference centers. However, it is important to note that both the Tidjani and KJ scores include the response to treatment, and thus are best used in settings in which patients are closely followed after treatment initiation as well.

Particular features of the present study population may have contributed to the success of the Tidjani system. The majority of TB cases in this cohort were bacteriologically confirmed, which automatically qualifies them as TB cases in this system. However, the Tidjani system still had the highest accuracy after excluding microbiological data, indicating that more than just bacteriological confirmation contributes to its success. Highly scored characteristics such as miliary TB, cavitary lesions, hilar adenitis, and improvement with anti-TB treatment are precisely the radiological findings most commonly identified in this population. This further supports the use of the Tidjani system in reference center settings, the source of the present patient population and where patients with similar disease manifestations are likely to be found.

The findings of high specificity and low sensitivity in the KJ system are similar to those of previous studies.^17,31 Moreover, the high sensitivity and low specificity of the BM system has also been observed previously.¹³ However, evaluation of the MoH system in HIV-uninfected pediatric patients from a reference center in Salvador, Brazil found a much lower rate of false-positives and negatives at both cut-off points.³⁸ Because the scores calculated in the present cohort are akin to those of a previous study of HIV-infected patients at an outpatient clinic in Rio de Janeiro,³³ the increased rate of false-negatives and positives in the present study relative to the Salvador cohort is likely due to the difference in HIV infection status. Together, this suggests that the MoH system is more effective in populations with low HIV infection rates.

A potential limitation of this study is that the retrospective cohort was composed of patients from a referral hospital for both TB and HIV, and outcomes may differ at locations with a more diverse patient population. However, when HIV-infected patients become ill and present persistent respiratory symptoms, they are often hospitalized for treatment and further evaluation. Thus, it is likely that TB evaluation in HIV-infected children would occur in a setting such as the present one, making this cohort representative of the population of interest in this study. Additionally, as the expert panel was composed of Brazilian physicians with experience using the MoH system, it is possible that the gold standard diagnosis was biased to be more similar to this system. However, the results demonstrate each system having different strengths and weaknesses relative to the expert panel diagnosis, indicating that the gold standard diagnosis used was not heavily influenced by the experts’ previous experience with the MoH system.

In conclusion, using an expert panel to define the gold standard diagnosis, it was possible to compare the accuracy of four diagnostic systems in HIV-infected children. Together, the findings of this study indicate that in HIV-infected pediatric populations, the MoH and BM systems are useful TB screening tools for primary care centers, whereas the KJ and Tidjani systems are most useful at reference centers. The authors recommend that special attention be given to training professionals who use the diagnostic systems in primary care and reference centers. Better understanding of the strengths and weaknesses of each will support improved case detection and more appropriate anti-TB treatment decisions in these different settings.