Evaluation of Tuberculosis Diagnostics in Children: 2. Methodological Issues for Conducting and Reporting Research Evaluations of Tuberculosis Diagnostics for Intrathoracic Tuberculosis in Children. Consensus From an Expert Panel

Abstract

Confirming the diagnosis of childhood tuberculosis is a major challenge. However, research on childhood tuberculosis as it relates to better diagnostics is often neglected because of technical difficulties, such as the slow growth in culture, the difficulty of obtaining specimens, and the diverse and relatively nonspecific clinical presentation of tuberculosis in this age group. Researchers often use individually designed criteria for enrollment, diagnostic classifications, and reference standards, thereby hindering the interpretation and comparability of their findings. The development of standardized research approaches and definitions is therefore needed to strengthen the evaluation of new diagnostics for detection and confirmation of tuberculosis in children.

In this article we present consensus statements on methodological issues for conducting research of Tuberculosis diagnostics among children, with a focus on intrathoracic tuberculosis. The statements are complementary to a clinical research case definition presented in an accompanying publication and suggest a phased approach to diagnostics evaluation; entry criteria for enrollment; methods for classification of disease certainty, including the rational use of culture within the case definition; age categories and comorbidities for reporting results; and the need to use standard operating procedures. Special consideration is given to the performance of microbiological culture in children and we also recommend for alternative methodological approaches to report findings in a standardized manner to overcome these limitations are made. This consensus statement is an important step toward ensuring greater rigor and comparability of pediatric tuberculosis diagnostic research, with the aim of realizing the full potential of better tests for children.

The global burden of childhood tuberculosis is unknown because there is a lack of routine recording and reporting of tuberculosis cases in children by national tuberculosis control programs and because of the difficulties of bacteriologically confirming the diagnosis. It is generally reported that at least 10%–15% of cases in the world and up to 25% of those arising in countries with high tuberculosis burden occur in children [1]. Thus, childhood tuberculosis represents a significant but still neglected clinical and public health problem [2].

Most of the difficulty in attaining a clinical diagnosis in children emanates from the diverse and relatively nonspecific clinical presentations of childhood tuberculosis, which are due in part to an immature immune system and the poor performance of traditional microbiological tests in children. Microbiological culture, which is the reference standard for test evaluation in adults, is often negative in children due to the paucibacillary nature of tuberculosis and the difficulty in producing sputum on demand. In addition, investigators have used a variety of case definitions for tuberculosis, hindering the comparability of studies [3]. A cycle has occurred in which the lack of diagnostics hinders the development of better treatment and control tools. In turn, the lack of standardized definitions and a poor reference standard leads to a general reluctance to enroll children in research studies [4], hindering research to improve diagnostics.

Developing standardized approaches that facilitate the evaluation of tuberculosis diagnostics among pediatric populations is thus a critical first step toward breaking these barriers and strengthening the scientific evidence to guide improved detection and confirmation of tuberculosis in children. Therefore, expert clinicians, researchers, and opinion leaders were invited to a workshop, Critical Issues in Pediatric Tuberculosis Diagnostics Research in HIV-Infected and Uninfected Children, organized by the National Institutes of Health in Washington, DC, in June 2011. The aims of the workshop were to obtain consensus on a rigorous clinical case definition to be used for uniform classification of intrathoracic tuberculosis in children and to explore methodological approaches for conducting and reporting research for the evaluation of new tuberculosis diagnostic tests in children. The literature was surveyed prior to the meeting to prepare statements for each aim, including the use of culture as the reference standard and consideration of possible approaches when a gold standard is unavailable. Participants volunteered to take part in 2 parallel discussion groups, each focusing on 1 of the aims stated above. We used formal group consensus methods, which included modifying the statements based on the discussions. All edited statements then underwent a formal vote. Voting rules included 3 options: agree, disagree, and veto. Consensus was declared for each definition if >75% of panel members agreed to the statement and <2 participants had exercised their right to veto the definition. All statements were further reviewed in a plenary workshop session.

An accompanying paper also published in this issue of the Journal of Infectious Diseases [5] presents the outcome of the panel’s discussion for the first aim and proposes a consensus clinical case definition for research on intrathoracic tuberculosis in childhood. Here, we summarize the discussions and recommendations, addressing further methodological considerations on test evaluations and the rationale for the development of the proposed clinical case definition.

PHASED APPROACH FOR TUBERCULOSIS DIAGNOSTICS RESEARCH

The group recommends a phased approach when planning the evaluation of novel diagnostics, as outlined in the document from the New Diagnostics Working Group of the Stop TB Partnership “Pathways to better diagnostics for tuberculosis: A blueprint for the development of tuberculosis diagnostics” [6].

Following preclinical feasibility and optimization studies, which assess the suitability of a new diagnostic test in pediatric populations and samples without necessarily focusing on accuracy, the clinical evaluation of a new diagnostic should proceed in 2 consecutive phases:

1. An early proof-of-principle evaluation phase, to demonstrate that the diagnostic can distinguish symptomatic children (with microbiologically confirmed tuberculosis) from healthy controls (with and without Mycobacterium tuberculosis infection) with reliable reproducibility. This phase explores the potential test accuracy and might use case-control or cross-sectional study designs or can be carried out on well-characterized banked specimens.

2. If the test fulfills prestipulated criteria and is judged promising in children, late evaluation studies would be pursued. It should be noted that in these early-phase evaluations the sampling strategy fails to represent the full spectrum of the disease, resulting in a type of bias referred to as “spectrum bias” [7], frequently making such evaluations overly optimistic.

Late evaluation studies aim to measure test accuracy in a pediatric population where it is clinically indicated and the test performance in clinical settings is as close as possible to real-life settings. This phase should provide quantitative estimates of the actual clinical validity/utility of the test and therefore needs to enroll children with the full spectrum of suspected intrathoracic tuberculosis, including those in whom a microbiological confirmation was not obtained or when alternative organisms were identified, avoiding the bias noted in earlier evaluations.

Both phases should enroll adequate representation of the appropriate age groups and be conducted in relevant subgroups where the test may perform differently, such as human immunodeficiency virus (HIV)–positive children, children with malnutrition, or young infants. Multiple coordinated studies at complementary sites should be encouraged.

ENTRY CRITERIA AND POPULATION SUBGROUPS

Late-stage evaluation studies should enroll children with a suspicion of tuberculosis disease. The criteria to define this population are listed in Table 1. The intent of using symptom-based enrollment criteria is to ensure that only children with possible clinical disease are enrolled while excluding asymptomatic contacts. These criteria thus have high sensitivity but low specificity, with the goal of allowing the inclusion of children across the whole spectrum of disease. Consensus was reached that pediatric enrollment for research studies on intrathoracic tuberculosis should be based on the presence at initial evaluation of any 1 of these clinical signs and symptoms. Although failure to thrive is considered an important inclusion criterion, many children with possible intrathoracic tuberculosis attend health services sporadically and information on longitudinal growth is rarely available. We also recognized that the definition of failure to thrive also varies widely because health services use different growth charts and tools to evaluate and report weight loss. For this reason, we suggested that children with significant weight-for-age or weight-for-height deficits (defined as z scores <−2 for both) should also qualify for enrollment.

Table 1.

Criteria for Enrollment of Children Into Tuberculosis Diagnostic Evaluation Studies

Age	Criterion	Observation
Children ≤10 y	Cough	Persistent (>2 weeks), nonremitting and unexplaineda
	Weight loss/failure to thrive	Unexplained weight loss (>5% reduction in weight compared with the highest weight recorded in last 3 mo)
		OR
		Failure to thrive (clear deviation from a previous growth trajectory or documented crossing of percentile lines in the preceding 3 mo or in the absence of information on previous/recent growth trajectory: weight for age or weight for height z scores ≤−2)
		AND
		Not responding to nutritional rehabilitation (or antiretroviral therapy if human immunodeficiency virus infected)
	Fever	Persistent (>1 wk) and unexplained fever (>38°C) reported by a guardian or objectively recorded at least once
	Lethargy or reduced playfulness	Persistent, unexplained lethargy or decrease in playfulness/activity reported by the parent/caregiver
Infants <60 d	Any of the criteria listed above
	Pneumonia, unexplained hepatosplenomegaly or sepsis-like illness not responding to appropriate treatment	Where other causes are excluded or not excluded

Enrollment should be based on the presence of any 1 of the signs and symptoms listed.

^aSome studies may investigate children with cough <2 weeks’ duration to explore whether children with acute symptoms have tuberculosis.

Given the variations in clinical presentation, whenever feasible, the types of specimens collected, and the potential variations in test performance, study populations should be stratified by age and HIV status. The following age groups are proposed: 0 to <2, 2 to <5, 5 to <10, and ≥10 years. An appropriate sample size should be selected to ensure the adequate representation of age and HIV status categories, and the same categories should be used to report study findings.

REFERENCE STANDARD

Culture of M. tuberculosis complex is the accepted reference standard in adults, which has a lower limit of detection of approximately 10 bacilli/mL of sputum. There are, however, many options for culture, including commercial and noncommercial methods and liquid and solid media. Liquid is more sensitive than solid media, and a greater sensitivity is achieved when >1 culture method is used or if >1 specimen is cultured. A systematic review of 37 studies in adults showed that in most symptomatic cases (85.8%), M. tuberculosis complex is detected in the first sputum culture, while the second and third specimens have additional incremental yields of only 11.9% and 3.1% [8]. Commercial and automated liquid-culture systems have greater sensitivity but higher bacterial contamination rates than solid media [9]. The most frequently used noncommercial systems include thin layer agar and the microscopic observation drug-sensitivity (MODS) assay. Thin layer agar had 92.6% sensitivity in a multicenter study [10] and MODS had 92%–97.5% sensitivity in several studies [11–13]. However, these sensitivity estimates are based on comparison with other culture methods and thus are only indicative of the actual performance of culture because the number of studies is small for some of these methods and may be biased in either direction.

There is a paucity of data on the performance of culture in children that varies with clinical presentation, disease severity, HIV status, age, and the type of specimen tested. For example, solid culture (Lowenstein-Jensen media) had a sensitivity of 38% and 52% in Spain and Ethiopia among 117 and 355 children with signs and symptoms of pulmonary tuberculosis [14, 15]. MODS and liquid culture had 81.3% and 88.6% sensitivity, respectively, in 96 children with microbiological confirmation in Vietnam [16]. In Peru, 1 study reported that 71% of 103 children with high probability of pulmonary tuberculosis were culture-positive by mycobacteria growth indicator tube, Lowenstein-Jensen media, or Middlebrook agar [17], while another study [18] reported that 10% of 218 children with moderate and high risk of pulmonary tuberculosis had positive culture. Most of these studies collected gastric aspirates (swallowed sputum) alone or in combination with expectorated and induced sputum. Although the sensitivity seems better in sputum and induced sputum, the sensitivity of culture in children still appears to be lower than in adults. However, the data are difficult to interpret and it is nearly impossible to compare across studies in the absence of uniform case definitions and a suitable reference standard to adequately discriminate between tuberculosis cases and nontuberculosis cases.

Because investigations related to children are initiated in the presence of nonspecific signs and symptoms and children are more likely to have paucibacillary disease than adults, it is not surprising that culture sensitivity is lower. Despite these limitations, the consensus is that culture should be used as the only confirmatory test in symptomatic children in early-stage evaluations. Because culture has high specificity, tuberculosis can be considered confirmed when culture is positive for M. tuberculosis complex. However, tuberculosis cannot be ruled out if the culture is negative because a large proportion of true tuberculosis cases may be missed by culture [19]; further speciation would be useful in culture-positive children with advanced HIV infection to rule out the possibility of bacille Calmette-Guérin disease [20]. It was agreed that characterizing discordant cases is an important part of this exploratory phase and that the stated performance of a diagnostic test should not be confused with analysis using discordance resolution methods, which are statistically flawed.

Late evaluation studies should report results using the clinical case definitions described in the accompanying paper [5], which discriminates the degree of certainty of a diagnosis of tuberculosis. All studies should plan for all children to submit specimens for culture because the clinical definition uses culture to classify children as microbiologically confirmed and standardizes the classification of the remaining children without laboratory confirmation.

Standard estimators of sensitivity and specificity assume that a reference standard always correctly identifies those with and without disease. Given that in this setting the sensitivity of culture is low and may be variable, the high proportion of tuberculosis cases missed by culture would bias the estimates of sensitivity, specificity of the diagnostic under consideration and the prevalence. Unless the presence and absence of tuberculosis can be confirmed with certainty, these parameters alone could be misleading and of limited value [21]. Instead, the new diagnostic test results should be reported as the proportion of positive tests among culture-positive children and the proportion of negative tests among culture-negative children. Characterization of the discordant cases (eg, those that are negative on culture but positive on the new diagnostic test) will further elucidate the properties of the new test. Broadening the definition of tuberculosis to include signs and symptoms other than culture confirmation would capture more tuberculosis cases. However, this will be offset by incorrectly classifying some children without tuberculosis as tuberculosis cases, as shown in Figure 1. Therefore, test summaries should be reported relative to the disease certainty classification (ie, confirmed, probable, possible, unlikely, not tuberculosis) as defined in the clinical case definition—namely, the proportion of positive and negative tests within each category to further characterize the properties of the test under evaluation (see Table 2). In addition, it is suggested that a summary table or figure with the number of children satisfying each component in the “AND” or “OR” rules should be provided. For example, for children with “probable tuberculosis,” include the respective frequencies of children with at least 1 sign and symptom suggestive of tuberculosis disease and chest radiographs consistent with intrathoracic tuberculosis disease who have (1) a positive clinical response to antituberculosis treatment or (2) documented exposure to tuberculosis or (3) immunological evidence of tuberculosis infection.

Figure 1.

Schematic of the disease spectrum within a study population. The large circle represents exposed children, a subset of whom will become infected with Mycobacterium tuberculosis. Of those infected, only a subset develops disease due to M. tuberculosis, and a smaller subset is confirmed microbiologically. Note that a small subset of children with microbiologic confirmation will not have active tuberculosis, as children experiencing an initial infection may have mycobacteremia, which is contained and may never develop symptoms. Also note that children with clinical symptoms consistent with tuberculosis can fall into any of the other circles, including those unexposed to tuberculosis.

Table 2.

Layout for Additional Reporting of Index Test Results

	Tuberculosis Research Case Definition
	Confirmed	Probable	Possible	Unlikely	Not Tuberculosis
Test +	N1 (%)a	N3 (%)a	N5 (%)a	N7 (%)a	N9 (%)a
Test −	2 (%)a	N4 (%)a	N6 (%)a	N8 (%)a	N10 (%)a

The research case definition is described in Graham et al.

^aColumn percentage.

Given the poor performance of culture in children, a composite laboratory-based reference standard could be defined for the classification of a larger number of children as microbiologically-confirmed cases once sufficient evidence becomes available on the performance of new diagnostic tests, such as automated nucleic acid amplification tests in children [22]. However, the group agreed that there is not enough information at this stage to develop this composite laboratory reference standard and further studies are needed. Statistical methods for the evaluation of diagnostics, such as latent class analysis and others, that aim to estimate sensitivity and specificity in the absence of a suitable reference standard were discussed by the group. These methods have many limitations, including that fact that estimated sensitivity and specificity may depend heavily on the model assumptions, which cannot be validated [23]. More research applying these methods in pediatric tuberculosis diagnostics is necessary to elucidate whether these limitations apply in this setting.

Finally, we achieved consensus that late-stage evaluation studies should be conducted in curative settings where children with tuberculosis can be routinely diagnosed. These settings should also have access to laboratories with quality-assured culture facilities. Both the reference standard and index tests should be administered simultaneously, which is particularly important in children at high risk of rapid disease progression.

STANDARDIZED APPROACH TO PROCEDURES, DEFINITIONS, AND DATA COLLECTION

As previously indicated, childhood tuberculosis diagnostic studies are characterized by heterogeneous methodologies (see [3]). Participants at the workshop reinforced previous calls for standardized approaches for all aspects of diagnostic studies [3, 24, 25], including laboratory and clinical assessment procedures and data reporting. Standard operating procedures should govern all aspects of laboratory methodologies, including sample collection, processing, transport and storage, and laboratory procedures for specific diagnostic tests and should include procedures for quality assurance. The study protocol should include detailed guidance on standard operating procedures for uniform collection of all data required in the clinical case definition (eg, chest radiography, documented exposure, response to treatment) and for obtaining microbiological confirmation (type of samples to be collected, number of cultures, and choice and sequence of laboratory tests). Patients should be classified objectively into the proposed diagnostic categories via independent systematic review, using all data collected as specified per protocol. To ensure consistent and reproducible application of the criteria, this categorization should be retrospectively reviewed by an independent reviewer(s) unaware of the initial classification. In all phases of evaluation, tests should receive blinded interpretation with regard to each other. Standard guidelines should be followed for data reporting including those outlined in the Standards for the Reporting of Diagnostic Accuracy Studies [26]. Reporting should include inconclusive results of all candidate tests performed and, as much as possible, stratification by age, HIV status, and clinical severity.

NEXT STEPS AND FUTURE DIRECTIONS

Developing standardized approaches for the evaluation of tuberculosis diagnostics among pediatric populations represents a critical first step for establishing the evidence base required to improve the detection and confirmation of tuberculosis in children, and for ensuring that the benefits of technological advances in tuberculosis diagnosis are realized for children as well as adults. The consensus process we undertook to develop these approaches highlighted the paucity of available data for the diagnosis of tuberculosis in children, the inadequacy of the current reference standard for this population, and the lack of comparability in published studies due to variability in methodologies, case definitions, and collection and reporting of data. Going forward, there is an urgent need for the wide dissemination and adoption of standard methodologies as outlined and for standardized collection of data across future studies to guide evolution of these case definitions and methodologies. It will also be important that the approaches and methodologies we have proposed are adapted based on scientific advances that may lead the field of tuberculosis diagnostics in directions not addressed here. These consensus documents should be viewed as living tools that will be refined and honed as the evidence base grows and the field of tuberculosis diagnostics expands.

These documents are also subject to a number of limitations. The scope of this initial work prioritized addressing the pediatric populations in whom confirmation of tuberculosis is the most challenging (infants and children <10 years) and focused on intrathoracic tuberculosis and the selection of study populations with higher likelihoods of having laboratory-confirmed disease (ie, those with signs/symptoms suggestive of tuberculosis). A number of critical areas not addressed include the evaluation of tuberculosis diagnostics among older children, adolescents, and populations that represent the full spectrum of childhood tuberculosis, including asymptomatic child contacts of infectious adult cases and children with extrathoracic tuberculosis. Additional work is also necessary to further specify appropriate and inappropriate study designs, the steps to identify eligible cases, how to handle any missing information needed for the case definition, and other methodological issues. Finally, many aspects of a statistical analysis plan are not directly addressed. For example, these include how to handle missing data and stratification. Each study should clearly describe their approach so that comparisons across studies can be made.

The potential impact of the efforts presented here will only be realized if there is wide distribution, broad endorsement, and uptake of these standardized approaches. Substantial, ongoing efforts and collaboration to collect and utilize standardized data to inform future revisions of these documents are essential. We believe this is an important first step toward ensuring greater rigor and comparability in the field of pediatric tuberculosis diagnostic research, realizing the promise of better tests to confirm the diagnosis of tuberculosis in children and adolescents and reducing morbidity and mortality from this preventable and curable disease.