Mtb-Specific Interferon-Gamma ELISpot Assays Have Greater Sensitivity for Detecting Mtb Infection Compared to Tuberculin Skin Test in a Cohort of Experimentally M. tuberculosis-Infected Macaques

Abstract

Tuberculosis outbreaks in nonhuman primate (NHP) colonies can result in disastrous financial, scientific, animal welfare, and public health outcomes. Tuberculin skin testing (TST) has been the standard diagnostic method for identifying Mycobacterium tuberculosis (Mtb) infection in NHPs but has limited sensitivity and specificity. In humans, Mtb-specific interferon-γ release assays (IGRAs) have become the preferred diagnostic method, overcoming significant limitations of the TST. We compared the diagnostic qualities of the TST and IGRA in a cohort of 673 rhesus and cynomolgus macaques with experimental Mtb infection. Serial TST and IGRA data were curated retrospectively for animals before and after Mtb infection and temporally matched to assess sensitivity, specificity, and agreement between the 2 tests. The overall sensitivity of the IGRA was 0.84 compared with 0.49 sensitivity of the TST test in detecting Mtb infection. Specificity of the IGRA and the TST was 0.83 and 1.0, respectively, with an overall agreement rate of 72% between the 2 tests. Sensitivity peaked at 3-5 weeks postinfection for TST and 5-7 weeks for IGRA. Data from a second cohort of animals (n = 227) that were not experimentally infected with Mtb, but for whom testing was done for screening and investigational purposes, demonstrated an agreement rate of 96%. To our knowledge, this is the largest reported comparison of TST and IGRA testing in NHPs. Although IGRAs are more labor intensive than TSTs, our data suggest that IGRAs have better diagnostic accuracy in detecting Mtb infection. Our data also suggest that the threshold for a positive TST could be lowered to improve sensitivity.

Introduction

Tuberculosis (TB) is a well-recognized zoonosis. Outbreaks of TB in nonhuman primate (NHP) colonies can negatively impact animal welfare and lead to significant loss of animal resources, finances, and scientific data. Accordingly, serial screening (in quarantine and then at semiannual to annual time points) of NHPs is considered the standard veterinary practice among breeding, research, and veterinary institutions.^1,2 While NHPs do not respond to purified protein derivative (PPD), skin responses to injected mammalian old tuberculin (MOT) (also called tuberculin skin testing [TST]) have been used for screening and diagnosis of Mycobacterium tuberculosis (Mtb) infection in NHPs for decades.¹ However, there are disadvantages of TSTs including the need to reexamine the injection site at 24, 48, and 72 hours, the subjective nature of measuring and interpreting test results, suboptimal accuracy and reliability, inconsistent serial TST results, potential cross-reactivity with other mycobacterial species, and unpredictable availability of MOT due to periodic shortages in supply.^3–7 Diagnostics for human Mtb infection are similarly challenging as TST (using PPD) requires multiple measurements that can be subjective, has limited sensitivity, and is prone to false positive results for those vaccinated with Bacillus Calmette-Guérin (BCG) or infected with nontuberculous Mycobacterium species.⁸ Over the last decade, interferon-γ release assay strategies (IGRAs) have become the preferred human Mtb diagnostic method over TST as they are less subjective, more specific for Mtb infection without false positives to nontuberculous mycobacteria or BCG vaccine, and have good sensitivity and specificity when used in high-risk populations.^9–11 In a similar fashion, NHP TB testing has paralleled human efforts to optimize diagnostic methods that include the use of lymphocyte proliferation assay,^4,7 assays that detect antibodies to highly immunogenic secreted proteins unique to Mtb complex, as well as IGRA strategies.^12,13 Many of these immune-based assays require laboratory expertise that can be a logistical and technical challenge. In NHPs, direct comparisons between the TST and IGRA have been limited to outbreaks with relatively small numbers of infected animals in each cohort. Confirmed cases were typically in NHPs exhibiting an overt, high burden of disease, but less severe or subclinical cases of Mtb infection could have been missed.^12,14,15

Few studies have systematically compared the diagnostic performance of the TST to another diagnostic modality with precise knowledge of Mtb infection status. Using retrospective data collected from over 20 years of studying Mtb in NHP models, we took advantage of this dataset to determine the accuracy, reproducibility, and timing of conversion from negative to positive after experimental Mtb infection for TST and IGRA. Here, we show that the IGRA, using peripheral blood mononuclear cell (PBMC) enzyme-linked immunospot (ELISpot), is more sensitive than the TST; however, the accuracy of these tests varies considerably based on the time between the test and when the animal was infected.

Ethical Review

Data from 2 cohorts of animals were used for these studies. For the animals at the University of Pittsburgh, this work was conducted by trained staff in compliance with the principles of the Guide for the Care and Use of Laboratory Animals¹⁶ with approval from the University’s IACUC. The University of Pittsburgh program is fully accredited by AAALAC International, adhering to the national guidelines established in the Animal Welfare Act¹⁷ as mandated by the US Public Health Service Policy (PHS D16-00118).¹⁸ For the animals at Bioculture Mauritius’ AAALAC-accredited program, this work was conducted by trained staff, in compliance with the Guide for the Care and Use of Laboratory Animals¹⁶ and the ETS123¹⁹ recommendations.

Materials and Methods

Study design and sample size.

This was an observational retrospective study using historical data from 2 different cohorts: 673 NHPs from the University of Pittsburgh and 227 NHPs from Bioculture Mauritius for a total of 900 animals. At the University of Pittsburgh, serial TST and ELISpot data were obtained before and after experimental Mtb infection and retrospectively compiled to determine the testing accuracy of each test. Animals from Bioculture were not experimentally infected but TST and ELISpot tests were performed either for screening purposes or investigation after exposure to confirmed TB cases. These data were also retrospectively obtained. Given the “real world” scenario of this latter cohort, data from this site were only used to assess agreement between TST and ELISpot.

At the University of Pittsburgh, data were compiled from multiple studies spanning nearly 2 decades. These data contained 782 ELISpot tests (from 356 animals) conducted from May 2004 to August 2016 and 2,154 TST tests (from 633 animals) conducted from October 2001 to September 2020. On average, each animal had 2.2 ELISpot tests (range: 1-13 tests per NHP) and 3.4 TST tests (range: 1-16 tests per NHP). Of the 782 ELISpot tests, 355 were collected preinfection, and 427 were collected postinfection. Of the 2,154 TST tests, 781 were performed preinfection and 1,373 were performed postinfection. For sensitivity and specificity calculations, animals were defined as having Mtb infection on the day after experimental Mtb infection, while animals were considered uninfected before experimental infection. Mtb infection was confirmed by PET CT imaging and/or histopathologic or microbiologic evidence of TB disease at necropsy.

At Bioculture in Mauritius, data were compiled from paired tests conducted from May 2020 to October 2023 (227 ELISpot and TST tests from 227 animals, one of each test per NHP).

Data from both cohorts (University of Pittsburgh and Bioculture) were used to assess test agreement. To assess agreement between the ELISpot and TST tests, an animal was required to have both tests at a similar time point. For tests conducted after Mtb infection, only tests within 14 days of each other were considered paired tests. For tests conducted in uninfected NHPs, tests within 90 days of each other were considered paired tests. These criteria were used to pair tests in both cohorts. In the University of Pittsburgh data, only 212 animals had concurrent tests contributing 260 paired tests; in the Bioculture cohort, 227 NHPs had paired ELISpot and TST tests contributing one pair per animal, 227 paired tests.

No formal power analysis was conducted; sample size was determined opportunistically, using animals for which we had existing data.

Inclusion and exclusion criteria.

To reduce any potential confounding factors, animals in vaccine studies, TB drug treatment, or those who underwent immune suppression (eg, simian immunodeficiency virus [SIV] and TNF neutralization) that might impair the ability of animals to mount an appropriate mycobacterial response were excluded among the University of Pittsburgh cohort. After screening 882 animals for exclusion criteria, 673 animals remained and were included in the analysis.

Animals from Bioculture were not experimentally infected and were only used to test for agreement between the 2 diagnostic tests. Only animals with both tests within 90 days of each other were included for this study (227 included).

Randomization.

There was no randomization performed in this study. As an observational study, animals were already predefined into groups, as preinfection and/or postinfection, based on the timing of either TST or ELISpot tests performed.

Blinding.

For both cohorts and before any analysis, test scores were compiled without regard to test date, infection date, inclusion, or exclusion criteria.

Experimental animals.

This retrospective study was comprised of 2 cohorts. The larger cohort from the University of Pittsburgh was used to compare both the diagnostic accuracy of TST and ELISpot testing and the agreement between the 2 tests. Archived data from serial TST and ELISpots collected either before or after experimental Mtb infection were analyzed from 673 NHPs. The majority of the animals were cynomolgus macaques (Macaca fascicularis) from China (n = 599 [89%]) or Mauritius (n = 57 [8.5%]). Fewer than 3% of the animals were rhesus macaques (Macaca mulatta) (n = 17 [2.5%]). Animals were mostly male (67.8% of the total NHPs) with 16% of unknown sex. Animals for which age was available (n = 571) were on average 6.7 years old (range: 1.4-13.1 years) (Table S1 ^{(18.9KB, docx)} ).

Before arrival at the University of Pittsburgh location, animals were screened for Strongyloides, hookworm, and whipworm and treated with antihelminthic treatment if any NHPs in the cohort were positive. Animals were also prescreened for measles, Mtb, simian retrovirus (SRV), SIV, simian T-cell lymphotropic virus 1 (STLV-1), and herpes B virus (Macacine herpesvirus 1); only SPF animals were accepted into the facility. Two negative TSTs, 30 days apart, were also required before entry into the quarantine facility. At the University of Pittsburgh, all NHPs were housed in accordance with the Guide for the Care and Use of Laboratory Animals and single or pair-housed in an AAALAC-accredited facility. Animals were maintained on a 12 hour light/dark cycle at 21.1 °C and 30%-40% humidity. Water was available ad libitum, and a standard primate diet supplemented with daily enrichment was provided. The main diet for the NHPs consisted of Purina 5038 Primate Biscuits (Purina LabDiet, St. Louis, MO) with fresh fruits and vegetables provided twice daily. An edible forage mix consisting of food-stuff items such as cereals, nuts, and dried vegetables/fruits was also provided daily. Environmental enrichment included structural components (eg, perches, swings), foraging devices, and daily rotation of toys. Enhanced enrichment consisted of an edible item (eg, cereals, noodles) and a destructible item (eg, a paper bag) provided 3 times a week.

The Bioculture cohort included 227 Mauritian cynomolgus macaques (M. fascicularis) with both TST and ELISpot test results. Animals in this facility were screened and confirmed to be free from SIV, SRV, STLV-1, human papillomaviruses, rabies, and malaria. NHPs in this cohort were not experimentally infected with Mtb. Data from these animals were only used to assess agreement between the 2 tests. On average, the animals were 10.3 years old (range: 3.1-20.4 years). Animals were mostly female (59.9%). These NHPs were group housed in an outdoor facility with regularly provided enrichment that was age specific. No specific temperature, humidity, or lighting parameters were controlled given the outdoor nature of the facility. Water was provided ad libitum, and standard primate diet supplementation was provided. Dry biscuits were custom-manufactured locally, and NHPs were fed fresh local vegetables and fruits once daily.

Experimental procedures.

TSTs were conducted by veterinary staff at the University of Pittsburgh and Bioculture Mauritius using 0.1 mL of MOT injected intradermally into the eyelid. The injection site was evaluated after 24, 48, and 72 hours. The skin reaction was then scored from 0 to 5 based on the degree of edema, swelling, and drooping of the eyelid (Table S2 ^{(17.9KB, docx)} ). TSTs were read by experienced veterinary technicians at both the University of Pittsburgh and Bioculture. The University of Pittsburgh MOT was sourced from National Logistics Services Animal Health of Montgomery, AL, now called Covetrus of Portland, ME; Bioculture MOT was sourced from Charles Calleja, Fondateur-DG at EuropExport of Biot, Sophia Antipolis, France. TST scores of 4 and 5 were considered positive, and scores of 1 and 2 were considered negative; a score of 3 (ie, mild swelling and erythema) was defined as indeterminate, or uncertain. All TST measurements were compiled, and the highest value from either 48 or 72 hours was used as the score for that time point.

Mtb-specific PBMC IFN-γ ELISpot assays for both sites were performed at the University of Pittsburgh as previously published.²⁰ In brief, fresh or cryopreserved PBMCs were incubated with phorbol 12,13-dibutyrate and ionomycin (P + I) as a positive control, media only (background, negative control), and Mtb-specific antigens (ESAT-6 peptide pools and CFP-10 peptide pools), and production of IFN-γ was measured by ELISpot in the form of spot-forming units (SFU) per 200,000 PBMCs. Samples in which the positive control (P + I) responses were less than 300 SFU or media-plus cells responses were greater than 200 SFU (high background) were excluded. Responses from nonstimulated cells were subtracted from Mtb-specific antigen stimulation wells (ESAT-6 or CFP-10) as previously established.^20,21 In samples for which ESAT-6 and CFP-10 were not in the same well, a summation was calculated from background-subtracted wells. A positive ELISpot test result was defined as a combination score greater than 10 SFU for ESAT-6 and CFP-10 peptide pools.²²

Preinfection testing was performed when animals entered quarantine at the University of Pittsburgh. Per University of Pittsburgh guidelines, domestic animals were quarantined for approximately 35 days with TSTs performed at 1, 3, and 5 weeks after facility entry. NHPs in the US for less than 1 year had an extended quarantine period lasting 63 days with an additional TST performed at 7 and 9 weeks after entry. Preinfection ELISpots were obtained anytime during or after quarantine but before Mtb infection. Animals were infected with Mtb (strain Erdman) via bronchoscopic instillation (99.1%) or aerosolization (0.9%) as previously published.⁴ The median infectious dose of Mtb was 12 CFU (range: 0-486 CFU), so the majority of infections were considered low dose (<50 CFU; 61%) (Table S1 ^{(18.9KB, docx)} ).

Blood draws for ELISpot tests from the Bioculture cohort were sent to the University of Pittsburgh for analysis in 4 batches, with dates ranging from August 2020 to October 2023. ELISpots performed in 2020-2022 (n = 175) were for screening purposes. In contrast, ELISpots results from 2023 (n = 52) were conducted as part of an investigation including 5 confirmed cases of TB and the remaining 47 with possible exposure. TB disease was confirmed by pathologic and microbiologic findings at necropsy, although the timing of infection, extent of exposure, and other details are not available.

Outcome measures.

The outcome measures for diagnostic accuracy of the TST and IGRA include scores for the tests, with preset thresholds defining negative or positive, and whether the animal was infected at the time the test was performed, considered infected, or not infected. These values were used to calculate sensitivity and specificity to detect Mtb infection and agreement between TST and IGRA.

Statistical methods.

Data were organized and collated in Excel (Microsoft Excel for Mac, Version 16.89) and R Statistical Software (Version 4.4.1, 2024-06-14) using the tidyverse, DT, shiny, readxl, and bslib packages. Counts, descriptive statistics, receiver operating characteristic (ROC) curves, and κ were calculated in JMP (JMP Student Edition 18.2.2). Sensitivity and specificity were calculated in RStudio (2025.050, Build 496) with R Statistical Software (Version 4.5.0, 2025-04-11) using the epiR package.

For overall sensitivity and specificity, serial tests of 673 animals from the University of Pittsburgh were analyzed (782 ELISpot tests and 2,154 TST tests total). ELISpot scores of greater than 10 SFU and TST scores of 4 or greater were considered positive.^4,22,23 When calculating sensitivity and specificity for TST tests, indeterminate TST scores (TST = 3) were not included; they did not count as either positives or negatives in the counts. For both ELISpot and TST, tests were also grouped together by time point (relative to Mtb infection), and sensitivity was calculated for multiple ranges of time points postinfection. For this temporal sensitivity analysis, each animal was only included once at each time point. If an animal had both a negative test and a positive test within the same time duration, the positive test was used in the analysis. Sensitivity and specificity were also calculated and reported for the tests stratified by species, sex, age, and infection dose.

Agreement between the 2 tests was assessed using both cohorts. Two tests were considered in agreement if they were simultaneously positive or negative. As this was a retrospective study and TST and ELISpots were not deliberately administered at the same time, we considered concurrent tests within 90 days of each other before infection and within 14 days of each other after Mtb infection. If tests did not meet the above conditions, they were excluded from the analysis. For the University of Pittsburgh tests, only 260 test pairs (51 preinfection and 209 postinfection from a total of 212 animals) satisfied these conditions. For the Bioculture tests, 227 pairs from 227 NHPs satisfied these conditions. Agreement was summarized by reporting the percentage of paired ELISpot and TST tests with the same results.

Test diagnostic accuracy was measured for both ELISpot and TST tests using ROC curves. An ROC curve is a visual tool used to measure the efficacy of a binary classification model, in this case, a positive and negative test result. The AUC provides a value that measures the classification: an AUC of 1 is a perfect diagnostic (ie, the test is positive with all true positive cases), and an AUC of 0.5 is considered only as good as random chance.

Results

Given the retrospective nature of the study, samples in the University of Pittsburgh that fit the criteria for analysis included 355 preinfection and 427 postinfection ELISpot tests (median: 6.6 weeks postinfection; range: 1-113 weeks) and 781 preinfection and 1373 postinfection TST tests (median: 5.3 weeks postinfection; range: 1.4-172 weeks) (Table S3 ^{(16.2KB, docx)} ).

Our data from the University of Pittsburgh cohort, in which testing data were obtained from a highly controlled experimental Mtb infection scenario, suggest that the overall sensitivity of Mtb-specific ELISpots is greater than that of TSTs in detecting Mtb infection. The overall sensitivity of the Mtb-specific ELISpot tests was 0.838 (95% CI: 0.80, 0.87), and specificity was 0.828 (95% CI: 0.78, 0.87) (Table 1). This specificity implies that 17.2% of tests were false positives. There were 61 animals with false positive ELISpot scores; 6 of which had discordant scores before Mtb infection. ELISpot sensitivity varied based on the time post-Mtb infection. Sensitivity was very low for tests administered between 0 and 3 weeks postinfection and very high for tests administered between 5 and 7 weeks postinfection (Table 2).

Table 1.

ELISpot and TST Test Sensitivity and Specificity

Test type	Test results		Total tests
	Sensitivity (95% CI)	Specificity (95% CI)
ELISpot tests	0.838 (0.800, 0.872)	0.828 (0.785, 0.866)	782
TST tests	0.485 (0.455, 0.515)	1.00 (0.995, 1.00)	1,903

Sensitivity and specificity with exact 95% CIs reported. For ELISpot tests, 356 NHPs are represented; for TST tests, 633 NHPs are represented. There were 251 TST tests with results of 3 (indeterminate). All animals for this analysis were in the University of Pittsburgh cohort.

Table 2.

ELISpot and TST Results Binned by Time

ELISpot results					TST results
Time postinfection	Total NHP	Percent positive	Sensitivity	Specificity	Time postinfection	Total NHP	Percent positive	Sensitivity	Specificity	Percent indeterminate a
Pre	305	19.0%		0.810	Pre	241	0.0%		0.992	0.8%
0-3 wk	6	16.7%	0.167		0-3 wk	338	9.5%	0.095		8.3%
3-5 wk	19	73.7%	0.737		3-5 wk	338	68.3%	0.683		19.2%
5-7 wk	241	91.7%	0.917		5-7 wk	186	49.5%	0.495		22.0%
7-9 wk	18	77.8%	0.778		7-9 wk	74	24.3%	0.243		21.6%
9-11 wk	6	100.0%	1.000		9-11 wk	8	62.5%	0.625		25.0%
11-13 wk	4	75.0%	0.750		11-13 wk	25	32.0%	0.320		16.0%
13-16 wk	8	87.5%	0.875		13-16 wk	12	41.7%	0.417		16.7%
16-20 wk	10	90.0%	0.900		16-20 wk	44	65.9%	0.659		15.9%
20-24 wk	9	88.9%	0.889		20-24 wk	23	78.3%	0.783		8.7%
24-28 wk	14	92.9%	0.929		24-28 wk	30	33.3%	0.333		40.0%
28-32 wk	6	33.3%	0.333		28-32 wk	28	39.3%	0.393		21.4%
32-36 wk	5	60.0%	0.600		32-36 wk	28	60.7%	0.607		21.4%
36-40 wk	10	90.0%	0.900		36-40 wk	22	40.9%	0.409		13.6%
40-44 wk	9	66.7%	0.667		40-44 wk	18	11.1%	0.111		33.3%
44-48 wk	6	50.0%	0.500		44-48 wk	23	30.4%	0.304		30.4%
48-52 wk	6	50.0%	0.500		48-52 wk	23	47.8%	0.478		13.0%

Animals are denoted only once each timepoint. If an animal had both a negative and a positive test within the same time interval, it was considered positive. Data from University of Pittsburgh cohort.

^a TST score of 3 is “Indeterminate.”

Overall, the sensitivity of the TST was only 0.485 (95% CI: 0.46, 0.51), while specificity was 1.0 (Table 1). There were no false positive TST tests; however, there were 2 “indeterminate” tests scored as a 3 before Mtb infection. Sensitivity of the TST was most robust for tests administered between 3- and 5-week postinfection (0.683). As with ELISpot tests, the sensitivity of TST tests was quite low before 3 weeks postinfection (0.095). TST sensitivity varied widely after 5 weeks postinfection, ranging from as low as 0.111 to as high as 0.783 (Table 2). Nearly 12% of all TST tests were considered “indeterminate,” a score of 3.

Testing results were then stratified by species, origin, sex, age, or infectious dose. Test sensitivity and specificity were similar across both species and sex; however, the sensitivity of TST tests was slightly lower (0.32) in rhesus macaques, although the sample size was small (Table S1 ^{(18.9KB, docx)} ). Juvenile (<4 years old) animals had an overall low sensitivity (0.29), but this was calculated from only 44 tests from 25 juvenile animals (Table S1 ^{(18.9KB, docx)} ). For animals with available Mtb infection dose information, over 60% were considered low-dose infections (<50 CFU). When stratifying by infectious dose, in animals infected with high CFU (>50 CFU), the sensitivity for ELISpot tests was the same as low dose, but the sensitivity for TST was significantly higher. The sensitivity of the ELISpot among animals infected with a high dose was 0.88 (95% CI: 0.78, 0.94) compared with 0.82 (95% CI: 0.76, 0.87) among NHPs infected with a low dose. TST tests in high-dose animals had a sensitivity of 0.56 (95% CI: 0.46, 0.65) compared with a low-dose sensitivity of 0.40 (95% CI: 0.35, 0.45). For animals where we lacked data, CFU dose information was available, ELISpot overall sensitivity was 0.85 (95% CI: 0.78, 0.90), and TST overall sensitivity was 0.53 (95% CI: 0.49, 0.57) (Table S1 ^{(18.9KB, docx)} ).

While the diagnostic power of both ELISpot and TST tests changes over time post-Mtb infection, both yield the highest sensitivity within the first 2 months of Mtb infection (Table 2; Figure 1). To address whether an infected animal had at least one positive diagnostic assay after infection, we found that there was at least one positive ELISpot test observed in 91.4% NHPs after Mtb infection, while 74.9% NHPs had at least one positive TST.

Figure 1.

Percentage of Positive Tests Post-Mtb Infection by Time Point. Red denotes TST tests; blue denotes ELISpot tests. The size of the symbols denotes the number of tests at each of the time points.

Comparing the sensitivity, or percent positive, of the tests in many NHPs over the course of Mtb infection shows that both tests are similarly variable over time, but this comparison does not indicate how often the 2 tests agree. Agreement of ELISpot and TST tests was measured by comparing tests within the same animal and performed at a similar time point. For the University of Pittsburgh animals, only a subset (n = 260) of the total tests used for the sensitivity analysis met the conditions for pairing. In this cohort, 71.5% of all paired tests yield the same result (Table 3). Concordance between the tests was moderate as measured by the Cohen κ (κ = 0.481; McNemar test: P < 0.0001). The tests had the least agreement at 6-8 weeks postinfection (48.4% agreement for 93 tests). This can be explained by the low sensitivity of the skin tests after approximately 7 weeks postinfection (only 0.243 sensitivity between 7 and 9 weeks postinfection; Table 2). For the Bioculture cohort, there was 96.0% agreement between tests (Table 3). Within this cohort, there were 5 confirmed cases of TB whose TST testing was positive but had negative ELISpot assays.

Table 3.

ELISpot and TST Test Agreement at the University of Pittsburgh and Bioculture Mauritius

Tests by site	Agreement (%)	Total tests
University of Pittsburgh tests
All tests	71.5%	260
Pre-Mtb infection	98.0%	51
Post-Mtb infection	67.9%	209
0-4 wk	57.1%	7
4-6 wk	79.2%	101
6-8 wk	48.4%	93
>20 wk	87.5%	8
Bioculture Mauritius tests
All tests	96.0%	227

Agreement is defined by the condition when tests (ELISpot and TST) conducted simultaneously are both positive or both negative. There were 212 animals represented in the University of Pittsburgh cohort; 227 animals were represented in the Bioculture cohort.

We used our data to calculate how these tests could be optimized. Here, ELISpot tests have an AUC of 0.896 (Figure S1A ^{(3.8MB, tif)} ), while TST tests have an AUC of 0.830 (Figure S1B ^{(3.8MB, tif)} ). Based on our previously established threshold of “positive” for the ELISpot test, greater than 10 SFU, sensitivity and specificity are nearly equal (Figure 2A).²¹ When considering all TST scores, a positive test score of 4 or 5 has a sensitivity lower than 0.40 and a specificity of 1 (Figure 2B). In theory, the score that maximizes the difference between the true positive rate and false positive rate (calculated as sensitivity − [1 − specificity]) would be an ideal diagnostic. For the ELISpot test, scores greater than or equal to 13 SFU yield a sensitivity of 0.83 and specificity of 0.85 (Figure 2A). Strikingly, for the TST test, scores as low as 2 yield a sensitivity of 0.70 with a specificity of 0.97 (Figure 2B). Based on this data, changing a TST score of 3 from “indeterminate” to “positive” would increase the sensitivity of the test from 0.485 to 0.578 without significantly reducing specificity (0.997). Thus, our data suggest that the minimum TST score used to diagnose Mtb infection should be lower than 4.

Figure 2.

Sensitivity and Specificity for the Full Range of Test Scores. Sensitivity (in purple) and specificity (in red) are shown for each potential cutoff of (A) ELISpot score (SFU) and (B) TST score.

Discussion

This study is generalizable to human diagnostic challenges where the suboptimal accuracy of the TST in detecting Mtb infection has been a widespread concern. In humans, the purified protein derivative-based TST has limited sensitivity and is prone to false positive results given the cross reactivity to BCG and nontuberculosis Mycobacterium spp. It also requires multiple measurements at 48 and/or 72 hours, which can be subjective.^8,24 To overcome these issues, IGRAs have become the preferred method of diagnosis when possible, as they have greater sensitivity (81%-95%) compared with TST (71%-82%) and better specificity, greater than 95% for IGRA but 60%-97% for TST.²⁵ Like human testing assays, the TST and IGRA can only detect Mtb infection and do not distinguish between latent (asymptomatic) or active disease. We previously published data demonstrating that rhesus and Mauritian cynomolgus macaques are more susceptible to disease and disease severity than Chinese cynomolgus macaques after Mtb infection was established,²⁶ but the purpose of this study was to compare the diagnostic accuracy of the TST and ELISpot to detect infection regardless of infectious outcome. This is particularly important given concerns raised in the NHP literature about the limited diagnostic ability of TSTs to detect latent infection.^12,27

In NHPs, the diagnosis of Mtb infection is a greater challenge compared with humans given the unreliable supply of MOT and limited resources available for microbiologic and clinical testing, particularly those that rely on cell culture techniques.¹² For decades, alternative testing strategies have been available and have been compared with the TST. Previous studies have reported the TST to have a sensitivity of 84% and a specificity of 87% and IGRAs (Primagam assay based on IFN-γ to Mycobacterium bovis/Mycobacterium avium PPD) to have a sensitivity of 68% and a specificity of 97%,^5,6 although these studies had limited numbers of animals in an outbreak setting. A recent study compared the diagnostic capabilities of the TST, using a score of 3 as a positive result, from a variety of different NHPs with suspected TB (n = 153).²⁸ While TST was performed on all the animals as the regulatory CDC standard protocol, only approximately 30% of animals also had proprietary IGRA performed from a commercial laboratory.²⁸ TB cases were confirmed postmortem by Mycobacterium tuberculosis complex culture. In that study, TSTs were reported to have a sensitivity of 94% and specificity of 18%, whereas the IGRA had a sensitivity of 67% and specificity of 45%.²⁸ Important limitations to this study included the lack of systematic testing for analysis, a heterogeneous collection of data in which not all sites conducted postmortem culture with only 30% of animals having non-TST testing, and a range of different types of Mycobacterium infections identified (eg, M. caprae, M. orygis, M. tuberculosis).²⁸ Other Mtb diagnostics in the literature include ESAT-6 antibody test and other similar proprietary tests, but these have not shown sustained success.¹⁴

The quest for an assay that can detect Mtb infection with 100% sensitivity and specificity has evaded both human and NHP diagnostics. In a potential outbreak situation where the time of infection and infectious dose is unknown, our data suggest that ELISpots have greater sensitivity than TST within the first 28 weeks of Mtb infection. The disadvantages of the ELISpot that includes the need for a blood draw, technical expertise to isolate PBMC and conduct the assay, and cost, likely outweigh the risks of missing the TB diagnosis with a less accurate and reliable TST. On the other hand, our data show that using a TST score of 2 as a positive test increases the sensitivity from 0.49 to 0.7 without significantly reducing the specificity. Additional diagnostic modalities may increase the accuracy of detecting Mtb-infected animals. For example, most primate centers have access to chest radiographs and perhaps even CT to identify radiographic patterns consistent with TB. Detection of Mtb from gastric aspirates as a surrogate of sputum, bronchoalveolar lavage, or sterile sites (eg, joint, nodule, brain lesion) can be performed by culture, PCR, or other methods, although it requires a specialized microbiology laboratory resource. Our data from multiple studies indicate that a positive culture from BAL or gastric aspirate confirms infection, but a negative result does not rule out TB based on our experience with hundreds of NHPs that have been experimentally infected with Mtb and confirmed by PET CT imaging and/or necropsy (PLL, M.D. & JLF, Ph.D, unpublished observations). In addition, final culture results for Mtb can take up to 6 weeks, which is an impractical timeframe given the threat of a potential outbreak in an NHP colony. In that setting, a screening test with a higher sensitivity is of greater value than the specificity. The trade-off of having a negative test that rules out infection is considered more important than higher false positive rates. Regardless, each method of testing is likely to have variable sensitivity and specificity that will require rigorous testing in a variety of different clinical situations.

To the best of our knowledge, our data represent the largest number of animals for which serial TST and IGRA testing have been performed and analyzed for diagnostic accuracy and agreement in the context of a highly controlled experimental condition. Knowing the exact date and estimated amount of experimental Mtb inoculum for infection allowed us to analyze the sensitivity, specificity, and agreement of both tests over time. Peak sensitivity of 0.92 for ELISpots was 5-7 weeks after Mtb infection but dropped dramatically by 28 weeks postinfection. The sensitivity of the TST of 0.68 peaked at 3-5 weeks after infection, but over time, the sensitivity and overall consistency of the TST scores became highly variable (Table 2). This pattern is consistent with other reports in which serial TSTs have shown inconsistent results.^4,5,7 Our results are unlike other reports in the literature in which tests are performed only once before necropsy on a small number of animals in an outbreak situation with a variety of diagnostic standards. Our data may be helpful in understanding the accuracy of the TST and ELISpot in a scenario where an estimated time of exposure can be surmised. The agreement rates of 72% and 96% were observed between the University of Pittsburgh and Bioculture cohorts, respectively. These differences are likely to be attributed to the different cohort conditions (housing conditions, environment), proportion and type of Mtb-infected animals (experimental infection compared with natural infection), and frequency and number of tests performed per animal. That said, our data suggest that the increased sensitivity of the ELISpot after Mtb infection might be helpful in clinical situations where there is a high index of suspicion for Mtb infection.

Our study had several important limitations. Given the retrospective nature of this study, the TST specificity of 100% is artificially high. Before infection and during quarantine, all of our animals were TST negative and considered to be TB naïve (“true negatives”). This is a potential bias because these animals had undergone multiple negative TSTs before purchase into our colony. On the other hand, we were able to confirm Mtb infection and determine the diagnostic accuracy of each test easily. Importantly, our results do not reflect the “real life” accuracy of the test such as an outbreak situation in which the duration of infection, dose, degree of exposure, and other variables can influence the positive and negative predictive value of the test. Specifically, the route of infection, that is presumably via aerosol during an outbreak, and the dose and frequency of exposure would be different from a controlled experimental setting. The retrospective nature of this study resulted in limited data points across all time intervals. In particular, there were fewer animals for analysis after 7 weeks postinfection. Some sensitivities were calculated with as few as 4 animals (Table 2), making interpretation less robust. Finally, ELISpots were performed by several different personnel using different lots of antibody and peptides, which could lead to increased variability in the data. That said, our data provide important insights into the timing of when TST and ELISpots convert from negative to positive and their testing accuracy in controlled conditions.

Supplementary Materials

Figure S1. ^{(3.8MB, tif)} Receiver operating characteristic (ROC) curves for ELISpot tests (A) and (B) TST tests.

Table S1. ^{(18.9KB, docx)} Description of University of Pittsburgh nonhuman primates (NHPs) with comparative sensitivity and specificity. Species, sex, age, and Mtb infection dose are shown. Sensitivity and specificity with exact 95% CIs reported for ELISpot tests and TST tests by subclassification; n = number.

Table S2. ^{(17.9KB, docx)} TST grading scale. Visual description of the test site and corresponding score.

Table S3. ^{(16.2KB, docx)} Number of tests at the University of Pittsburgh, separated by pre- and post-Mtb infection. The number of tests by infection status is shown. TST tests are broken up as “positive or negative” or “indeterminate score” (score of 3).

Conflict of Interest

The authors have no conflicts of interest to declare.

Funding

This work was supported by the following funding: Bill and Melinda Gates Foundation (to J.L.F. and P.L.L.), AERAS Global Tuberculosis Vaccine Foundation (to J.L.F.), and NIH (AI037859, AI094745, HL110811, AI122811, HL131072, AI123036, HL131072, HL114674, HL075845, AI047485, AI050732, AI123093, AI123036, AI122295, AI114674 [to J.LF.]; AI111871, AI134195 [to P.L.L.]; AI122811, AI142662, AI118195-01A1 [to C.A.S.]; and AI134183 [to J.M.]).

Protocol registration

All NHP procedures and protocols conducted at the University of Pittsburgh were approved by the University’s IACUC.

Data availability

The datasets used in this analysis are available upon request.