Evaluation of Serum Cryptococcal Antigen Testing Using Two Novel Semiquantitative Lateral Flow Assays in Persons with Cryptococcal Antigenemia

Early cryptococcal disease can be detected via circulating antigen in blood before fulminant meningitis develops, when early antifungal therapy improves survival. Two semiquantitative cryptococcal antigen (CrAg) lateral flow assays (LFAs) have been developed, but their diagnostic performance has not been defined. Cryopreserved serum samples from HIV-infected Ugandans obtained as part of a prospective CrAg-screening cohort were tested in duplicate for CrAg by the CrAgSQ (IMMY) and CryptoPS (Biosynex) lateral flow assays.

ABSTRACT

Early cryptococcal disease can be detected via circulating antigen in blood before fulminant meningitis develops, when early antifungal therapy improves survival. Two semiquantitative cryptococcal antigen (CrAg) lateral flow assays (LFAs) have been developed, but their diagnostic performance has not been defined. Cryopreserved serum samples from HIV-infected Ugandans obtained as part of a prospective CrAg-screening cohort were tested in duplicate for CrAg by the CrAgSQ (IMMY) and CryptoPS (Biosynex) lateral flow assays. Case-controlled diagnostic performance was measured using the FDA-approved CrAg LFA (IMMY) as a reference standard via McNemar’s test. Of 99 serum samples tested, 57 were CrAg positive (CrAg⁺) by the CrAg LFA reference standard. By CrAgSQ, 57 were read as positive, with 98% sensitivity (56/57; 95% confidence interval [CI], 0.91 to 0.99) and 98% specificity (41/42; 95% CI, 0.88 to 0.99) (McNemar’s, P = 0.99). The sample with a false-negative result by CrAgSQ (n = 1) had a titer of <1:5, while the sample with a false-positive result (n = 1) yielded a 1+ result. By CryptoPS, 52 samples were read as positive, with 88% sensitivity (50/57; 95% CI, 0.76 to 0.95) and 95% specificity (40/42; 95% CI, 0.84 to 0.99) (McNemar’s, P = 0.18). The CryptoPS false-negative results included samples with titers of <1:5 (n = 1), 1:5 (n = 5), and 1:20 (n = 1), while samples with false-positive results by CryptoPS (n = 2) yielded Positive results. The CryptoPS assay missed 35% (7/20) of samples with CrAg LFA titers of ≤1:20. The new semiquantitative CrAg LFAs allow rapid estimation of titer levels in easy-to-perform platforms. The CrAgSQ demonstrated better qualitative sensitivity and specificity than the CryptoPS compared to the reference standard. The exact grading of the CrAgSQ results has some subjectivity, with interreader variability; however, qualitative reads were generally concordant for both assays.

INTRODUCTION

Cryptococcal disease continues to disproportionately burden sub-Saharan Africa, where an estimated 73% of cases of cryptococcal antigenemia occur (1). Cryptococcal antigenemia is a risk factor for developing meningitis or death (2–4), and systematic cryptococcal antigen (CrAg) screening of blood samples to guide preemptive therapy is a cost-effective way to save lives (1). Furthermore, plasma CrAg titers predict mortality, with titers of 1:640 or greater conferring ∼50% 6-month mortality in persons with asymptomatic antigenemia (5). Unfortunately, in low-resource settings, reliable access to CrAg testing is limited, and the materials and expertise needed to perform assays of CrAg titers are not usually available.

CrAg testing and titer assays were traditionally performed using latex agglutination or enzyme immunoassay (EIA) testing, though these methods require additional resources, electricity, and technical expertise. CrAg testing has been greatly simplified by the introduction of the lateral flow assay (LFA)—an inexpensive and rapid diagnostic test that has >99% sensitivity and specificity (6). However, determining titers using the LFA still requires ample strips and reagents, along with technical expertise. Given the emerging data linking CrAg titers to disease progression and mortality, the development of a rapid semiquantitative LFA is an attractive option, potentially allowing precise management decisions based on titers to be made at the bedside at a lower cost.

The objective of this diagnostic-accuracy case-control study was to assess the diagnostic performance in serum samples of two new semiquantitative CrAg lateral flow assays, the CryptoPS (Biosynex, Illkirch-Graffenstaden, France) and the CrAgSQ (IMMY, Norman, OK, USA), among HIV-infected persons with CD4⁺ T cell counts of ≤100 cells/μl, using the FDA-approved CrAg LFA (IMMY) as a reference standard.

MATERIALS AND METHODS

Cryopreserved serum samples from 99 HIV-infected Ugandans were tested in parallel for CrAg by the Biosynex CryptoPS and IMMY CrAgSQ lateral flow assays. Samples were collected at the Infectious Diseases Institute in Kampala, Uganda, during two prospective studies: the Operational Research in Cryptococcal Antigen Screening (ORCAS 1.0) study (n = 49), which occurred from July 2012 to December 2014 (7), and a new prospective CrAg-screening cohort that started in 2018 and is presently ongoing (n = 50). Both studies used participant samples that were tested for serum CrAg as part of a screening program based on a prior CD4⁺ T cell count of ≤100 cells/μl. The patient samples tested were included and analyzed regardless of clinical outcome (i.e., whether they developed cryptococcal meningitis or not), although none had diagnosed meningitis at the time of sample collection. CrAg-negative samples were included. All tests were performed on cryopreserved serum samples.

The CryptoPS uses a three-line cassette system and gives results as Negative, Positive, or Strong Positive. The CrAgSQ uses a four-line system on a strip and gives results as Negative, 1+, 2+, 3+, 4+, or 5+. Both assays use an arrangement of test 1 (T1), test 2 (T2), and control (C) lines, with T1 being the first line encountered by the wicking process and C being the last in a vertical orientation. Interpretation tables for each assay are provided in Fig. 1.

FIG 1.

Example images of the CrAgSQ and CryptoPS assays. (A) Example of CrAgSQ strip with an interreader discrepancy, read as either 2+ or 3+. Photo was sent to IMMY who advised that 3+ was the correct reading, based on the line intensity of T1 being greater than that of T2. (B) CryptoPS cassette read as Positive, with clear lines at T1 and C and no line at T2.

The semiquantitative tests were compared against the IMMY CrAg LFA as the reference standard. Historical, prospectively run results for CrAg LFA qualitative reads and titers were available for each sample as a comparison but were not known by the readers at the time of the semiquantitative assay testing. The semiquantitative results were linked with the historical values in a deidentified manner using either the participant’s study identifier (ID) or the laboratory-assigned accession number.

The assays were performed in the CAP (College of American Pathologists)-accredited Infectious Diseases Institute laboratory according to the manufacturers’ instructions. Notably, the CryptoPS calls for 20 μl of sample and 3 drops of diluent, whereas the CrAgSQ requires 40 μl of sample and 1 drop of diluent. Instructions for both tests recommend that the test be read precisely 10 min after initiating the assay. Two individuals experienced in cryptococcal diagnostics—a laboratory technician and a physician—were trained on the new assays and acted as independent, blinded readers. Manufacturer-supplied reference materials were available during the reads. The readers read the tests independently and recorded their interpretations on prespecified data forms. A third reader was available to resolve any interreader discrepancies in real time. Finally, 10 tests were retained for each assay and read at 1 h from the original read time to evaluate test result stability over time.

The data were analyzed using SPSS version 25 (IBM, Armonk, New York, USA). McNemar’s test was used to assess for marginal homogeneity among the paired nominal data. Kappa statistics were used to measure variability of qualitative agreement. We also recorded the real-world costs of purchasing the assays from the manufacturer and shipping them to Uganda.

The parent studies received full institutional review board approval from both Uganda and University of Minnesota regulatory authorities. Informed consent was obtained from all participants, including for storage and future testing of the samples collected. CrAg LFA and BiosynexPS tests were purchased directly from the manufacturers. CrAgSQ tests were donated by IMMY, as this is a nonapproved experimental assay. Neither company provided financial support, input into the study design, data analysis, or interpretation.

RESULTS

Of 99 serum samples tested, 57 were CrAg positive (CrAg⁺) by the CrAg LFA reference standard (FDA-approved assay), based on historical results from assays run in real time during the prospective parent studies. Demographic data were available for 94 participants, demonstrating a cohort with 48 women (51%) and 46 men (49%) with a mean age of 38 (±11) years. The median baseline CD4⁺ T cell count for CrAg⁺ persons (n = 46 with data available) was 20 cells/μl (interquartile range [IQR], 7 to 48). The CrAgSQ demonstrated 98% sensitivity (56/57; 95% confidence interval [CI], 0.91 to 0.99) and 98% specificity (41/42; 95% CI, 0.88 to 0.99) (McNemar’s test, P = 0.99). The overall qualitative agreement was 99% (98/99; kappa = 0.959). The sample with a false-negative result by CrAgSQ (n = 1) had a titer of <1:5; while the sample with a false-positive result (n = 1) yielded a 1+ reading.

The CryptoPS demonstrated 88% sensitivity (50/57; 95% CI, 0.76 to 0.95) and 95% specificity (40/42; 95% CI, 0.84 to 0.99) (McNemar’s test, P = 0.18). The overall agreement was 91% (90/99; kappa = 0.817). The samples with false-negative results by CryptoPS included samples with titers of <1:5 (n = 1), 1:5 (n = 5), and 1:20 (n = 1), while the samples with false-positive results (n = 2) both yielded a Positive reading. Overall, the CryptoPS assay missed 35% (7/20) of specimens with IMMY LFA titers of ≤1:20 and 67% (6/9) of specimens with titers of ≤1:5 as false negatives. Contingency tables for both tests can be found in Table 1.

TABLE 1.

Contingency tables comparing the results of the CrAgSQ and CryptoPS tests and the CrAg LFA reference standard

Test	Result	No. of specimens with indicated result by IMMY CrAg LFA
		Positive	Negative
IMMY CrAgSQa	Positive	56	1
	Negative	1	41
Biosynex CryptoPSb	Positive	50	2
	Negative	7	40

^aThe CrAgSQ had 98% sensitivity (95% CI, 0.91 to 0.99) and 98% specificity (95% CI, 0.88 to 0.99).

^bThe CryptoPS had 88% sensitivity (95% CI, 0.76 to 0.95) and 95% specificity (95% CI, 0.84 to 0.99).

CrAg LFA titers (IMMY CrAg LFA) were available for 54 of the 57 prospectively run CrAg⁺ specimens. The minimum titer was <1:5, and the maximum titer was 1:40,960. The median CrAg titer for all positive samples was 1:40. The median serum CrAg titer for the CrAgSQ at 1+ was 1:5 (n = 8), at 2+ was 1:40 (n = 23), at 3+ was 1:320 (n = 11), and at 4+ was 1:640 (n = 11). Of CrAgSQ 1+ and 2+ specimens, 10% (3/31) had titers of 1:320, and none had titers of ≥1:640.

The median CrAg titer for the CryptoPS at Positive was 1:40 (n = 28) and at Strong Positive was 1:320 (n = 19). Of CryptoPS Positive readings, 25% (7/28) were among specimens with titers of ≥1:320. Thus, the lower Positive indicator of the CryptoPS to represent low titers was less specific than the CrAgSQ indicator. Titers at each semiquantitative mark are summarized in Table 2. Overall, CrAg LFA titers scaled positively with increasing semiquantitative values.

TABLE 2.

CrAg LFA titers for each semiquantitative interpreted value

Novel CrAg assay	Result	No. of specimens	Titer obtained with IMMY CrAg LFAa
			Median	IQR
IMMY CrAgSQ	False negative	1	<1:5
	1+	8	1:5	1:5–1:20
	2+	23	1:40	1:20–1:160
	3+	11	1:320	1:40–1:2,560
	4+	11	1:640	1:80–1:5,120
	5+b	0
Biosynex CryptoPS	False negative	7	1:5	1:5–1:20
	Positive	28	1:40	1:20–1:320
	Strong Positive	19	1:320	1:80–1:5,120

^aThe IMMY CrAg LFA is the U.S. Food and Drug Administration (FDA)-approved reference assay. LFA, lateral flow assay; IQR, interquartile range.

^bNo CrAgSQ 5+ readings were observed in our testing, which would occur if there was a potential prozone effect.

In assessing interreader variability, qualitative reads were 99% (98/99) concordant for CrAgSQ and 96% (95/99) concordant for CryptoPS, with differences in trace readings adjudicated by a third reader. For semiquantitative differences in the degree of positivity, 13 discrepant reads occurred for the CrAgSQ, versus 10 discrepancies for CryptoPS. All the incidents with qualitative concordance but semiquantitative reader discordance varied only by one grade, most commonly on CrAgSQ at 2+ (versus being read as 3+) and on CryptoPS at Strong Positive (versus being read as Positive). Overall, both tests demonstrated some complexity as evidenced by interreader variability, with the CrAgSQ proving slightly more difficult to interpret than the CryptoPS. Fortunately, the qualitative agreement was high, and there was low risk for the reader making a negative interpretation if the test was positive.

Both assays state that the test should be read 10 min after adding the sample and reagent to ensure accuracy of the result. In order to assess “result stability,” we reinterpreted 10 paired tests of each assay at 1 h from the initial interpretation. All 10 CrAgSQ and CryptoPS tests were interpreted with the same semiquantitative results at 1 h as they were at 10 min.

With shipping, the delivered cost was $3.13 for the CrAgSQ versus $3.24 for Biosynex CryptoPS. Shipping costs to Uganda accounted for 4.3% of the total cost of the CrAg LFA and 32% of the total cost of the bulkier CryptoPS assay. These delivered costs exclude laboratory labor, overhead, or any local distributor profit, which is often >100%. Within the cohort, the average number of CrAg LFAs needed to generate titers was 7.3 ± 3.4 LFAs, including the diagnostic LFA. Overall, approximately 449 CrAg LFAs were used in this 99-person cohort at a total assay cost of $956. Conversely, running 99 CrAgSQ or CryptoPS assays would be 3-fold less expensive at ∼$315; however, the CrAgSQ yields improved cost savings due to its preserved sensitivity, whereas the value of the CryptoPS is reduced due to the problem with false-negative results.

DISCUSSION

We assessed the diagnostic performance of two new semiquantitative point-of-care LFA CrAg assays versus the FDA-approved CrAg LFA test. We found that the IMMY CrAgSQ had excellent test performance, with 98% sensitivity and 98% specificity, with differences found in two discordant trace readings. In comparison, the CryptoPS test had worse performance, with 88% sensitivity and 95% specificity. The CryptoPS assay routinely missed serum CrAg-positive specimens with low CrAg titers while still having false-positive results. As CrAg⁺ persons with lower fungal burden have better survival when promptly initiated on antifungal therapy (5, 7, 8), missing an early diagnosis may put a person at higher risk for mortality.

The two semiquantitative assays are constructed very differently. The CrAgSQ is more complicated to read, but the assay preserves analytical sensitivity. In contrast, the two-band CryptoPS assay is easier to read, but the two bands reduce the analytical sensitivity. The IMMY CrAg LFA has a 5-fold better analytical sensitivity for detecting the Cryptococcus glucuronoxylomannan polysaccharide antigen (i.e., CrAg), as the published limit of detection by the IMMY CrAg LFA is 5 ng/ml (9), whereas the CryptoPS package insert claims a 25-ng/ml limit of detection. This difference in analytical sensitivity explains our findings. IMMY previously experimented with a two-band design but found the reduced analytical sensitivity, and therefore redesigned the CrAgSQ assay to preserve the same analytical sensitivity as the original CrAg LFA (Sean Bauman, personal communication).

Detection of antigen at low concentrations depends on the assay’s threshold of detection. However, detection of antigen at very high concentrations can be complicated by the prozone effect, a phenomenon in which excess cryptococcal antigen saturates the binding sites on both the fixed and free-floating antibodies, preventing the antibody-antigen-antibody sandwich formation and colorimetric change from occurring. This effect results in a false-negative test result. The IMMY CrAgSQ is specifically designed to detect a sample experiencing a prozone effect, resulting in a 5+ reading. While we did not explicitly test a known prozone sample, both assays detected all high-titer (i.e., high concentration) samples with no false negatives observed.

The diagnostic performance of the CryptoPS reported herein is worse than that reported from a cohort of 14 CrAg-positive subjects and 172 CrAg-negative subjects in Cameroon (10). The authors defined CrAg status by the IMMY CrAg LFA result but then compared against the Meridian CrAg enzyme immunoassay (EIA) as the reference standard, which is known to have poor sensitivity (11). Specifically, in one cross-validation study, the Meridian CrAg EIA sensitivity was approximately 71%, particularly missing Cryptococcus serotype C strains (11), which are more common in Africa (12). In this prior CryptoPS validation, the authors reported 100% (8/8) sensitivity and 98.3% (175/178) specificity for the CryptoPS assay in comparison to the Meridian CrAg EIA. Ultimately, a novel test should be evaluated against an appropriate reference standard in order to declare sensitivity and specificity.

Our validation study utilized a larger number of CrAg-positive persons across a wide range of CrAg LFA titers, as well as an accepted gold standard for cryptococcal antigen detection, the FDA-approved IMMY CrAg LFA. And yet, there remain limitations. The false positives detected by the CryptoPS should be explored further, as they are not well explained by the reduced sensitivity of the test. The number of controls was relatively small (n = 42), and for financial reasons (in buying the tests), we did not conduct a full validation on the specificity. Having the interreader variability adjudicated by a third reader as a tie breaker is somewhat problematic, as that person is also subject to interpretation variability. Ideally, a validated camera-assisted LFA reader could be used in the future to adjudicate discrepant results. Finally, testing was performed on stored serum, and it is not certain how CrAg detection may change with one freeze-thaw cycle. This freeze-thaw cycle may account for some discrepancies in titer and/or differences in trace positivity.

In conclusion, we present the results of two novel semiquantitative CrAg lateral flow assays tested on serum in persons with cryptococcal antigenemia. The CrAgSQ displayed 98% sensitivity and specificity compared to the results from the reference standard CrAg LFA. In contrast, the CryptoPS did not perform as well, most notably with false negatives, having lower sensitivity (88%) and specificity (95%). A major role for a semiquantitative CrAg assay is to risk stratify the probability of central nervous system (CNS) disease at diagnosis. A CrAgSQ 3+ to 5+ result or a CryptoPS Strong Positive result in blood would be useful for bedside management decisions, and yet, the CrAgSQ is better at distinguishing the 1:160 threshold above which asymptomatic CNS involvement becomes increasingly common (5). A prospective study to establish optimal clinical management or antifungal treatment targeted to CrAg titers would establish a useful niche for these semiquantitative tests.