Evaluation of the Performance of the Associates of Cape Cod STAT Assay for the Diagnosis of Invasive Fungal Disease in Critical-Care Patients with COVID-19

ABSTRACT

During the COVID-19 pandemic, there have been increasing reports of invasive fungal disease (IFD) in critical care, where rapid access to (1-3)-β-d-glucan (BDG) testing may have enhanced diagnosis. The potential benefit of rapidly accessible BDG results is limited by local availability of BDG testing, with low demand resulting in testing being performed in specialist centers. The recent release of the Associates of Cape Cod STAT assay provides a simple, low-throughput BDG platform, potentially increasing accessibility. During the pandemic, BDG testing using the Fungitell assay (FA) was a critical component of screening for IFD in our critical care. The performance of the STAT was retrospectively determined through a case-control study of 107 serum samples from critical-care COVID-19 patients with IFD defined according to international guidelines. The STAT demonstrated excellent qualitative (observed agreement, 97.2%; kappa, 0.94) and quantitative (Spearman’s coefficient, 0.8962) agreement with the FA. Sample positivity was greater (P < 0.0001) in samples from cases (67.7%) versus controls (6.1%). Using the manufacturer’s threshold (≥1.2), sensitivity and specificity for the detection of proven/probable IFD were 67.9% and 93.9%, respectively. Using a lower positivity threshold of ≥0.87 increased sensitivity to 71.4% without compromising specificity. When the STAT BDG index was >2.86, specificity was 100%. The STAT provides a simple, comparable alternative to the FA for detecting BDG. Sensitivity is moderate, and specificity is excellent for the diagnosis of IFD in the critical-care COVID-19 patient. The potential for enhancing access to BDG testing through the uptake of STAT at centers where FA is not available is beneficial, especially during the COVID-19 pandemic.

INTRODUCTION

The Associates of Cape Cod recently released a new FDA-approved and CE-marked assay to detect (1-3)-β-d-glucan (BDG) in serum. The STAT assay is a simple low-throughput method for testing up to seven patient samples, incorporating a single positive reference sample in place of standards and generating an index value (BGI) rather than a BDG concentration. Initial analysis confirmed qualitative (98%, excluding indeterminate concentrations) and quantitative agreement (Pearson’s r > 0.92) with the original Fungitell assay (FA), demonstrating a linear response to initial BDG concentration and good interassay reproducibility (1). The concept behind the STAT assay was to improve access to BDG testing through methodological simplicity and improving cost-effectiveness by limiting the requirement for high-throughput testing currently evident with the FA.

With the onset of the COVID-19 pandemic and subsequent increasing evidence of secondary invasive fungal disease (IFD; primarily Candida and Aspergillus), rapid access to fungal tests is required given the acute, critical presentation of the patient (2, 3). The detection of biomarkers, such as galactomannan EIA (GM-EIA) in the circulation of nonneutropenic patients, has limited clinical utility. However, the detection of BDG using the FA has shown superior sensitivity for the diagnosis of invasive aspergillosis in the critical-care patient, including COVID-19-associated pulmonary aspergillosis (CAPA) (2, 4, 5). Subsequently, the FA was deemed an integral part of screening for IFD in critically ill COVID-19 patients across Wales (2). With the release of the STAT assay, it was decided to evaluate its performance when testing serum samples from critical-care COVID-19 patients previously tested by FA; the manuscript describes the findings.

MATERIALS AND METHODS

Study design.

Serum samples were selected to represent a range of BDG concentrations as previously determined by the FA. Sixty-five samples had BDG concentrations of <60 pg/ml (FA negative), 15 samples had concentrations between 60 and 199 pg/ml (FA indeterminate/weakly positive), 12 samples had concentrations between 200 to 500 pg/ml (FA moderately/strongly positive), and 15 samples were >500 pg/ml (FA very strongly positive). CAPA was classified according to recent international definitions, whereas fungemia was documented by the recovery of yeast from the bloodstream (6, 7). Culture of yeast from a central venous catheter tip was considered a line infection but not documented as IFD. The original FA result played no role in the classification of IFD. The study was a retrospective, anonymous, case-control performance evaluation, using surplus clinical samples from critical-care patients with PCR-confirmed or clinically suspected COVID-19 patients, with no impact on patient management and not requiring ethical approval. Samples were stored at −80°C for a median of 70 days (range, 19 to 338 days), with 94.4% of samples tested within 147 days of FA testing. Poststorage samples were thawed only once, immediately prior to performing the STAT assay.

Routine Fungitell testing.

The FA was performed following the established manufacturer’s instructions. Each sample was tested in duplicate, with the mean BDG concentration recorded. On no occasion did the coefficient of variation of each sample exceed 20%, eliminating the need for repeat testing. FA positivity was assigned to BDG values of ≥80 pg/ml, with negativity attributed to BDG values <60 pg/ml; BDG concentrations between 60 and 79 pg/ml were considered indeterminate.

Retrospective STAT testing.

STAT was performed following the manufacturer’s instructions, blinded to the original BDG concentration and classification of IFD. Each sample was tested once, in line with the expected routine process. In short, each sample was mixed for 20 s before 50 µl of serum was transferred to an empty vial, and 200 µl of alkaline pretreatment solution was added and mixed. The Fungitell STAT standard was reconstituted using water and pretreatment solution, both volumes dependent on the concentration of the individual batch of standard. All vials were incubated for 10 min at 37°C. The corresponding number of main reagent vials were reconstituted with 300 µl of water and 75 µl of pretreated sample added to the corresponding main reagent. Sample and standard were placed in the kinetic reader in the correct orientation, with kinetic reading performed for 40 min at 37°C. STAT positivity correlated with BGI ≥1.2, with negativity attributed to BGI ≤0.74 and BGI between 0.75 and 1.1 considered indeterminate.

Statistical evaluation.

The clinical accuracy of the STAT assay was determined by comparing positivity rates in samples originating from cases to the false positivity rate in control samples. The clinical performance of the STAT was determined against the classification of IFD using 2 by 2 tables to generate sensitivity/specificity, positive and negative likelihood ratios, and the diagnostic odds ratio. Given the case-control study design and artificially high incidence of IFD (proven/probable IFD, 28.0%; proven/probable/possible IFD, 43%), predictive values were calculated for discussion purposes, using incidences previously documented in this population (CAPA, 6.7%, and candidemia, 6.6%) (8, 9). Qualitative agreement with the FA was calculated by the generation of observed agreement (accuracy) and a kappa statistic. Quantitative agreement was determined by calculating the Spearman’s coefficient between the BGI and paired BDG concentration as generated by the STAT and FA, respectively. Bland-Altman analysis was performed to compare the BDG concentrations generated originally by the FA and retrospectively back calculated from the STAT BGI using the equation for the line of regression. Receiver operator characteristic (ROC) analysis was performed to identify an optimal positivity threshold for the STAT assay. Median BGI values were compared using a Mann-Whitney test. For proportional values, 95% confidence intervals (CIs) were calculated, and Fisher’s exact test used when comparing values. A P value of ≤0.05 was considered significant for all analyses.

RESULTS

One-hundred patients (107 samples) were tested (Table 1), involving 34 samples from 28 cases of CAPA (21 samples from 17 probable patients and 13 samples from 11 possible patients), 9 samples from nine cases of candidemia, 1 sample/case of Rhodotorula fungemia, and 1 sample/case of invasive yeast infection (unidentified yeast in sterile fluid). There were 4 sample/cases of possible IFD (clinical signs typical of IFD, e.g., lung nodules/halos, hepatic/splenic lesions, fungal ball in sinus without etiological specific evidence), 8 samples from 8 Candida line infections, and 50 samples from 49 patients with no evidence of IFD (controls).

TABLE 1.

Comparison of the results of the Associates of Cape Cod Fungitell and STAT assays for the detection of (1-3)-β-d-glucan in serum according to patient population^b

Infection type	No. of patients	No. of samples	STAT assay sample results			Fungitell assay sample results
			No. positive (BGI ≥ 1.2)	No. indeterminate (BGI, 0.75–1.1)	No. negative (BGI ≤ 0.74)	No. positive (≥80 pg/ml)	No. indeterminate (60–79 pg/ml)	No. negative (<60 pg/ml)
Probable CAPA	17	21	14	1	6a	16a	1	4
Possible CAPA	11	13	7	0	6	7	0	6
Proven yeast infection	11	11	7	0	4	7	0	4
Possible IFD	4	4	3	0	1	3	0	1
Candida line infection	8	8	1	0	7	1	0	7
NEF	49	50	3	4	43a	4	3	43
Total	100	107	35	5	67	38	4	65

^aOne sample with documented interference.

^bAbbreviations: BGI: B-d-glucan index; CAPA, COVID-19-associated pulmonary aspergillosis; IFD, invasive fungal disease; NEF, no evidence of fungal infection.

Sample positivity.

Thirty-five of 107 samples were STAT positive, 5 were indeterminate, 65 were negative, and interference was documented for 2 samples with an otherwise negative BGI, 1 of which had interference documented on FA but with a weak positive result (89 pg/ml) (Tables 1 and 2). Sample positivity was significantly greater in proven/probable IFD cases (21/31; 67.7%; 95% CI, 50.1 to 81.4; one sample excluded due to documented assay interference) and possible IFD cases (10/17; 58.8%; 95% CI, 36.0 to 78.4) compared to controls (3/49; 6.1%; 95% CI, 2.1 to 16.5; one sample excluded due to documented assay interference) (P < 0.0001). One of eight samples from patients with a Candida line infection was STAT positive. The median BGI value for samples originating from cases of proven/probable/possible IFD was significantly greater than that for control samples (Mann-Whitney test, P < 0.0001; Table 2). With the exception of Rhodotorula fungemia and unidentified yeast infection, the median BGI values and sample positivity rates for case samples were significantly greater than controls, irrespective of the etiology of infection (Table 2).

TABLE 2.

Sample positivity and median (1-3)-β-d-glucan indices generated by the STAT assay when testing various patient populations^e

Infection type (no. of samples)	Median BGI	BGI range	25% percentile	75% percentile	Sample positivityd (% [95% CI])
Probable CAPA (20)a	1.35	0.10–4.00	0.71	2.22	70.0 (48.1–85.5)
Possible CAPA (13)	2.07	0.10–4.00	0.10	2.81	53.8 (29.1–76.8)
Proven yeast infection (11)	2.24	0.10–4.00	0.32	3.24	63.6 (35.4–84.8)
Possible IFD (4)	3.54	0.41–4.00	1.08	4.00	75.0 (30.1–95.4)
Proven/probable/possible IFD (48)b	1.72	0.10–4.00	0.30	2.85	64.6 (50.4–76.6)
Candida line infection (8)	0.10	0.10–3.42	0.10	0.67	12.5 (2.2–47.1)
NEF (49)c	0.10	0.10–2.83	0.10	0.21	6.1 (2.1−16.5)

^a21 samples from cases of probable CAPA were tested by STAT, but one demonstrated interference and was excluded.

^bForty-nine samples from cases of proven/probable/possible IFD were tested by STAT, but one demonstrated interference and was excluded.

^cFifty samples from control patients with no evidence of fungal infection were tested by STAT, but one demonstrated interference and was excluded.

^dIndeterminate results were considered negative.

^eAbbreviations: BGI: B-d-glucan index; CAPA, COVID-19-associated pulmonary aspergillosis; IFD, invasive fungal disease; NEF, no evidence of fungal infection.

Agreement between the Associates of Cape Cod STAT and Fungitell assays.

Qualitative agreement between STAT and FA was excellent (103/106; 97.2%; 95% CI, 92.0 to 99.0 [one positive sample by FA/interference by STAT was excluded]), generating a kappa statistic of 0.94. Of the three discrepant results, two were weak FA positives (CAPA case, 89, and control, 133 pg/ml) that were STAT indeterminate, and one was FA indeterminate (CAPA case, 68 pg/ml) that was STAT negative. There was significant quantitative agreement between STAT BGI and FA BDG concentration with a Spearman’s coefficient of 0.8962 (P < 0.0001) (Fig. 1). Using the equation for the line of regression, a BGI of 3.0 would correlate with a BDG concentration of 423 pg/ml; the median BDG concentration determined by FA for 6 samples with a STAT BGI between 2.75 and 3.25 was 465 pg/ml, which is comparable given the expected variability of both assays (coefficient of variation, 10 to 20%). Comparing the BDG concentration originally generated by the FA with the back-calculated BDG concentration generated through conversion of the STAT BGI showed minimal bias (Bland-Altman Bias, −0.003245), and 93.5% (100/107) were within the 95% limits of agreement (Fig. 2). For the seven results outside the 95% limits of agreement, the median original FA BDG concentration was 489 pg/ml; all remained positive by STAT with median BGI of 2.29 and a median back-calculated BDG concentration of 326 pg/ml.

FIG 1.

Correlation between (1-3)-β-d-glucan concentration as generated by the Fungitell assay and the (1-3)-β-d-glucan index as generated by the STAT assay when testing 105 (two further samples demonstrated interference) serum samples from patients with COVID-19 screened for invasive fungal disease.

FIG 2.

Bland-Altman analysis of (1-3)-β-d-glucan (BDG) concentrations as determined by the Associates of Cape Cod Fungitell and STAT assays when testing serum. Dotted line represents the 95% limits of agreement.

From a qualitative agreement perspective, by using back-calculated BDG concentrations, the two discrepant FA-positive/STAT-indeterminate results were resolved, but the FA indeterminate/STAT negative became FA indeterminate/STAT positive. A further 11 discrepancies between the FA and STAT were noted when using the BDG concentration as back calculated from the STAT BGI, generating an observed agreement of 88.7% (94/106; 95% CI, 81.3 to 93.4), significantly less than when the agreement between STAT BGI and FA BDG concentration (P = 0.029)

Retrospective clinical performance of the Associates of Cape Cod STAT assay.

Clinical performance for a range of IFD using the manufacturer’s thresholds is shown in Table 3. The STAT sensitivity for combined candidemia/probable CAPA was 73.1% (19/26; 95% CI, 53.9 to 86.3), considering indeterminate results as negative. Subsequent negative likelihood ratios were not sufficient to exclude IFD. Conversely, specificity was high (93.9%), and positive likelihood ratios were consistently sufficient to confirm the presence of IFD when the STAT was positive. When considering indeterminate STAT results as positive (i.e., lowering the positivity threshold to a BGI of 0.75), sensitivity was not significantly improved, albeit at the expense of specificity, and compromised the positive likelihood ratio and diagnostic odds ratio (Table 2). ROC analysis comparing the BGI of all samples from proven/probable IFD cases with those from controls generated an area under the curve of 0.8693 (95% CI, 0.7811 to 0.9575) and indicated a BGI of 0.87 as optimal for defining positivity (Fig. 3). Using this threshold, one additional case of probable CAPA would be STAT positive, increasing sensitivity in line with considering indeterminate results as positive, but without compromising specificity. When the STAT BGI >2.86 specificity was 100%, it was not possible to lower the BGI threshold to provide a sensitivity/negative likelihood ratio sufficient to confidently exclude IFD when negative.

TABLE 3.

Clinical performance of Associates of Cape Cod STAT assay for the diagnosis of invasive fungal disease in critical-care COVID-19 patients^a

Infection type in population vs control (n = 49)	Sensitivity (% [95% CI])		Specificity (% [95% CI])		LR +tive		LR ‐‐‐tive		DOR
	Ind ‐‐‐tive	Ind +tive	Ind ‐‐‐tive	Ind +tive	Ind ‐‐‐tive	Ind +tive	Ind ‐‐‐tive	Ind +tive	Ind ‐‐‐tive	Ind +tive
Probable CAPA (n = 17)	70.6 (46.9–86.7)	76.5 (52.7–90.4)	93.9 (83.5–97.9)	87.8 (75.7–94.3)	11.6	6.3	0.31	0.27	37.0	23.4
Probable/possible CAPA (n = 28)	64.3 (45.8–79.3)	67.9 (49.3–82.1)	93.9 (83.5–97.9)	87.8 (75.7–94.3)	10.5	5.6	0.38	0.37	27.7	15.2
Candidemia (n = 9)	77.8 (45.3–93.7)	77.8 (45.3–93.7)	93.9 (83.5–97.9)	87.8 (75.7–94.3)	12.8	6.4	0.24	0.25	53.9	25.2
Proven yeast infection (n = 11)	63.6 (35.4–84.8)	63.6 (35.4–84.8)	93.9 (83.5–97.9)	87.8 (75.7–94.3)	10.4	5.2	0.39	0.41	26.9	12.6
Proven/probable IFD (n = 28)	67.9 (49.3–82.1)	71.4 (52.9–84.8)	93.9 (83.5–97.9)	87.8 (75.7–94.3)	11.1	5.8	0.34	0.33	32.6	17.7
Proven/probable/possible IFD (n = 43)	65.1 (50.2–77.6)	67.4 (52.5–79.5)	93.9 (83.5–97.9)	87.8 (75.7–94.3)	10.7	5.6	0.37	0.37	28.7	15.3

^aAbbreviations: 95% CI, 95% confidence interval; LR +tive, positive likelihood ratio; LR −tive, negative likelihood ratio; DOR, diagnostic odds ratio; Ind −tive, indeterminate considered negative; Ind +tive, indeterminate considered positive; CAPA, COVID-19-associated pulmonary aspergillosis; IFD, invasive fungal disease.

FIG 3.

Receiver operator characteristic analysis of the STAT assay when testing proven/probable candidosis and CAPA (a), proven/probable invasive fungal disease (b), and proven/probable/possible IFD (c), all versus the control population with no evidence of fungal disease.

DISCUSSION

The Associates of Cape COD STAT assay provides a simple alternative to the established FA for the detection of BDG in serum samples. Both qualitative (96%) and quantitative (Spearman’s coefficient, 0.90) agreement between the assays were excellent, corroborating the results of the previous study (1). For centers with less demand, it makes a sensible alternative to sending samples to specialist reference centers performing the FA, which will inevitably be associated with a delay in results. The concept of a BGI simplifies interpretation, and it correlates with qualitative interpretation according to the FA BDG concentration. While there is a strong quantitative correlation between BGI and original FA BDG concentration, back calculating BDG concentration using the equation of the line generated in this study compromises qualitative agreement between the STAT and FA assays (8.5%; 95% CI, 1.5 to 16.2). A further nine samples were STAT positive based on back-calculated concentrations, two were previously positive by FA, three were indeterminate, and four were negative; five samples originally negative by FA were indeterminate by STAT according to back-calculated concentrations. Of the 12 discordant results, 5 (2 STAT positive/FA negative, 2 STAT indeterminate/FA negative, and 1 STAT positive/FA indeterminate) were from potential cases (1 probable CAPA, 1 proven yeast infection, 1 possible IFD, and 2 Candida line infections), but 7 (3 STAT indeterminate/FA negative, 2 STAT positive/FA indeterminate, and 2 STAT positive/FA negative) were from control patients. The detrimental effect on qualitative agreement with no significant gain in clinical performance suggests that back calculating the BDG concentration using the equation of the line in this study is unwarranted until further large-scale studies are able to provide a more robust means of back calculation.

The clinical performance of STAT for the detection of IFD in COVID-19 patients is comparable to the FA, and for the two main fungal pathogens (Candida and Aspergillus), it is in line with expected performance for BDG assays (Table 2) (10). STAT performance for detecting CAPA is encouraging given that testing is based on serum and other biomarkers (e.g., GM-EIA) provide limited performance when testing this specimen type in this cohort (4, 6, 8). While the STAT specificity in this study is high and its broad detection range beneficial, it can complicate clinical interpretation, given the range of fungal pathogens potentially infecting the COVD-19 patient can require different antifungal therapy. At a CAPA incidence of 6.7% and using the STAT sensitivity/specificity values for probable CAPA described in Table 2, the positive and negative predictive values are 45.2% and 97.8%, respectively. At a candidemia incidence of 6.6% and using the STAT sensitivity/specificity values for candidemia described in Table 2, the positive and negative predictive values are 47.5% and 98.3%, respectively. If a patient is consistently negative by STAT, then IFD caused by Aspergillus or Candida is unlikely (approximately 2% probability) but cannot definitively be excluded based on the negative likelihood ratio (>0.1). Conversely, STAT positivity should be confirmed by repeat testing, and alternative mycological support is required to provide etiological certainty. As with any BDG assay used in this setting, the STAT should be incorporated into a strategic diagnostic pathway involving other etiologically specific assays and a range of sample types (2, 5, 6).

Considering indeterminate STAT results as positives is counterproductive, as the increase in sensitivity is offset by the reduction in specificity, the diagnostics odds ratio consistently indicated that overall assay performance was better when not considering indeterminate results as positive (Table 2). ROC analysis identified a slightly lower BGI positivity threshold of 0.87 as optimal; this improved sensitivity on par with considering indeterminate results as positive, but without compromising specificity (Fig. 3). If the STAT BGI was >2.86, then IFD in the COVID-19 patient could be confidently diagnosed.

Four CAPA patients were falsely negative by both STAT and FA when testing serum. One patient was diagnosed with CAPA due to several bronchoalveolar lavage fluids being positive by GM-EIA (index, 1.1 to 2.0); no blood samples were positive by any fungal biomarker test, indicating disease may have been restricted to the respiratory tract. The three other patients had biomarker positivity other than BDG in respiratory and/or blood samples. Two patients with candidemia were falsely negative by both STAT and FA; one had Candida albicans fungemia diagnosed 7 days post-BDG testing, indicating suboptimal timing, while the other had Candida glabrata fungemia with appropriate BDG testing on and around time of infection. Two patients with unusual or unidentified yeast infection were negative by both STAT and FA. Given the unusual or lack of identification, it is not possible to predict whether these infections would have been expected to be positive for BDG. These findings support the need for strategic testing incorporating multiple (biomarker and conventional mycology) assays on a range of specimen types and spanning the critical-care admission period to enhance the diagnosis of IFD in the COVID-19 patient.

With Pneumocystis pneumonia (PCP) occasionally reported as a secondary complication of COVID-19 infection, and given the excellent sensitivity (>90%) of existing BDG assays for the diagnosis of PCP, it is likely that the STAT assay will provide similar levels of performance (11, 12). Unfortunately, no cases of COVID-19-associated PCP have been documented locally, preventing assessment of the STAT assay for detection of this manifestation. Further limitations of this study include its retrospective case-control design, which can be prone to selection bias that enhances clinical performance statistics. Samples were selected for this study on the basis of FA BDG concentration, selected to provide a range of concentrations and independent of IFD diagnosis that did not involve BDG positivity. In general, repeat sampling was not assessed, so it is not possible to determine whether assay performance would change when testing sequential samples.

Sequential BDG positivity also assists clinical interpretation, increasing specificity and the likely significance of positive results, particularly useful in patients with gastrointestinal disruption and/or post-major surgery where false-positive BDG results have been widely documented (13). With persistent BDG positivity possibly associated with gastrointestinal translocation, and in the absence of IFD documented with worse sequential organ failure assessment (SOFA) score and increased mortality, BDG positivity should be interpreted in the clinical context of each patient (14).

The STAT provides a simple alternative to the FA for the detection of BDG that provides comparable performance while permitting the testing of fewer samples. Sensitivity is moderate, and specificity is excellent for the diagnosis of IFD, particularly CAPA and candidemia, in the critical-care COVID-19 patient, although alternative tests are required to support a diagnosis and provide a genus-level diagnosis. The potential for enhancing access to BDG testing through the wider uptake of STAT at centers where FA is not a viable option is beneficial, especially during the COVID-19 pandemic, and would provide further validation of this novel application (2, 5, 6).