Performance evaluation of direct fluorescent antibody, Focus Diagnostics Simplexa™ Flu A/B & RSV and multi-parameter customized respiratory Taqman® array card in immunocompromised patients

Highlights

• •Limited sensitivity of direct fluorescent antibody testing in comparison to molecular methods.
• •Simplexa™ direct assay is sensitive and specific for the detection of Flu A, B and RSV.
• •Performing the TAC would increase the diagnostic yield and detection of co-infections significantly.
• •Immunocompromised patient population with overall positivity rate of 62%.
• •Co-infection rate of 15.3% found by TAC.

Abstract

Background

Molecular assays for diagnosis of Flu A, Flu B, and RSV with short turn-around-time (TAT) are of considerable clinical importance. In addition, rapid and accurate diagnosis of a large panel of viral and atypical pathogens can be crucial in immunocompromised patients.

Objectives

First, to evaluate the performance of the Simplexa™ Direct assay system in comparison with direct fluorescent antibody (DFA) and customized Taqman® Array Card (TAC) testing for RSV, Flu A, and Flu B in immunocompromised patients. Second, to evaluate different algorithms for the detection of respiratory pathogens in terms of cost, turn-around-time (TAT) and diagnostic yield.

Study design

We collected 125 nasopharyngeal swabs (NTS) and 25 BAL samples from symptomatic immunocompromised patients. Samples for which Simplexa™ and TAC results were discordant underwent verification testing. The TAC assay is based on singleplex RT-PCR, targeting 24 viruses, 8 bacteria and 2 fungi simultaneously.

Results

The overall sensitivity was significantly lower for DFA testing than for the two molecular methods (p < 0.05). Performance characteristics of Simplexa™ testing were not significantly different compared to TAC testing (p > 0.1). For BAL samples only, the sensitivity and specificity of the Simplexa™ assay was 100%.

In total, 6.7, 16 and 18% of samples were positive for Flu A, Flu B or RSV by DFA, Simplexa™ and TAC testing, respectively. When considering not only these pathogens but also all results for TAC, the method identified 93 samples with one or more respiratory pathogens (62%). A co-infection rate of 15.3% was found by TAC.

The estimated costs and TAT were 8.2€ and 2 h for DFA, 31.8€ and 1.5 h for Simplexa™ and 55€ and 3 h for TAC testing.

Conclusions

Performing the Simplexa™ test 24 h a day/7 days a week instead of DFA would considerably improve the overall sensitivity and time-to-result, albeit at a higher cost generated in the laboratory. Performing the TAC would increase the diagnostic yield and detection of co-infections significantly.

1. Background

Viral respiratory infections are a leading cause of morbidity and mortality, especially during winter months (Mahony et al., 2011). Accurate clinical diagnosis of acute viral respiratory infection is challenging because of overlap of symptoms associated with various viruses and overlap with symptoms associated with other illnesses. Flu A, Flu B and RSV are clinically most important since they are the most frequently encountered, they cause substantial disease burden and a targeted treatment exists. Using a first-line method with a high sensitivity and short TAT to guarantee a result 24 h a day/7 days per week for these pathogens could be of considerable clinical importance. Many commercial as well as in-house developed assays are available, including the Simplexa™ Flu A/B & RSV Direct assay system (FDA-cleared), a real-time RT-PCR system that enables the direct amplification, detection and differentiation of Flu A, Flu B and RSV virus RNA from unprocessed nasopharyngeal swabs (no separate sample extraction steps are required). This system could also be used as a point-of-care test.

In addition, rapid and accurate diagnosis of a large panel of viral and atypical pathogens could be crucial for an appropriate and sometimes life-saving clinical management in specific patient populations such as immunocompromised patients. At the University hospital Erasme, a recent study showed a relatively low positivity rate of routine conventional tests with direct fluorescent antibody (DFA) assays and viral culture for the detection of respiratory viruses in symptomatic immunocompromised patients (Steensels et al., 2015). In addition, more than half of all Flu and RSV positive samples were missed. The study compared conventional testing with a customized Taqman® Array Card (TAC) assay, a microfluidic technology based on singleplex, reverse transcription real-time PCR, targeting 24 viruses, 8 bacteria and 2 fungi simultaneously.

2. Objectives

The goals of this study are, first, to compare the sensitivity and specificity of Simplexa™ for the detection of RSV, Flu A, and Flu B in comparison to DFA, and TAC testing. Second, to compare the cost, turn-around-time (TAT) and diagnostic yield of different algorithms for the detection of respiratory pathogens in this group of immunocompromised patients. And finally, we aim to test the Simplexa™ assay on BAL samples, which has only been validated on nasopharyngeal swabs so far.

3. Study design

3.1. Patients and samples

This study was approved by the ethical committee of the Erasme Hospital. In parallel with previously published study performed during winter season 2014–2015 (Steensels et al., 2015), we included adult immunocompromised patients with symptoms of an upper or lower respiratory tract infection from November 2015 to April 2016. Based on review of the electronic medical records, subjects were enrolled in five different patient groups according to underlying conditions. For detailed composition of disease groups, see Table 1 . After conventional and Simplexa™ testing, the samples were aliquoted and stored at −80 °C until TAC testing.

Table 1.

Baseline characteristics of the 150 patients from whom respiratory samples were collected.

Characteristic	Value
Age, Median (range), Yrs.	58 (18, 86)
Gender, Male, No. (%)	81 (54.0)
Underlying condition, No. (%)
Solid organ transplantation	70 (46.7)
Solid malignancy	27 (18.0)
Hematological malignancy	28 (18.7)
Other underlying disease needing long-term corticosteroids therapy or immunosuppressive therapy	24 (16.0)
HIV CD4 < 200/mm3	1 (0.67)
Type of solid organ transplant, No. (%)
Lung	34 (22.7)
Kidney	17 (11.3)
Liver	10 (6.7)
Heart	3 (2.0)
Combined*	6 (4.0)

^*lung + kidney (n = 2), lung + heart (n = 2), kidney + heart (n = 1), lung + kidney + heart (n = 1).

3.2. Conventional testing (prospective)

DFA respiratory virus tests and viral culture were performed as described previously (Steensels et al., 2015).

3.3. Simplexa™ Flu A/B & RSV testing (prospective)

The Simplexa™ Flu A/B & RSV (Focus Diagnostics, Cypress, CA) testing was performed according to manufacturer guidelines, on unprocessed NTS and BAL samples, on the same day as the routine conventional testing. In the case of an invalid result, the sample was diluted 1:2 and rerun. The Simplexa™ assay does not include a sample quality control.

3.4. Nucleic acid extraction and TAC testing

Patient samples were retrospectively tested with a customized TAC respiratory panel (v10.0 premarket version Cambridge-Brugge) which included testing for the pathogens shown in Figure 1 (Supplementary file). Two human DNA genes are included as cellular (sample quality) controls. Nucleic acids were extracted on the QiaSymphony (Qiagen, Valencia, CA) using the DSP viral pathogen midi kit and the TACs were run on QuantStudio 7 (Thermo Fisher Scientific, Life Technologies, Carlsbad, CA). The detailed procedure was described previously (Steensels et al., 2015).

3.5. Verification testing

A consensus of the two molecular tests was considered as the golden standard. Discordant Simplexa™ and TAC assay results were verified using the multiplex real-time PCR assay FTD Flu/HRSV (Fast-track Diagnostics) according to manufacturing guidelines, on the nucleic acid extract that was used for TAC testing. This confirmatory test was chosen as an independent molecular test for Flu A, Flu B and RSV which targets different genes from those targeted by Simplexa™ and TAC.

3.6. Statistical analysis

DFA, Simplexa™ and TAC testing were compared using the exact two-sided McNemar's test. All statistical analyses were performed using the MedCalc software (Mariakerke, Belgium).

3.7. Estimation of costs and turn-around-time

Total cost per sample and cost per pathogen were calculated, including all necessary consumables and personnel costs. The cost of consumables was based on negotiated prices for our laboratory. In contrast to official list prices, these prices reflect the reality and will be taken into account by laboratories in their ultimate choice between different methods. For each technique, three different lab technicians were timed during sample manipulation on a subset of samples included in the study. The mean hands-on time was used to calculate personnel costs, and the mean time to result was used to estimate turn-around-time.

4. Results

One hundred-twenty five nose–throat samples (83.3%) and 25 (16.7%) BAL samples were obtained from 150 immunocompromised patients who underwent respiratory pathogen testing for symptoms of upper or lower respiratory tract infection. Patient characteristics are shown in Table 1.

Table 2 shows the performance characteristics of DFA, Simplexa™ and TAC testing for RSV, Flu A and Flu B. As expected, the overall sensitivity was significantly lower for DFA testing than for the two molecular methods, even when the 42 inadequate samples were excluded from the analysis (p  = 0.0044). These 42 samples were uninterpretable by DFA due to insufficient cells derived from the respiratory sample. However, when considering results for each pathogen separately, the difference in sensitivity between DFA and molecular methods was only statistically significant for Flu A.

Table 2.

Performance characteristics of conventional, Simplexa™ Flu A/B & RSV Direct assay system and customized TAC assay.

	RSV			Influenza A			Influenza B			Overall
Test	No. Positive	Sensitivitẙ (% [95% CI])	Specificity (% [95% CI])	No. Positive	Sensitivitẙ (% [95% CI])	Specificity (% [95% CI])	No. Positive	Sensitivitẙ (% [95% CI])	Specificity (% [95% CI])	No. Positive	Sensitivitẙ (% [95% CI])	Specificity (% [95% CI])
DFA*	5	83.3 [35.9–99.6]	100 [97.5–100]	5	29.4 [10.3–56.0]	100 [97.3–100]	0	0.00 [0.00–60.2]	100 [97.5–100]	10	37 [19.4–57.6]	100 [97.1–100]
Simplexa™ Flu A/B & RSV direct assay system	5	83.3 [35.9–99.6]	100 [97.5–100]	17	94.1 [71.3–99.9]	99.3 [95.9–100]	2	50.0 [6.76–93.2]	100 [97.5–100]	24	85.2 [66.3–95.8]	99.2 [95.6–100]
custom TAC	6	100 [54.1–100]	100 [97.5–100]	17	100 [80.5–100]	100 [97.3–100]	4	100 [39.8–100]	100 [97.5–100]	27	100 [87.2–100]	100 [97.1–100]

^*42 of 150 samples tested by DFA were reported as inadequate but were counted here as DFA negative because no results were generated; seven DFA-inadequate samples were positive by Simplexa™, 9 were positive by TAC for RSV, Flu A or Flu B.

° Reference method was consensus of two molecular tests, and multiplex real-time PCR assay FTD Flu/HRSV (Fast-track Diagnostics) for discordant results.

The Simplexa™ direct test missed one RSV, one Flu A and two Flu B positive samples in comparison to the TAC assay and verification PCR. The amplification curves of these Simplexa™ results were completely flat. One sample was found strongly positive for Flu A by Simplexa™ (Ct 15), but was negative by DFA, TAC and verification testing. Unfortunately, there was no sample left to repeat Simplexa™ testing. Nevertheless, the difference in individual and overall sensitivity and specificity was not significant compared to TAC testing (p  > 0.1). Two samples tested as invalid using Simplexa™ due to internal control failure. Since this could be due to presence of an inhibitor, the samples were diluted and rerun, giving valid results. For BAL samples, the sensitivity and specificity of the Simplexa™ assay was 100%.

In total, DFA identified 10 samples (6.7%) with one pathogen each among 150 samples. The Simplexa™ testing found 24 (16%) and TAC 27 (18%) samples positive with one pathogen each. For nose–throat swabs only, DFA, Simplexa™ and TAC testing were positive in 5.6, 15.2 and 17.6%, respectively. For BAL samples, this was 12, 20 and 20%, respectively.

An additional objective of the study was to evaluate these different methods for the detection of respiratory pathogens in terms of diagnostic yield. When considering not only Flu A, B and RSV but all results for the TAC, the method identified 93 samples with one or more respiratory pathogens (62%). TAC testing was positive for other pathogens than RSV, Flu A and B in more than half (54%) of Simplexa™ negative samples. Detailed results are shown in Figure 2 (Supplementary file).

A co-infection rate of 15.3% was found by TAC (21 samples positive for 2 pathogens, 2 samples positive for 4 different pathogens). Those most frequently involved in co-infections were Coronaviruses (n  = 12), HSV (n  = 11), Flu A (n  = 8) and CMV (n  = 5). Out of 24 Simplexa™ positive samples, TAC found 7 samples positive for 2 pathogens (all positive for Flu A plus rhinovirus (n  = 1), coronavirus (n  = 2), HSV (n  = 1), CMV (n  = 1), Aspergillus species (n  = 1), P. jirovecii (n  = 1)).

Furthermore, the estimated costs and turn-around-time of DFA, Simplexa™ and TAC testing were evaluated and are shown in Table 3 . Important to note is that these costs are valid for our laboratory, but not necessarily for another lab since this depends greatly on activity and can vary from one country to another. Remarkably, the TAC technology is the most expensive method in terms of cost per sample, but it is the least expensive in terms of cost per parameter. Even though the technical TAT of the different methods is comparable, the true time-to-result can vary greatly depending on organizational aspects.

Table 3.

Estimated costs (excl. taxes) and TAT of DFA, Simplexa™, and TAC testing.

	Costs/sample		Costs/pathogen	Turn-around-time (h)a
DFA	Consumables	2.4 €		2
	Lab technician	5.8 €
	Total	8.2 €b	2.7 €
Simplexa™	Kit	30.6 €		1.5
	Lab technician	1.2 €
	Total	31.8 €	10.6 €
TAC	Extraction	4.0 €		3
	MasterMix	7.6 €
	Customized card	35.6 €
	Lab technician	7.4 €
	Total	54.6 €	1.6 €c

^aEstimation based on immediate testing, without considering organizational aspects.

^bCosts/sample includes DFA for RSV, Flu A and Flu B.

^cCalculation based on 34 pathogens detectable.

Based on our results, performing a molecular method such as the Simplexa™ or TAC method instead of DFA would be necessary to obtain an acceptable overall sensitivity, albeit at a higher cost generated in the laboratory. Performing the TAC would increase the diagnostic yield and detection of co-infections significantly.

5. Discussion

In line with our previous findings (Steensels et al., 2015), this study confirms the limited sensitivity of direct fluorescent antibody testing in comparison to molecular methods. Although sampling was performed according to consistent procedure, an important number of samples were uninterpretable by DFA which indicates poor sample quality. Sample quality is very important for traditional methods such as DFA because of their lower sensitivity in comparison to molecular methods and since they require the presence of live infected cells and/or live pathogens.

Several papers were published on Simplexa™ Flu A/B & RSV, using an extracted nucleic acid and/or direct sample testing and comparing to conventional methods or molecular tests (Alby et al., 2013, Hindiyeh et al., 2013, Selvaraju et al., 2014, Ko et al., 2013, Woodberry MW1 et al., 2013, Landry and Ferguson, 2014, Svensson et al., 2014). In this study, the Simplexa™ Flu A/B & RSV Direct assay system showed comparable performance characteristics for Flu A, Flu B and RSV in comparison with TAC testing. The four samples missed by Simplexa™ in this study had a low viral load (median cycle threshold of 34), suggesting a slightly lower sensitivity of the assay in comparison with that of the TAC assay. A possible explanation could be the simple lysis of the sample with insufficient viral RNA return compared to a real extraction of the nucleic acids. The TAC technology targets a large panel of viral, bacterial and fungal respiratory pathogens. Logically, the diagnostic yield of this multi-pathogen molecular method was much higher compared to that of DFA and Simplexa™ testing which target only a limited number of pathogens. Also, in almost one-third of the Simplexa™ positive samples, TAC detected Flu A and the presence of an additional pathogen. Respiratory co-infections, in particular, the association of Flu A with (an)other viral, bacterial or fungal pathogen(s), could be associated with longer hospital stay, a longer ICU stay and even higher mortality (Shah et al., 2016, Echenique et al., 2013, Crotty et al., 2015). Detecting these co-infections could have an important clinical impact. Performing a method that allows a syndromic approach on all samples – or at least those coming from specific patient populations such as those included in this study – would be diagnostically valuable. When comparing TAC results from this study with those from previous winter season (Steensels et al., 2015), the overall positivity rate was comparable (62 versus 66.4%).

Although the estimated technical TAT of DFA, Simplexa™ and TAC testing are comparable, the true time-to-result can vary greatly depending on organizational aspects. DFA requires experienced lab technicians and is labor intensive. In our lab, DFA is only performed during regular laboratory operating hours and not in weekends, so it can take up to 2–3 days to have a result if the sample is taken on a Friday evening. On the other hand, the Simplexa™ direct assay system is an all-in-one system (no separate sample extraction steps needed) which requires minimal hands-on-time and results are available in about 1 h 15 min from the start of a test run. The assay is easy to perform, without the need of specialized equipment or experienced molecular lab technicians, so it can easily be performed 24 h a day/7 days per week, and even as point-of-care test. Performing the Simplexa™ assay as first-line test would guarantee a fast result for Flu A, Flu B and RSV, regardless of the sampling timing. Having a fast negative result for these pathogens could also result in cost savings thanks to reduced time spent in isolation, shorter length of inpatient stay and a faster discontinuation of empiric antiviral therapy such as oseltamivir (Pettit et al., 2015). For TAC testing, a separate nucleic acid extraction step is required, as are specialized equipment and lab technicians, and samples cannot be treated individually in the TAC format, so a same-day-result cannot always be guaranteed. Nevertheless, laboratories with a high respiratory sample turn-over could ensure an acceptable TAT, providing their clinicians with a much more complete result.

Even though molecular methods still have a significant higher cost than conventional methods such as DFA, it is important to point out the reasonable cost of the TAC technology considering the large number of agents detectable. The TAC method is actually the least expensive in terms of cost per pathogen. In addition, the gain in sensitivity and diagnostic yield could improve patient outcomes and could reduce overall costs for the hospital, making it cost-effective (Olofsson et al., 2011, Mahony et al., 2009). The added cost of testing with a multi-pathogen panel such as the TAC assay should be weighed against the potential benefits of increased identification of the causative pathogen(s). Even though targeted antiviral treatment not always exists, other treatment options such as antifungals or corticosteroids could be important. In addition, the diagnosis of a viral respiratory infection can potentially reduce the use of antibiotics (Gelfer et al., 2015, Falsey et al., 2013, Rogers et al., 2015), it can assist infection control practitioners in providing appropriate infection control measures (e.g. droplet and/or contact precautions and considerations in creating cohorts) and it can stop the more thorough search for a diagnosis avoiding unnecessary medical procedures. Moreover, co-infections are routinely reported using multi-pathogen molecular assays (Chung et al., 2007, Mahony et al, 2007, Pierangeli et al., 2007). Testing for multiple pathogens in a single assay may also provide a savings of resources including technologist time and expendables and cost less than the aggregate cost of performing multiple uniplex PCR tests. And finally, these tests will contribute to our understanding of the epidemiology and clinical importance of respiratory tract infections. Many commercial as well as in-house developed assays exist today which offer a syndromic approach for the detection of respiratory pathogens. Each system has its advantages and disadvantages, and each user should determine which system is appropriate for their specific laboratory and/or patient population. However, more studies will be needed to determine populations or situations in which these methods would be the most useful in order to optimize their clinical and financial impact (Vallières and Renaud, 2013). Several testing algorithms could be proposed. Ideally, the TAC technology or another comparable multi-pathogen method would be performed on all patient samples with a guaranteed result within 24 h. Another option would be to perform the Simplexa™ assay 24 h a day/7 days a week as a first-line test in winter period for all patients, and to reserve the more complete TAC technology for specific patient populations such as immunocompromised patients, patients in ICU, neonates, etc.

There are some limitations to our study. In contrast to DFA and Simplexa™ testing, TAC and verification testing was performed retrospectively. Therefore, the quality of the samples could have deteriorated due to the freeze–thaw cycle before TAC and verification testing. This study was also limited by the lack of verification testing on all samples. Finally, the cost estimation of the different methods is valid for our laboratory, but would not necessarily be the same in another laboratory.

In summary, the Simplexa™ direct assay system is a sensitive and specific method for the detection of Flu A, B and RSV. Performing a molecular method such as the Simplexa™ test 24 h a day/7 days a week instead of DFA would considerably improve the overall sensitivity and time-to-result, albeit at a higher cost generated in the laboratory. Performing the TAC would increase the diagnostic yield and detection of co-infections significantly, and this in turn could be crucial for appropriate clinical management in special populations such as immunocompromised patients.

Conflict of Interest

None to declare.

Appendix A. Supplementary data

The following are Supplementary data to this article: