Performance and ease of use of a molecular point-of-care test for influenza A/B and RSV in patients presenting to primary care

Abstract

Annual influenza epidemics cause substantial morbidity and mortality, and the majority of patients with influenza-like illness present to primary care. Point-of-care influenza tests could support treatment decisions. It is critical to establish analytic performance of these platforms in real-life patient samples before uptake can be considered. We aimed to assess the analytical performance and ease of use of the cobas® Liat® PCR POCT in detecting influenza A/B and RSV in samples collected from patients with influenza-like illness in primary care. Sensitivity and specificity of the cobas® Liat® POCT are calculated in comparison with a commercial laboratory-based PCR test (Fast-Track Respiratory Pathogens 21 Plus kit (Fast-Track Diagnostics)). Samples with discordant results were analysed additionally by the RespiFinder 2Smart (PathoFinder) using an Extended Gold Standard (EGS). Acceptability was scored on a five-point Likert scale as well as a failure mode analysis of the cobas® Liat® POCT was performed. Nasal and oropharyngeal swabs were obtained from 140 children and nasopharyngeal swabs from 604 adults (744 patients). The cobas® Liat® POCT had a sensitivity and specificity of 100% (95% CI 99–100%) and 98.1% (95%CI 96.3–99%) for influenza A, 100% (95% CI 97.7–100%) and 99.7% (95%CI 98.7–99.9%) for influenza B and 100% (95% CI 87.1–100%) and 99.4% (95%CI 98.6–99.8%) for RSV, respectively. According to trained lab technicians, the cobas® Liat® POCT was considered easy-to-use, with a fast turn-around-time. Cobas® Liat® POCT is a promising decentralised test platform for influenza A/B and RSV in primary care as it provides fairly rapid results with excellent analytic performance. Point-of-care influenza tests could support treatment decisions in primary care. Cobas® Liat® POCT is a promising decentralised test platform for influenza A/B and RSV in primary care as it provides fairly rapid results with excellent analytic performance.

Electronic supplementary material

The online version of this article (10.1007/s10096-020-03860-5) contains supplementary material, which is available to authorized users.

Background

Annual influenza epidemics cause substantial morbidity and mortality and the 2009 H1N1 influenza outbreak was defined as a pandemic by the World Health Organisation (WHO). [1, 2]

About 50% of patients presenting to primary care with influenza-like illness (ILI) during a period of high influenza incidence, with some variation by country, usually have laboratory-confirmed influenza infection. [3] Unlike in Japan for example, [4] in European primary care, point-of-care testing for influenza is not routine practice, making it difficult for clinicians to distinguish influenza from other viral infections that cause ILI at the point of care with a high degree of certainty. [5] It is common practice throughout Europe to base treatment or advice to patients with ILI on signs and symptoms alone, given that point-of-care tests (POCTs) may neither be feasible nor perform adequately, and that antiviral treatment is hardly ever prescribed in primary care. [6] However, guidelines recommend use of antiviral drugs during the influenza season for high-risk groups. [7, 8] POCTs for routine use in primary care and (cost) effective treatment and willingness to prescribe would have major implications for care delivery. The performance of novel POCTs intended for use in primary care is seldom determined using samples obtained from patients consulting in primary care. [9] A valid, rapid test to detect influenza in primary care patients could help target antiviral treatment and informing patients about avoidance of risk for transmission and likely clinical course. Several new POCTS for common respiratory infections in primary care are merging onto the market. [10, 11] We aimed to use the opportunity of an existing trial, the “Antivirals for influenza-Like Illness? An rCt of Clinical and Cost effectiveness in primary CarE (ALIC⁴E)” trial, [12, 13] to evaluate the cobas® Liat® POCT, selected as it is easy to use and has excellent proof-of-principle supporting data. [14–18] However, before uptake into routine care, it is critical that analytic performance be established using samples obtained from real patients in the setting in which the test might be marketed.

Our primary objective was to assess the analytical performance of the cobas® Liat® POCT in detecting influenza A/B and RSV in samples collected from patients consulting with ILI in primary care. In addition, we aimed to explore ease of use.

Methods

Study endpoints

The primary endpoint was the sensitivity and specificity, and positive and negative predictive values of the cobas® Liat® POCT in comparison with laboratory-based PCR testing. The secondary endpoint was the acceptability and failure mode analysis of the cobas® Liat® POCT.

Study population

We included a selection of swabs taken from patients presenting to primary care with symptoms of ILI, willing to participate in the ALIC⁴E trial, who met the inclusion criteria and gave informed consent. [12, 13] There were no additional exclusion criteria. For the purposes of the trial, ILI was defined as a sudden onset of self-reported fever, with at least one respiratory symptom (cough, sore throat, running or congested nose) and one systemic symptom (headache, muscle ache, sweats, chills or tiredness), with symptom duration of 72 h or less. [12, 13]

Recruitment for the ALIC⁴E trial was during periods of heightened influenza incidence which was determined by reviewing the data from the European Centre of Disease Prevention and Control (ECDC) in combination with local and regional alerts during the winters of 2015–2016, 2016–2017 and 2017–2018. The ALIC⁴E Trial team confirmed to each recruiting network when their sites could begin recruitment. Recruitment took place through recruiting sites (GP Practices, Out of Hours surgeries or Paediatric Centres); recruitment was paused when influenza activity dropped again below the epidemic threshold.

Design

This was a method comparison study where a selection of swabs taken at baseline during the ALIC⁴E trial from participants presenting to primary care are analysed both on the cobas® Liat® POCT with the Influenza A/B & RSV Assay, as well as the laboratory-based Fast-Track Respiratory Pathogens 21 Plus kit (FTD, Fast-Track Diagnostics). The selection of swabs was aimed to detect a representative range of test positive and test negative samples for the viral pathogens relevant to the assay under investigation, including: all RSV positive samples, and, in function of the epidemiology of the circulating strains, a more or less equal proportion of samples testing positive for influenza A H1N1, influenza A H3N2 and influenza B.

For the laboratory-based PCR test, extraction was performed on NucliSENS easyMag (bioMérieux) and amplification on Lightcycler 480 (Roche).

Sample analysis on the cobas® Liat® POCT started 12th February 2018 and ended 30th June 2018.

For the POCT, all patients aged up to 16 years recruited during the ALIC⁴E trial had an oropharyngeal and nasal flocked swab taken at baseline by the responsible clinician or recruiter. These two samples were placed in one 3-ml Universal Transport Medium (UTM) (Copan). Those aged 16 years or older had a nasopharyngeal swab taken which was also placed in a 3-ml UTM. Once at the local laboratory, the samples were frozen and stored at − 70 °C (if a deep-freezer was not available on site at the local laboratory, storage at − 20 °C was deemed acceptable) and then transported to a central laboratory in Antwerp, Belgium. Samples went through a freeze/thaw cycle prior to aliquoting at the central laboratory in Antwerp.

A selection of the samples collected during the three influenza seasons (2015–2016, 2016–2017, 2017–2018) of the ALIC⁴E trial, frozen and stored at − 70 °C, were used, to ensure a sufficient large number of positive samples of the various viral pathogens. We enriched the sample for flu positivity.

The fresh samples collected in the third influenza season in Belgium were sent to the central laboratory for analysis both before and after a freeze-thaw cycle.

All laboratory analyses were done in the central laboratory at the University of Antwerp by trained lab technicians.

Prior to testing patient samples, the 2016 Quality Control for Molecular Diagnostics (QCMD) influenza panel (proficiency testing) was tested on the cobas® Liat® POCT, containing 10 samples with dilutions of different subtypes of influenza A and influenza B, to assess the performance of the POC device.

Samples with discordant results were analysed additionally by the RespiFinder 2Smart (PathoFinder).

The participant’s date of recruitment, gender and participant trial ID number were used as identifiers for these samples. Only the laboratory and trial team had access to this information for the purposes of sample identification and tracking.

Acceptability and failure mode analysis

To assess acceptability and certain aspects of failure mode analysis, [19] the device operators were asked to complete a survey (see electronic supplementary material S1).

Acceptability was scored on a five-point Likert scale (1 = completely disagree to 5 = completely agree), including questions regarding device start-up, cartridge handling, operating the device, duration of the test, the provided manual and presentation of error codes. The risk of misinterpretation of failure modes was scored as: low risk–medium risk–high risk–difficult to assess, considering the following errors: expired cartridge, insufficient sample volume, incorrect cartridge insertion, unexpected test result and mismatch between the error code and actual error. The detection of the failure modes was scored as: always perceivable–probably perceivable–not perceivable for the same set of potential errors.

Operators were asked to list the strengths and weaknesses of the device.

Sample size calculation and statistical analysis

We aimed to analyse a selection of swab results obtained using both platforms from a minimum of 730 participants recruited during three consecutive winters (samples selected from all stored frozen samples from season 1 and 2 and all new samples from season 3 until required sample size achieved), giving 90% power at 95% confidence to detect a maximum difference in results obtained from the two methods of 10% at a presumed sensitivity of 95%.

An expanded gold standard (EGS) approach was used to calculate sensitivities and specificities (positive by at least two tests). Results of this analysis were considered the reference standard. Sensitivity, specificity and positive and negative predictive value were determined. Wilson 95% confidence intervals were calculated with the Binom Package. All statistical analyses were performed using R 3.4.3, (https://www.r-project.org/).

Results

Quality control for molecular diagnostics panels

The 2016 influenza QCMD panel, containing 10 samples with dilutions of different subtypes of influenza A and B demonstrated excellent performance of the POC device (see electronic supplementary material S2.a).

Description of population

Nasal and oropharyngeal swabs were obtained from 140 children and nasopharyngeal swabs from 604 adults (744 patients); median age was 33 years (range 1 to 88 years); 55.6% of patients were women, 91.8% had not been vaccinated against flu in the preceding 6 months and 93.3% had not received pneumococcal vaccination in the preceding 5 years (Table 1).

Table 1.

Baseline characteristics of patients providing samples for this analysis.

		N = 744 (100%)
Age	Median (range), years	33 (1–88)
	12 months to < 5 years	37 (5.0%)
	5 years to < 18 years	103 (13.8%)
	18 years to < 65 years	560 (75.3%)
	+ 65 years	44 (5.9%)
Country of origin	Belgium	208 (28%)
	Poland	117 (15.7%)
	Spain	1 (0.1%)
	UK	99 (13.3%)
	Czech Republic	54 (7.3%)
	Denmark	17 (2.3%)
	France	13 (1.7%)
	Greece	21 (2.8%)
	Hungary	58 (7.8%)
	Ireland	9 (1.2%)
	Lithuania	72 (9.7%)
	The Netherlands	24 (3.2%)
	Norway	26 (3.5%)
	Sweden	25 (3.4%)
Sex	Male	330 (44.4%)
	Female	414 (55.6%)
Flu season	1 (2015–2016)	423 (56.9%)
	2 (2016–2017)	255 (34.3%)
	3 (2017–2018)	66 (8.9%)
Ethnicity	White	553 (74.3%)
	Black	3 (0.4%)
	Hispanic	4 (0.5%)
	Asian	6 (0.8%)
	Arabic	6 (0.8%)
	Other	23 (3.1%)
	Unknown	149 (20.0%)
Flu vaccination in the last 6 months	Yes	58 (7.8%)
	No	683 (91.8%)
	Unknown	3 (0.4%)
Pneumococcal vaccination in the last 5 years	Yes	37 (5.0%)
	No	694 (93.3%)
	Unknown	13 (1.7%)
Smoking status	Yes	121 (16.3%)
	No	591 (79.4%)
	Occasionally	32 (4.3%)
Sample type	Nasopharyngeal swab UTM	619 (83.2%)
	Oropharyngeal + nasal swab	125 (16.8%)
POC-PCR results	Negative rapid test	227 (30.5%)
	Positive rapid test for:	517 (69.5%)
	- influenza A	318 (42.7%)
	- influenza B	165 (22.2%)
	- RSV	30 (4.0%)

Analytical performance

The analytical performance of the cobas® Liat® (Roche Diagnostics) POCT for diagnosing influenza A/B and RSV, and a laboratory-based platform, the Fast-Track Respiratory Pathogens 21 Plus kit (FTD, Fast-Track Diagnostics), was evaluated on 744 samples. Eleven samples with discordant results were analysed additionally by the RespiFinder 2Smart (PathoFinder). We encountered 8 run errors in total on the cobas® Liat®, resulting in 736 successfully run POCT tests.

Time to result for cobas® Liat® was within 22 min (2 min hands on time and 20 min analysis time).

The cobas® Liat® POCT had a sensitivity and specificity of 100% (95% CI 99–100%) and 98.1% (95%CI 96.3–99%) for influenza A, 100% (95% CI 97.7–100%) and 99.7% (95%CI 98.7–99.9%) for influenza B and 100% (95% CI 87.1–100%) and 99.4% (95%CI 98.6–99.8%) for RSV, respectively, with the EGS as reference standard (Table 2).

Table 2.

Analytical performance of cobas® Liat® POCT on frozen samples using the Expanded Gold Standard (EGS) as reference

	TP	FP	FN	TN	Sensitivity (95% CI)	Specificity (95% CI)	PPV (95% CI)	NPV (95% CI)
cobas® Liat® influenza A	310	8	0	418	100% (99.0–100%)	98.1% (96.3–99.0%)	97.5% (95.1–98.7%)	100% (99.0–100%)
cobas® Liat® influenza B	163	2	0	571	100% (97.7–100%)	99.7% (98.7–99.9%)	98.8% (95.7–99.7%)	100% (99.3–100%)
cobas® Liat® RSV	26	4	0	706	100% (87.1–100%)	99.4% (98.6–99.8%)	86.7% (70.3–94.7%)	100% (99.5–100%)

TP true positives, FP false positives, FN false negatives, TN true negatives, 95% CI 95% confidence intervals, PPV positive predictive value, NPV negative predictive value

The results for the fresh samples (n = 19) are shown in the electronic supplementary material S2.b, with similar findings and no samples with discordant results.

Acceptability and failure mode analysis

Five independent trained device operators (KJ, KL, KVDV, MK, VM) assessed the user-friendliness of the cobas® Liat® POCT device with a median acceptability of four out of five on the five-point Likert scale. Potential concerns identified were the touch screen readability and proficiency of the included manual (median scores of 3 out of 5) and interpretation of error codes and how to respond to them (median scores of 2 out of 5). (Fig. 1) The risk of misinterpretation of failure modes was generally considered low, apart from the risk of misinterpreting collection of an insufficient sample volume (deemed as high risk). Although most items regarding detection of failure modes were scored as ‘always perceivable’, agreement of an error code to the actual error was scored as ‘not perceivable’ by all raters.

Fig. 1.

Acceptability of the POCT device

The device’s size, analysis speed, ease of use and limited sample preparation were considered important strengths. The lack of a sample holder during pipetting, the size of the touch screen and difficulties reading the cycle threshold (Ct) values were generally considered potential weaknesses.

Discussion

Principal findings

The cobas® Liat® POCT demonstrated excellent analytical performance to detect influenza A, influenza B and RSV, and was considered easy to use, with a fast turn-around-time.

Comparison with previous studies

The high sensitivity and specificity for influenza A, influenza B and RSV found in our study was comparable to previous findings, with better diagnostic accuracy than other routinely available point-of-care devices. [14–18, 20–30] The diagnostic accuracy of POCTs for influenza has been examined in individual studies and several systematic reviews. [31–33] Although the impact of the cobas® Liat® POCT on clinical decision making during visits to the emergency department has been established, [34] the effect on patient-relevant clinical outcomes still has to be determined.

Strengths and limitations

We were able to collect paired samples in a real-life clinical setting of patients presenting to primary care with ILI in this adequately powered study, from 14 different countries across three influenza seasons. Samples were collected, transported and analysed in a standardised and rigorous way and compared to routinely available standard central laboratory multiplex tests. Sampling method varied by age which needs to be taken into account when considering the applicability of our results. Nasopharyngeal swabs were taken from adults and nasal and pharyngeal swabs were taken from children. Our results may therefore not be replicated fully through self-swabbing or nasal swabbing. While nasopharyngeal swabbing may provide the most accurate results, the choice of sample type for children was in accordance with routine care constraints, where more invasive methods are considered less appropriate.

Combining the samples from three different influenza seasons provided us with a representative range of influenza A and B samples, with very few cases of samples testing positive for RSV, which reflects clinical reality.

The vast majority of our samples were stored and frozen samples. However, fresh samples collected during season 3 indicated similar patterns, further strengthening our findings.

The cobas® Liat® POCT in our trial, was operated and maintained by trained lab technicians, familiar with standard operating procedures of POCT devices. This limits the generalisability of our findings to routine primary care both in terms of clinical performance and usability, even when the cobas® Liat® POCT would be performed by trained general practice staff. Although similar results were found in fresh samples, using frozen samples does not necessarily reflect routine primary care where samples would normally be analysed at the point of care, immediately following sampling.

Implications for research

Studies of POCTs for influenza are needed, both in adults and children, with a particular evidence gap for primary care settings. Now that analytical performance has been established in primary care, trials of the impact on clinical course with cost-effectiveness analysis are required to establish the impact on patient-related outcomes, such as antibiotic prescribing rate, antiviral treatment and use of additional tests (chest radiography, blood cultures and routine blood tests). Combining influenza tests with other POCTs, such as C-reactive protein, may help rule out bacterial co-infection to reduce antibiotic prescribing rate in primary care settings.

Implications for clinical practice

We have shown that the cobas® Liat® POCT is accurate, providing timely results in the hands of trained operators. However, the evidence to implement the test in primary care is still scarce. Considering the patient pathway is primordial to ensure adoption of a novel POCT. Time to result should ideally be less than 15 min to accommodate the already pressured consultation length in primary care. Furthermore, these tests should not substitute clinical appraisal. Clinicians should consider local initiatives before widespread implementation to evaluate costs, appropriateness and impact on further testing. The use of antiviral treatment and the impact of POCT to decide whether or not to prescribe still needs to be established and will guide future research on the role of POCT for influenza in primary care.

Conclusions

Cobas® Liat® POCT is a promising decentralised test platform for detection of influenza A/B and RSV in primary care settings, provide fairly rapid results with excellent analytic performance.

Author’s contributions

JV and CB conceived the study. KL, VM, MK and GI performed and supervised data collection and provided raw data. JV analysed the data and wrote the first draft and VM, KL, MK, HG, GI and CB co-drafted and commented on the final version. JV affirms that the manuscript is an honest, accurate and transparent account of the study being reported, that no important aspects of the study have been omitted. All authors have read and approved the final manuscript.

Funding information

This work was supported by the European Commission’s Seventh Framework Programme (FP7), [HEALTH-F3-2013-602525], as well as Roche Molecular Diagnostics, [LIAT-IIS-007] as an investigator initiated clinical study.

Compliance with ethical standards

Conflict of interest

Dr. Butler reports receiving advisory board fees from Roche Molecular Systems and Dr. Butler and Dr. Verbakel report receiving grant support from Roche Molecular Diagnostics. No other potential conflict of interest relevant to this article was reported.

Ethical approval

All procedures performed in this study involving human participants were in accordance with the ethical standards of the institutional research committee (NRES Committee South Central (Oxford B) approval and Clinical Trial Authority (CTA) approval by The Medicines and Healthcare products Regulatory Agency was obtained and all participating countries gained national Research Ethics Committee and CTA approval as required) and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

Informed consent

We included a selection of swabs taken from patients presenting to primary care with symptoms of ILI, willing to participate in the ALIC⁴E trial, who met the inclusion criteria and gave written informed consent. [13]