Diagnostic accuracy of an app-guided, self-administered test for influenza among individuals presenting to general practice with influenza-like illness: study protocol

Abstract

Introduction

Diagnostic tests for influenza in Australia are currently only authorised for use in clinical settings. At-home diagnostic testing for influenza could reduce the need for patient contact with healthcare services, which potentially could contribute to symptomatic improvement and reduced spread of influenza. We aim to determine the accuracy of an app-guided nasal self-swab combined with a lateral flow immunoassay for influenza conducted by individuals with influenza-like illness (ILI).

Methods and analysis

Adults (≥18 years) presenting with ILI will be recruited by general practitioners (GP) participating in Australian Sentinel Practices Research Network. Eligible participants will have a nasal swab obtained by their GP for verification of influenza A/B status using reverse transcription polymerase chain reaction (RT-PCR) test at an accredited laboratory. Participants will receive an influenza test kit and will download an app that collects self-reported symptoms and influenza risk factors, then instructs them in obtaining a low-nasal self-swab, running a QuickVue influenza A+B lateral flow immunoassay (Quidel Corporation) and interpreting the results. Participants will also interpret an enhanced image of the test strip in the app. The primary outcome will be the accuracy of participants’ test interpretation compared with the laboratory RT-PCR reference standard. Secondary analyses will include accuracy of the enhanced test strip image, accuracy of an automatic test strip reader algorithm and validation of prediction rules for influenza based on self-reported symptoms. A post-test survey will be used to obtain participant feedback on self-test procedures.

Ethics and dissemination

The study was approved by the Human Research and Ethic Committee (HREC) at the University of Adelaide (H-2019-116). Protocol details and any amendments will be reported to https://www.tga.gov.au/. Results will be published in the peer-reviewed literature, and shared with stakeholders in the primary care and diagnostics communities.

Trial registration number

Australia New Zealand Clinical Trial Registry (U1111-1237-0688).

Strengths and limitations of this study.

• Accuracy of nasal self-testing for influenza using the QuickVue Influenza A+B assay will be compared with reference standard of nasal or nasopharyngeal swab obtained by a general practitioners (GP) and tested using RT-PCR.

• Recruitment will be nested within an ongoing Australian Surveillance Practices Research Network recruiting patients presenting to general practice with influenza-like illness (ILI)

• Patients attending primary care with ILI may differ in terms of disease spectrum compared with individuals with ILI at home, which is the population where the self-test is intended to be used.

• Self-swabbing of the nose and conducting a lateral flow test unsupervised and guided by an app may select individuals with greater smartphone experience, manual dexterity and/or sociodemographic status.

• Self-report of ILI symptoms using an app may differ from symptoms obtained from GP consultations or from research staff, limiting the ability to validate clinical prediction rules for influenza.

• The mobile app was validated in the USA, but was not adapted or validated for the Australian context.

Introduction

Seasonal influenza occurs annually, causing disease with substantial morbidity and mortality worldwide, especially in the elderly and those with chronic disease.¹ Despite the availability of the influenza vaccine, repeated influenza infections are common throughout life and result in a considerable healthcare burden. In Australia, it is estimated that each year influenza causes an average 310 000 general practitioners (GP) consultations, 18 000 hospital admissions and 1500–3000 deaths.^2–4 Influenza places a particular burden on primary care services during the winter months, contributing to high consultation rates for acute respiratory tract infections. Detection of influenza is thought to provide value clinically by identifying patients who may be at higher risk of complications, and also to potentially inform the use of antivirals and efforts to reduce transmission.⁵

GPs generally diagnose influenza based on a combination of symptoms and risk factors present in each patient, and diagnostic confirmation requires a laboratory test.⁵ Multiple tests are available for influenza, including immunoassays and molecular tests with varying levels of sophistication and cost, that can be used in different clinical settings.⁶ While some point-of-care tests are approved for, and suitable in primary care settings, others can only be conducted in formal laboratory facilities.

Because there is considerable overlap in symptoms caused by influenza and other respiratory pathogens, many patients who are tested for influenza receive influenza-negative results. To reduce the number of unnecessary tests that are requested by GPs, clinical prediction rules have been derived to stratify individuals more accurately than individual symptoms into those with a various likelihood of influenza infection.⁷ Currently, there are no diagnostic tests for influenza that are approved for use by individuals outside of clinical settings in Australia or the USA. The ability to accurately test individuals at home for influenza could provide several potential advantages over current practice. One advantage for patients would be convenience by reducing the need for primary care consultations. Home-testing may also facilitate the earlier use of antivirals when they are most likely to provide beneficial effects on symptom resolution and reduce transmission, and help identify individuals at higher risk of complications compared with those with other causes of influenza-like illness (ILI).⁸

The primary aim of the current study is to determine the accuracy of a self-test for influenza that involves individuals self-swabbing their nose and conducting an immunoassay lateral flow test guided by a mobile app, compared with the gold standard RT-PCR influenza test obtained by their GP. Several studies have already demonstrated the feasibility of collecting patient-reported ILI symptoms.^9–13 This study expands on this field by leveraging smartphone mobile app to instruct participants through conducting a rapid diagnostic test (RDT). We also aimed to explore additional methods for reading the test strip, and validating existing clinical prediction rules for influenza.

Methods

Study design

A prospective observational study of the comparative accuracy of a patient-run, mobile app-guided (see online supplemental appendix A), lateral flow test for influenza (QuickVue Influenza A+B assay test, from Quidel Corporation) using a low nasal self-swab (referred to in this protocol as ‘flu@home’), compared with clinician-collected nasal or nasopharyngeal swab for influenza detected by a commercial RT-PCR (see figure 1).

Figure 1.

flu@home Australia study procedure. ASPREN, Australia Sentinel Practices Research Network; GP, general practitioners.

Study population

A systematic sample of adult patients with ILI presenting to general practices participating in the Australia Sentinel Practices Research Network (ASPREN), which is a network of over 350 general providers from over 200 sentinel sites throughout Australia. GPs in the network participate in routine surveillance studies of respiratory infections by the Commonwealth^{14 15} (see online supplemental appendix B for ASPREN protocol).

Inclusion and exclusion criteria

Inclusion criteria are: (1) age ≥18 years, (2) presenting to ASPREN clinic sites¹⁶ with fever, cough and fatigue, (3) agree to have their GP/nurse practitioner obtain a nasal or nasopharyngeal swab for surveillance purposes and (4) have their own Android or iOS smartphone or tablet. Exclusion criteria will be non-English speakers, people who are incarcerated, people highly dependent on medical care who may be unable to give consent and people with a cognitive impairment, an intellectual disability or mental illness. We did not exclude people with physical disabilities or impaired vision, but rather left the decision to recruit a patient up to their GP at the time of their visit.

Recruitment

Each clinic will recruit any patient presenting with an ILI who is 18 years and older, has a smartphone and agrees to participate in the study.

Clinical setting

Study participants will be recruited from practices participating in ASPREN, which is a network of sentinel GPs who report deidentified information on ILI as well as other infectious disease conditions.¹⁶ The deidentified information will include date of symptom onset, influenza vaccination history, comorbidities related to influenza and whether the patient is a healthcare worker. Data from ASPREN are used by State and Commonwealth Departments of Health for infectious disease surveillance and vaccine effectiveness estimates.¹⁷ ASPREN data contribute to the Global Influenza Vaccine Effectiveness Movement and the WHO Collaborating Centre for Reference and Research on Influenza.

Outcome measurements

Primary outcome

• Accuracy of detection of influenza A/B infection based on self-reading of the flu@home test compared with laboratory RT-PCR testing.

Secondary outcomes

• Accuracy of detection of influenza A/B infection based on self-reading of an enhanced high contrast image of the flu@home test strip compared with laboratory RT-PCR testing.

• Accuracy of detection of influenza A/B infection based on the app’s automatic interpretation algorithm of the flu@home test strip image compared with laboratory PCR testing.

• Accuracy of clinical prediction rules including the Influenza Score⁷ based on individual and combinations of presenting symptoms obtained from the app and/or the patient’s GP compared with laboratory PCR testing.

• Satisfaction and experience of patients interacting with the flu@home app.

Other variables

The app will collect information on demographics (age, sex, race), household composition, influenza vaccination history, risk factors for influenza infection, presence and duration of ILI symptoms (e.g., cough, fever, fatigue, chills or sweats) (see online supplemental appendix C). These variables will be used to facilitate interpretation of test results in terms of these various participant characteristics.

Study procedures

Patients who participate in the ASPREN study will be invited to participate in the flu@home self-testing study from July 2019 until all 2300 kits are distributed, no later than December 2020. Each participating GP will be provided with a set number of test kits, based on the numbers of patients with ILI encountered in previous influenza seasons and the number of patients with ILI swabbed during the current 2019 influenza season. Participating GPs will be asked to recruit all patients who meet study eligibility criteria. After completing the standard ASPREN protocol (see online supplemental appendix B), a GP will ask the participant if they would like to participate in the flu@home study. Once the participant consents, the GP will hand them the test kit and instructions for downloading the free app and the patient will be asked to conduct the remainder of the study procedures at home on that day or the following day. A post-test survey will be sent to participants via the app 24–48 after they complete the test procedure.

Influenza testing methods

Home/self-testing

Patients will be provided with a self-test kit by their GP containing a Quidel QuickVue Influenza A+B lateral flow test (rebranded as the flu@home kit for research purposes), and asked to download the free flu@home app¹⁸ to their personal iOS or Android smartphone or tablet. Each test kit includes a unique 8-digit study ID number that will be linked to reference test results, but cannot be used to personally identify participants. The app collects the variables noted above through a questionnaire, and guides the patient through the self-swabbing and testing procedure. They will be instructed to obtain a low nasal swab using a single foam-tipped swab inserted into each nostril, and then perform the steps to conduct the lateral flow test.

Having completed the test steps, the app guides the patient to read their test strip by first asking them whether they see a blue line (control line) and any pink lines (first interpretation). A pink line above the blue line indicates influenza A, and a pink line below the blue control line indicates influenza B. If the patient indicates they do not see the blue control line, they are informed that they have a defective test strip and interpretation guidance is not provided. For patients who indicate they see a blue control line, the app guides the patient to obtain a photo of their test strip using their smartphone camera. During this process, the app provides a guided test strip image capture, including on-screen feedback to the participant to ensure proper alignment, lighting, positioning, scale and rotation of the test strip prior to taking a photo. Once a photo of the strip is captured, the user is presented with a high-contrast image of their test strip and asked to reinterpret the test results by indicating how many lines they now see on the strip (second interpretation). Presenting a high-contrast image to the patient may help them see lines on the test strip that may have previously been too faint to easily identify. Initially, the app uses the patient’s direct observation of the strip to inform the patient whether it is likely their test result was positive for influenza. During the study, we may adjust this process to inform the patient of their likely test result based on autointerpretation of the images captured.

While the test strip differentiates between influenza A and B, we will not ask individuals to make this determination. If the guided test strip image capture is not successful, the app requests the patient to manually take a normal photo of their test strip using their smartphone for later analysis. The app uses the patient’s observations to inform the patient of their likely test result.

Patients will be given links to publicly-available information on influenza from healthdirect¹⁹ and provided with usual care recommendations in the app depending on their test results (from either the first or second interpretation). The app includes a medical disclaimer indicating ‘The interpretation of your result may differ from a medical test conducted in a clinical lab environment. In no circumstances should the results of this test be relied on without independent consideration and confirmation by a qualified medical practitioner’.²⁰ Patients will be notified of the results of the reference test by their GP, who will provide standard care based on the RT-PCR results. Study materials will clearly indicate that the flu@home test is an experimental research test, and participants should trust the reference test results provided by their GP. Participants whose flu@home results are discordant with those of their GP will be asked to contact their GP for any clinical management decisions or changes that their GP would recommend.

Reference testing

Influenza and other respiratory pathogens will be detected using RT-PCR on the swabs obtained by the GP at ASPREN clinical sites (see online supplemental appendix B for list of pathogens tested). Samples will be sent to SA Pathology in Adelaide, South Australia, via Australia Post’s Express postsystem, allowing for next-day delivery from all capital cities.²¹ Results of the laboratory PCR test, home self-test kit and survey data from the app will be linked by the 8-digit number available on the test kit and PCR sample.

Post-test survey

A link to a reflective online survey created in Qualtrics will be delivered to participants who complete the test procedure. The request to complete the survey will be delivered via participants’ smartphone or tablet 24–48 hours after completing their self-test. The survey will solicit responses regarding the respondent’s (a) health behaviours and attitudes, (b) perceptions of their experience and usability of the self-test impact, (c) perceived value of self-testing and (d) intention to act on self-test results. Survey items will be close-ended and, generally, call for a response to a five-point Likert scale with anchors ranging from strongly agree to strongly disagree (see online supplemental appendix D for follow-up survey items categorised by construct, ie, focal topic).

Participant discontinuation

Individuals who start the app, provide consent, but fail to complete all steps of the test procedure will be excluded from the primary comparative accuracy analysis. If any participants who were swabbed by their GP as part of ASPREN surveillance test positive for influenza, they will be contacted by their general practice clinic to discuss further clinical management; this will not be affected by failure to complete the flu@home procedure.

Data analysis

We will conduct a descriptive analysis of demographics, presenting symptoms and baseline variables such as household composition, vaccination status and general health questions. Prevalence of influenza will be obtained from the positivity rate of PCR laboratory testing. Sensitivity, specificity, positive and negative predictive values with 95% confidence intervals (CIs) will be calculated for the presence of influenza based on patient interpretation of self-test results compared with the reference standard PCR result. Accuracy will also be calculated for participants’ interpretation of the enhanced high-contrast photo of the test strip, as well as the automatic test strip interpretation algorithm, compared with the reference standard PCR result. We will also measure the accuracy of clinical prediction rules based on individual and combinations of ILI symptoms based on Influenza Score⁷ and other prediction rules, compared with the reference standard PCR result. Subgroup analyses will explore test accuracy based on age, symptom profile, duration of illness and influenza type (A/B).

We will conduct a descriptive analysis of post-test survey results related to demographics, health behaviours and attitudes, experience and usability of the self-test, impact and perceived value of self-testing and intention to act on self-test results. In addition, we will conduct multivariate analysis of variance (MANOVA) to determine if statistically significant differences exist among various subpopulations (e.g., age group, gender) regarding their responses to survey items related to variables associated with experience and usability of the self-test and impact and perceived value of self-testing. MANOVA analysis permits simultaneous testing of the variables associated with one construct, for example, experience and usability of the self-test, simultaneously to arrive at a holistic assessment and recognises the potential correlation related to these variables. Analysis of variance will be used to determine if statistically significant differences exist among the responses from various subpopulations (e.g., age group, gender) to survey items related to intention to act on self-test results. We will also use partial least squares regression to construct predictive models to assess relationships among demographics, health behaviours and attitudes, experience and usability of the self-test, impact and perceived value of self-testing and intention to act on self-test results.

Sample size calculation

The sample size required for this study was determined based on (1) expected completion rate of the home test kit, (2) influenza positivity rate, (3) availability of test kit materials and (4) number of flu-positive test results which are typically provided in U.S. Food and Drug Administation (FDA) submissions for regulatory approval of rapid influenza diagnostic tests. To our knowledge, there have been no other comparative accuracy studies of a smartphone-enabled respiratory illness diagnostic test conducted in Australia. Therefore, the expected completion rate of the home testing procedure is based on a USA-based pilot study that found that 60% of individuals completed the flu@home test kit when it was mailed to them. In the current study, we expect a higher completion rate given that participants will be recruited by their GP rather than online. The influenza positivity rate among patients presenting with ILI to ASPREN clinics is based on data from previous years, which indicated a 20% positivity rate among recruited adults (of all ages) in the July to December period. Assuming that 60% of the 2300 self-test kits distributed to GPs are completed (1380), we expect 20% (276) to be influenza positive. This absolute number of influenza positive specimens exceeds that required by FDA in regulatory submissions to evaluate the accuracy of new tests designed for clinical settings, which is typically 120.²² There are not currently any recommendations for sample sizes needed for evaluation of the accuracy of home-based tests for influenza.

Patient and public involvement

The flu@home app has undergone several iterations of usability and user acceptance testing with a diverse population in the USA. This included usability testing conducted during a pilot phase in the USA using an independent user research firm, which provided input on app usability, time to conduct questionnaire and the appearance and design of the app. There has not been any prior testing of the app in Australia, however, the research study members from Australia reviewed the app prior to launch to ensure the language in the app was appropriate for the Australian context.

Ethics and dissemination

The study procedures will follow Australian clinical and ethical standards as outlined by the University of Adelaide Human Research Ethics Committee (HREC). All activities will follow the Code of Good Practice in Clinical Research. Participants will provide informed consent for the flu@home study within the app that is downloaded. The study was approved by the Human Research and Ethics Committee at the University of Adelaide (HREC Number: H-2019-116). The authors will seek approval for any protocol modifications, which will also be reported to the clinical trials registration site. Results of this study will be reported using the Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidelines for reporting diagnostic accuracy studies and published in the peer-reviewed scientific literature.²³

Confidentiality and data management

All study data collected are non-identifiable. No participant names, addresses or private information are collected for the purposes of the study. Samples from the ASPREN survey and app data are linked via a unique barcode. The researchers cannot link the barcode to identifiable patient details such as name, address or other private information. ASPREN surveillance data will be stored on University of Adelaide computers, which can only be accessed by authorised representatives. All data will be non-identifiable. All data collected by the flu@home app will be protected with industry-standard encryption on systems hosted through Amazon Web Services and the Google Cloud Platform, which are only accessible by authorised representatives of the app development organisation, Audere. Audere is a non-profit application development organisation that runs the flu@home application. During data analysis, non-identifiable data will be transferred to a University of Washington approved data storage location, which is only accessible to authorised parties and the University of Adelaide drives for analysis. Further information about confidentiality and data management in the mobile app can be found in online supplemental appendix A.

Discussion

Influenza is a common infection that occurs annually in the southern and northern hemispheres. Consultations for respiratory tract illnesses are one of the most common reasons for presentation in primary care settings in Australia and most other high-income and middle-income countries.² In Australia, influenza season occurs between the months of May to October.²⁴ Differentiating aetiology of respiratory tract infection based on symptoms alone is limited, and current confirmatory diagnosis of respiratory pathogens involves laboratory testing.²² Diagnostic tests for influenza are commonly used in laboratory settings, and in many countries are used in primary care or pharmacy settings.^25–27 Regulatory approval varies between countries, but typically tests approved for primary care involve simple point of care assays that do not require laboratory technician expertise.

We will use an existing RDT for influenza A and B that has been approved in the USA for use in primary care clinics since 2004 (QuickVue Influenza A+B assay test, from Quidel Corporation).²⁸ This test has adequate performance as demonstrated by regulatory approval in the USA, with a 2017–18 clinical study comparing this test to an FDA cleared A+B molecular test, showing sensitivity of 94% for type A and 70% for type B and specificity of 90% for type A and 97% for type B.^{29 30} However, we note that additional evaluations of this test (and similar lateral flow tests) for influenza show lower test accuracy in further clinical evaluations. A 2017 meta-analysis of 162 studies of rapid tests for influenza found noted that the pooled sensitivity of such tests favoured industry-sponsored studies by 6.2%–34.0%.³¹

The potential for individuals to test themselves for influenza follows a pathway for home-based testing that has revolutionised pregnancy testing with commonly available lateral flow assays, glucose testing using home-based monitors, as well as electronic devices for measuring blood pressure. While there is strong evidence that individuals are able to obtain swabs themselves from the nose or throat,^32–34 there is no evidence currently for the accuracy of individuals performing a diagnostic test on self-obtained samples for influenza.

The potential value of a self-test for influenza could lead to changes in practice and behaviour, assuming the test has sufficient accuracy. For individuals in the community, this could lead to faster diagnosis, improved access to diagnostic testing, improved diagnostic certainty and reduced need to contact healthcare services. For primary healthcare services, it could reduce the burden of consultations for ILI and facilitate more rapid or targeted use of antivirals if these can be prescribed remotely (by telephone or telemedicine consultations). In terms of public health, self-testing could also influence infection control and transmission reduction strategies at the community level. Combining a diagnostic test with a smartphone where the user’s steps are process-controlled (e.g., embedded timers ensure the patient adheres to the test procedure) may both facilitate support for the user, and potentially allow enhanced interpretation of test results using the existing camera and software found in current devices. A downside to home/self-based testing for influenza is that easier access to testing could lead to the diagnosis of mild cases of influenza where antiviral treatment is not indicated. Increased access to self-testing include has financial implications including added costs to individuals who might have to purchase the tests, and to the healthcare system that might need to interpret, repeat or act on test results. Inaccurate tests could further cause harm through false-negative and/or false-positive results.

Limitations of the study

The study has several potential limitations. First, recruitment of participants will not be entirely consecutive, although this follows the procedures that the participating clinics use for ongoing surveillance activities. Limiting this study to general practices means that some patients with ILI are excluded, such as those attending hospitals and emergency departments, receiving medical care from locum doctors or not seeking any medical treatment for ILI. Second, the spectrum of individuals presenting with ILI to GPs may be different to that expected in the community, with higher influenza prevalence, more severe symptoms and/or longer time since onset of infection. The time point at which individuals present to their GP with influenza may have a critical impact on test sensitivity, as there is strong evidence that the sensitivity of rapid antigen influenza tests declines markedly beyond the initial 48–72 hours of illness.^{35 36} Third, the performance of the nasal swab, and conduct of the lateral flow test is unsupervised, and therefore we will not be able to determine the impact of these factors on test accuracy. There is robust evidence that individuals are able to collect mid turbinate and low nasal swabs with similar performance to healthcare professionals for influenza, but we will not be able to further verify this in the current study.³⁷ Fourth, conduct of the test may vary with participant characteristics, such as age or limitations in ability to handle smartphones, and their ability to visualise lines on the test strip. We will explore these using subgroup analyses (based on age), and user feedback from follow-up surveys. Fifth, differences in interpretation of the enhanced image may depend on the technical capabilities of individuals’ smartphones. Sixth, we are aware that the flu@home app has not been validated in this population and setting, and may need additional validation before being implemented or being used with a commercial device. Finally, while we do ask study participants about multiple aspects of their experience with the home-based influenza test, we will not ask specifically about their feelings regarding testing positive for influenza using a home-based test. Understanding the emotional impact of receiving a positive result using a self-test is out of scope for this study. Additionally, we will not be able to evaluate comparative costs of the flu@home test compared with usual care within this study.