Radiograph accelerated detection and identification of cancer in the lung (RADICAL): a mixed methods study to assess the clinical effectiveness and acceptability of Qure.ai artificial intelligence software to prioritise chest X-ray (CXR) interpretation

Abstract

Introduction

Diagnosing and treating lung cancer in early stages is essential for survival outcomes. The chest X-ray (CXR) remains the primary screening tool to identify lung cancers in the UK; however, there is a shortfall of radiologists, while demand continues to increase. Image analysis by machine-learning software has the potential to support radiology workflows with a focus on immediate triage of suspicious X-rays. The RADICAL study will evaluate Qure.ai’s ‘qXR’ software in reducing reporting time for suspicious X-rays in NHS Greater Glasgow & Clyde.

Methods and analysis

This is a stepped-wedge cluster-randomised study consisting of a retrospective technical evaluation and prospective clinical effectiveness study alongside the assessment of acceptability via qualitative work and evaluation of cost-effectiveness via a cost utility analysis. The primary objective is to assess the clinical effectiveness of qXR to prioritise patients suspected with lung cancer on CXR for follow-up CT. Secondary objectives will look at the utility, safety, technical performance, health economics and acceptability of the intervention. The study period is 24 months, consisting of an initial 12 month data collection period and a 12 month follow-up period. All the standard care CXRs from outpatient and primary care requests will be securely transmitted to Qure.ai software ‘qXR’ for interpretation. Images with features of cancer will be flagged as ‘Urgent Suspicion of Cancer’ and be prioritised for radiologist review within the existing reporting workflow.

Ethics and dissemination

The study will follow the principles of Good Clinical Practice. The protocol was granted REC approval in August 2023 from North West—Greater Manchester West Research Ethics Committee (REC 23/NW/0211). This study was registered on clinicaltrials.gov (NCT06044454). An interim report will be produced for use by the Scottish Government. The results from this study will be presented at artificial intelligence, radiology and respiratory meetings and published in peer-reviewed journals.

Trial registration number

NCT06044454.

STRENGTHS AND LIMITATIONS OF THIS STUDY.

• Addresses key evidence gaps highlighted in NICE Early Value Assessment Guidance.

• Real-world implementation with a large study population and broad dataset will allow exploration of a number of secondary outcomes.

• Extensive qualitative evaluation and health economics analysis.

• Restricted to one National Health Service health board; results should be analysed with similar studies worldwide to demonstrate generalisability.

• Analysis of artificial analysis utility is complicated by changes to radiology workflow for which the stepped study design seeks to mitigate.

Introduction

Lung cancer is the most common cause of cancer death in the UK.¹ Survival outcomes are strongly linked to early diagnosis²; however, the National Health Service (NHS) is struggling to meet targets on cancer diagnosis and treatment,³ issues that were significantly exacerbated by the COVID-19 pandemic.⁴ Artificial intelligence (AI) has the potential to help manage the backlog of patients and assist in the diagnosis of lung cancer patients.

Pressure on radiology departments

CXRs are a common and accessible imaging modality used to investigate a wide range of symptoms for which cancer is a differential diagnosis. CXRs are the initial tool used to detect lung cancer in the UK⁵; however, radiology departments are suffering from an increasing shortfall of radiologists combined with higher workloads.⁶ The NHS currently tackles this issue through a combination of outsourcing, insourcing and hiring locums at significant costs.⁶ In some trusts, reporting of plain X-rays by non-radiology staff offers support in tackling demand; however, the Care Quality Commission have published concerns of inadequate identification of subtle cancers that are more amenable to treatment and linked to positive outcomes.⁷

The NHS Scotland Lung Cancer Diagnostic Pathway⁸ consists of an outpatient referral by a primary care clinician or an inpatient referral by secondary care clinician. The time period between the X-ray being taken by a radiographer and being reported by a radiologist depends on the clinical context; all Urgent Suspicion of Cancer (USC) referrals should be reported within 2 weeks. If the radiologist suspects cancer, then it will be communicated back to the referring clinician who will make an additional referral for CT scan. This process is variable in the duration depending on context but can take weeks as an outpatient. Further investigations are still required before a conclusive diagnosis, and the treatment plan can be established.

AI opportunity

Machine learning processes are rapidly being considered for adoption across healthcare systems to respond to unprecedented demand.⁹ Realising the benefits of these technologies requires the integration into clinical workflows and an appreciation of capabilities and limitations.⁹ AI has the potential to expedite lung cancer pathways through almost instantaneous detection of USC CXRs to prompt priority reporting by a radiologist, who may then proceed to request CT without the need for intermediary input by primary care. Our study will be a real-word pragmatic application of AI technology to expedite patient care using routinely collected data, in a heterogenous service and patient environment.

Objectives

The objectives and outcomes of this study (table 1) address the majority of key evidence gaps highlighted by Early Value Assessment NICE guidance on AI-derived CXR analysis software.¹⁰

Table 1. Summary table of study objectives and outcomes.

Primary objective	Outcome measures
To assess the clinical effectiveness of qXR to prioritise patients that have suspected lung cancer (identified from AI analysis of a chest X-ray (CXR)) for follow-up CT	Time to ‘decision to recommend CT’, or a decision not to undergo CT for CXR acquired with USC (CXR acquired to CXR reported)
Secondary objectives
Assess the potential utility of qXR within the optimised lung cancer pathway in terms of the impact on both patient treatment and radiological workflow	Time from acquisition of CXR to reporting Time to reporting of CXR classified as USC or other Time to diagnosis of lung cancer Time to treatment initiation for lung cancer Number of hospital visits during screening pathway Hospitalisation within 6 and 12 months of CXR acquisition Death within 6 and 12 months of CXR acquisition
To assess the safety (false-negative rate) of qXR at ruling out patients from entry onto the cancer pathway	Percentage of CXRs not identified by qXR as suspected lung cancer that the radiologist refers for CT for USC Percentage of non-USC that are referred for CT with subsequent detection of lung cancer
To evaluate the technical performance of qXR	Model performance, for example, sensitivity, specificity, positive and negative predictive values
To conduct a Health Economic Assessment of qXR.	Compare costs and health benefits between pre- and post-implementation of qXR
To assess the acceptability of qXR among NHS service users and staff.	Provide an in-depth understanding of the acceptability of qXR among key stakeholders

USCUrgent Suspicion of Cancer

Methods and analysis

Study design

This is a stepped-wedge cluster-randomised study that will be conducted over a 12 month period starting from 26 October 2023. The study protocol is compliant with the SPIRIT-AI framework. Blinding was not possible due to the clinical support nature of the intervention.

Eligibility criteria

Inclusion criteria:

• Patients over 18 years old with frontal chest radiograph, acquired consecutively during the usual care through the outpatient referral pathway only, whose radiograph has not already been reported.

• Patients over 18 years old with frontal chest radiograph, sampled from images already acquired and reported in the current or previous calendar year.

• Key stakeholders such as NHS service users, healthcare staff and NHS management, required to address the qualitative evaluation.

Exclusion criteria:

• The patient has requested that they are removed from the study or has objected to the use of AI in their routine clinical care, and this has been subsequently upheld by the health board.

The study intervention is randomised and applied at an institutional level; therefore, individual patient consent is not required and will not be sought. Patient information will be made available to facilitate those who wish to opt-out.

Study setting

NHS Greater Glasgow and Clyde (NHS GGC) is a large organisation with a complex organisational structure. More than ten sites acquire over 76 000 outpatient CXRs per annum, and the health board employs over 150 radiologists ranging in specialism and seniority. While the study modifies an outpatient referral pathway, the physical setting is within secondary care over the three clusters that comprise NHS GGC:

• Clyde Cluster: The Royal Alexandra Hospital, Inverclyde Royal Hospital, Vale of Leven Hospital

  • 20 246 outpatient CXRs per annum.

• North Cluster: Glasgow Royal Infirmary, Stobhill Hospital

  • 17 557 outpatient CXRs per annum.

• South Cluster: Queen Elizabeth University Hospital, New Victoria Hospital, Gartnavel General Hospital, West Glasgow Ambulatory Care Hospital

  • 38 510 outpatient CXRs per annum.

The order in which the clusters will receive the intervention will be determined by computer-based randomisation. qXR will be implemented over a 30-day period in each cluster. General practitioners in each cluster will be informed of the study through electronic letters in the pre-implementation phase after discussion with the primary care directorate and clinical advisory group.

Study intervention: qXR

Qure.ai has developed and tested qXR software, a CE class IIB medical device that can recognise 25 different abnormalities associated with chest pathology; see online supplemental file 1 for further details. As shown in table 2, we will be using five different abnormalities to identify images suspicious for malignancy (USC): cavity, mediastinal widening, mass, nodule and hilar enlargement. qXR will be used as a triage tool to expedite radiology reporting. All non-USC abnormalities will be reported as per the standard care.

Table 2. Criteria used to generate USC flag following qXR analysis.

qXR criteria to generate ‘USC’ flagIf one or more present:

CavityMediastinal wideningMassNoduleHilar enlargement

USCUrgent Suspicion of Cancer

All patients referred through the outpatient pathway including those with suspected lung cancer will be incorporated in the study. Three alterations to the routine standard of care pathway are made:

1. The software medical device (qXR) will add annotations to flag X-rays identified as USC, producing a secondary capture image placed within the study in the Picture Archiving and Communication System (PACS). Flagged X-rays will be placed in a priority radiological workflow.

2. The acquiring radiographer will flag prioritised X-rays to the radiologist on duty who will report the image during their current shift.

3. If the radiologist agrees USC, then they will request a follow-up CT to be acquired within 72 hours.

Technical requirements

qXR is a cloud-based product that will integrate with the existing Radiology Information System (RIS) using specific AI connection software. Information will be sent from the PACS to qXR where it is processed and then returned to the RIS with an annotated ‘secondary capture’ version of the image as well as a reporting priority. The image is then stored in the local PACS in accordance with the NHS data retention policy.

Technical evaluation

A technical retrospective study will use a sample of 1000 CXR images from all referral sources (including inpatients and emergency department), sampling will be stratified by age, sex and month of year. These data will provide the basis for sensitivity, specificity, positive and negative predictive value calculations to evaluate the software performance.

Health economic evaluation

An economic evaluation will compare costs and outcomes with and without qXR introduction. qXR can impact costs through two mechanisms: improving the efficiency of CXR reporting by identifying normal cases and prioritising CXRs with signs of lung cancer, leading to faster CT provision, diagnosis, treatment and better outcomes. The evaluation is planned as a cost-utility analysis from the NHS perspective. The analysis will include the development of a decision analytic model that will allow for extrapolation from clinical evaluation endpoints to estimates of overall survival, incremental costs and quality-adjusted life years. A budget impact analysis will also be conducted. A health economics analysis plan will be published elsewhere.

Qualitative evaluation

Extensive qualitative work will be conducted to explore the acceptability of qXR among NHS staff and service users. Repeat, in-depth interviews will be conducted with approximately 20 staff from all clusters before and after the implementation of the software. Rather than offering snapshots of staff’s dynamic and complex experiences with the technology, the longitudinal design is expected to facilitate a more nuanced understanding of how these experiences may evolve over time, while capturing processes involved in change. To capture the patient’s perspective, three online focus groups, each containing 6–7 patients, will also be conducted. Groups will be divided as follows:

• Group 1 will consist of service users who had CT performed and were diagnosed with cancer.

• Group 2 will consist of service users who had CT performed but were not diagnosed with cancer.

• Group 3 will consist of service users who did not have CT performed.

With participant permission, all interviews and focus groups will be recorded and fully transcribed. An inductive thematic approach will be used for data analysis.

Usability

Usability will be examined by distributing a structured quantitative questionnaire to users, the results from which will be used to calculate a system usability score. Users may provide additional qualitative information, which will be interpreted to provide context to the usability score.

Data collection

Figure 1 illustrates the simplified data flow between the Safe Haven, NHS GGC’s secure data repository and Clinical Trials Unit (CTU) for analysis. See online supplemental file 1 for further details on data flow.

Figure 1. Study data analysis flows. CTU, Clinical Trials Unit; NHS GGC, National Health Service Greater Glasgow and Clyde.

NHS GnGC Safe Haven will periodically collect, clean, link and de-identify data in accordance with established data protection impact assessment and system security policy. It will be augmented with ‘ground truth’ data, qXR results and any relevant outputs from the qualitative evaluation. Data will be provided by the NHS GGC Safe Haven to the CTU every 3 months.

Data will be collected from all clusters for 2 months immediately preceding live clinical use of qXR in the first cluster and for 10 months thereafter. Outcome data will continue to be collected every 3 months and linked to the study data, for 5 years beyond the 12-month prospective study period. This will allow follow-up studies to incorporate patient outcome that were not known within the initial study period.

Training and human–AI interaction

Radiographers, radiology registrars and radiologists will be trained in the implementation of qXR through a series of face-to-face and online Q&A sessions, referencing a set of standard operating procedures and supported by local ‘champions’. Radiographers and radiology clinicians will use the RIS to examine the reporting priority that qXR generates and examine the annotated image produced by qXR through the PACS system. The PACS output from qXR consists of a ‘bounding box’ overlying each detected abnormality on the CXR (figure 2).

Figure 2. Example of qXR secondary capture overlay function.

AI device data quality and availability

Motion artefacts, incorrect patient positioning or scan exposure can cause poor-quality image data. qXR will only process frontal (PA/AP view) CXR images, which have a minimum 1440×1440 resolution with a complete chest view. The performance of qXR will be affected by images taken with inadequate exposure and inspiration, images that are inverted/flipped/rotated, images with an incomplete view of chest or lung parenchyma and images with significant artefacts.

If qXR rejects a CXR image due to sub-optimal image quality issues, then rejection reasons will be logged for further analysis. Malfunctions in the qXR device could occur through incorrect pre-processing of the image data or incorrect identification of features. These may be logged in the device error log, which will be investigated by Qure.ai. Failure rates will be identified through feature-level ground truth performance assessment throughout the prospective and retrospective arms of the study.

Ground truth

In this study, the ‘ground truth’ is defined as the presence or absence of radiological pathology as determined by a team of senior radiology clinicians who are blinded to all image meta-data and clinical context, that is, ‘pixel-only’. This term is required to provide comparison in performance evaluation of qXR as well as generate actionable outputs when there is inter-observer disagreement between qXR and radiologist or between two radiologists.

Ground truthing is used in the pre-implementation period for qXR calibration. It is then used in the prospective phases for 1000 consecutive CXR images per cluster as well as managing discrepancies in USC detection between radiologists and qXR (figure 3). Finally, ground truthing will be used retrospectively on a sample of 1000 CXRs across all clusters.

Figure 3. Prospective ground truth data flow. CXR, chest X-ray; USC, Urgent Suspicion of Cancer.

Two post-fellowship radiologists will carry out ground truthing and an additional senior radiologist (>5 years post fellowship) will review any disagreements. Each radiologist will be allocated images via a computer-generated randomisation process from the list and may only ground truth the images they have been allocated. Images from all sources will be gathered in a single list for ground-truth generation so that radiologists cannot identify the source or reason for ground truthing.

Data analysis and statistics

The three participating NHS GGC clusters will implement the intervention in a stepwise manner within a 12 month study period. During the study, some clusters will have implemented the intervention, and some will not. The division between the two groups will be random. If the intervention is delivered in the first cluster at month 3 and in subsequent clusters at 2 month intervals, then there will 2, 4 and 6 months of pre-implementation data from the three clusters. Ignoring the first month of implementation, there will also be 9, 7 and 5 months of post-implementation data.

We estimate an average 146 suspected cancers each month in the smallest of the three clusters. Comparing pre-implementation to post-implementation within each cluster, there will be 90% power to detect a change in time from CXR acquisition to decision to CT (TAT) of 0.21σ or less, where σ is the SD of TAT for suspected cancer cases. This would generally be considered a small effect, though if we conservatively assume that TAT has a coefficient of variation of around 1, this could be viewed as a 21% reduction in mean TAT, which would be a clinically relevant improvement.

Simple before-and-after analyses (two-sample T-tests or Wilcoxon–Mann–Whitney tests, depending on the distribution of TAT) will therefore be well-powered within each cluster. Linear regression (possibly with transformation of TAT values, to satisfy modelling assumptions) will allow adjustment for patient-level covariates. By combining data across the three clusters, we also expect to be well-powered to detect intervention effects within (and intervention effect differences between) subgroups of patients, defined by age, sex, deprivation and final diagnosis, through the use of interaction terms in regression models. Intervention effect differences between clusters will also be assessed. Regression models will initially compare outcomes between pre- and post-intervention periods overall but will be expanded to explore the time course of outcomes within clusters, allowing for within-cluster correlations over time. These models will be used to assess changes in the mean TAT in relation to the timing of the implementation of the intervention, allowing for differences between sites, between sub-groups and over time across all clusters.

Patient and public Involvement

NHS Greater Glasgow and Clyde Patient Experience Public Involvement Team facilitated two early public engagement sessions in March 2022 and another session in August 2023. Participants included members of the general public, people with a specific interest in the use of AI in healthcare and patients who have or had cancer and their carers/family members. Feedback has informed the Terms of Reference and early focus for the Patient Reference Group (PRG), which is currently engaged in an exercise to inform the content, design and accessibility of patient and public facing information materials in relation to the launch of the study. The main aim of the PRG is to support and facilitate ongoing public awareness of and engagement with the study, including outcomes, via a range of methods such as further public awareness and engagement sessions, focus groups, surveys and web and social media content. The group will meet four to six times between October 2023 and June 2024, with additional engagement activities scheduled as required.

Ethics and dissemination

Ethics

This study was granted approval from North West - Greater Manchester West Research Ethics Committee in August 2023 (REC 23/NW/0211). The study will adhere to GCP requirements as well as the NHS GGC research quality management system, supplemented with study-specific SOPs as necessary.

Dissemination

The protocol, statistical analysis plan and final report will be published on clinicaltrials.gov (NCT06044454). An interim report will be produced and shared as required by the Scottish Government in its role as a co-funder. This it to inform the early value case for the adoption of AI to prioritise reporting of lung cancer. On completion of study analysis and publication of the final report, data controllership of the study dataset will transfer from the sponsor to the University of Glasgow. Qure.ai shall negotiate its future rights to the study data with the appropriate data controller(s). On completion of study analysis, a final report compliant with the CONSORT-AI guidelines will be produced. The results from this study will be presented at AI, radiology and respiratory meetings and published in peer-reviewed journals.