AUGUR‐AIM: Clinical validation of an artificial intelligence indocyanine green fluorescence angiography expert representer

Abstract

Aim

Recent randomized controlled trials and meta‐analyses have demonstrated a reduction in the anastomotic leak rate when indocyanine green fluorescence angiography (ICGFA) is used versus when it is not in colorectal resections. We have previously demonstrated that an artificial intelligence (AI) model, AUGUR‐AI, can digitally represent in real time where experienced ICGFA users would place their surgical stapler based on their interpretation of the fluorescence imagery. The aim of this study, called AUGUR‐AIM, is to validate this method across multiple clinical sites with regard to generalizability, usability and accuracy while generating new algorithms for testing and determining the optimal mode of deployment for the software device.

Method

This is a prospective, observational, multicentre European study involving patients undergoing resectional colorectal surgery with ICGFA as part of their standard clinical care enrolled over a 1‐year period. Video recordings of the ICGFA imagery will be computationally analysed both in real time and post hoc by AUGUR‐AI, with the operating surgeon blinded to the results, testing developed algorithms iteratively versus the actual surgeon's ICGFA interpretation. AI‐based interpretation of the fluorescence signal will be compared with the actual transection site selected by the operating surgeon and usability optimized.

Conclusion

AUGUR‐AIM will validate the use of AUGUR‐AI to interpret ICGFA imagery in real time to the level of an expert ICGFA user, building on our previous work to include a larger, more diverse patient and surgeon population. This could allow future progression to develop the AI model into a usable clinical tool that could provide decision support, including to new/infrequent ICGFA users, and documentary support of the decision made by experienced users.

INTRODUCTION

Anastomotic leak (AL) remains one of the most serious complications of major colorectal resectional surgery, with recent randomized controlled trials (RCTs) of patients undergoing anterior resection of the rectum reporting an incidence of up to 23% [1, 2, 3, 4]. It is associated with prolonged hospital stay as well as increased morbidity, mortality and healthcare costs [5, 6]. While many identified risk factors for AL, such as male sex, smoking, advanced age, diabetes, tumour location and neoadjuvant therapy [5, 7, 8, 9], cannot be altered intraoperatively, anastomotic vascular perfusion is considered an important controllable risk factor [10].

Traditional methods for intraoperative assessment of bowel perfusion involve subjective measures such as palpation of mesenteric arterial pulsations and observation for tissue discoloration and active bleeding with regard to marginal arteries [9], all which have been demonstrated to be unreliable [11]. Indocyanine green fluorescence angiography (ICGFA) involves the near‐infrared (NIR) light visualization of systemically administered ICG and permits real‐time evaluation of the perfusion of the bowel segment intended for anastomotic construction. Many clinical studies have assessed the possible effects of the use of ICGFA, with recent RCTs and meta‐analyses demonstrating a reduction in AL rates when ICGFA is used intraoperatively versus when it is not [2, 12, 13, 14].

There are some potential barriers to the widespread adoption of ICGFA, such as the associated learning curve and significant interuser variability [15, 16, 17]. Quantification of the ICGFA signal has been proposed as a method to overcome these factors, with many studies reporting exploration of quantification methods. However, before now, only one published study has utilized a developed quantification method to inform intraoperative decision‐making [18]; three other studies concerned real‐time quantification that simply presents additional information to surgeons at the time of surgery (but did not report this) [19, 20, 21].

We have recently demonstrated that a novel, advanced artificial intelligence (AI) model (AUGUR‐AI) can digitally represent in real time where experienced ICGFA users would place their surgical stapler based on their interpretation of the fluorescence imagery [21]. This is an advance in providing an interpretable classification of quantified ICG signalling. In this study, AUGUR‐AIM, we will broadly clinically validate our AI method and models across multiple clinical centres with the operating surgeon blinded to the model's representations at the time of the clinical decision regarding transection, while also generating new models utilizing videos from the participating sites for subsequent, in‐study prospective testing. We will also examine the generalizability and usability of the technology while determining the optimal methods for displaying the generated information to the operating surgeon.

METHOD AND ANALYSIS

Trial aims

The primary aim of this study is to develop, validate and test trained algorithms as software, including in theatre on a blinded basis. To do this, ICGFA will be used as approved in patients undergoing open, laparoscopic or robotic major colorectal resectional surgery as per the usual clinical care protocols in centres that already routinely utilize ICGFA. Video recordings of the ICGFA imagery and the surgeons’ interpretations will be used to inform the AI models, with their accuracy being examined alongside that of the surgeons doing the operation in real time. Importantly, in this study the model representations will not be used to influence the surgeon or the surgery.

The secondary aims of this study are to examine and optimize the usability and functionality of the software in clinical practice, including the best means to display the information generated from the software to an operating surgeon. The study will also assess this system's real‐world applicability and its suitability for patients, surgical teams and clinical services.

Trial sites and participating surgeons

AUGUR‐AIM is a multicentre, prospective study across Europe. Lead surgeons at each clinical site have expert experience in performing ICGFA during colorectal resectional surgery and use ICGFA routinely in such cases. Additional sites may be included as the study progresses. The study will be coordinated by University College Dublin (UCD). The study has not received any external funding, and the protocol is registered with clinicaltrials.gov (identifier NCT06708819).

Trial population

All patients undergoing major colorectal resectional surgery as part of standard clinical care are potentially suitable for inclusion in the trial, subject to inclusion/exclusion criteria, as shown in Table 1.

TABLE 1.

Inclusion and exclusion criteria.

Inclusion criteria	Exclusion criteria
Participant is willing and able to give informed consent for participation in the study	Participant who is pregnant, lactating or planning pregnancy during the course of the study
Aged 18 years or above	Significant renal or hepatic impairment
Colorectal disease requiring segmental resection with anastomosis	Allergy to intravenous contrast agent or indocyanine green
Participant has clinically acceptable laboratory results, including liver function tests	Any other significant disease or disorder which, in the opinion of the investigator, may either put the participant at risk because of participation in the study, or may influence the result of the study or the participant's ability to participate in the study
In the investigator's opinion, is able and willing to comply with all study requirements
Willing to allow his or her general practitioner and consultant, if appropriate, to be notified of participation in the study

Trial design

AUGUR‐AI will be performed in a phased way, moving from initial development of both site‐specific and aggregated algorithms, to validation and testing of these and pre‐existing developed algorithms. The dye and imaging systems will being used within their approved licences and there will be no change to patient care as the operating surgeon will be blinded to the interpretations made by the algorithm. Validation will be assessed based on whether the actual stapler placement by the operating surgeon falls within the boundaries of the ‘expert’ zone as predicted algorithmically. The observational nature of the study minimizes divergence from the patient's standard clinical pathway and allows for assessment of the applicability and suitability of the method to fall within current clinical workflows.

Sample size and power calculations

Based on our existing prototypes, any new or iterated AI model will need ten patient videos from each participating site to allow a new AI model to be developed. This model will then be tested on 20 new cases prospectively alongside the original prototype AI model in parallel with the original model. Datasets from collaborating sites will be combined to train a consensus or aggregate expert model, which will also be tested prospectively. After 20 cases from each participating site, a second site‐specific algorithm will be developed for parallel testing to evaluate if any improved performance results from the inclusion of additional cases (Figure 1).

FIGURE 1.

Phases of algorithm testing and development (ICGFA, indocyanine green fluorescence angiography).

For algorithm validation, based on our a priori work demonstrating 88% accuracy, a sample size of 164 patients is needed for a 95% confidence interval with a half‐width of approximately 0.05, which we consider sufficiently precise for this estimation [precision estimates calculated for the Wilson score confidence interval method using the ‘presize package’ in R (R Development Core Team) [22]]. A sample size of 200 patients in total would allow completion of study phases 2 and 3 with up to 18% dropout.

Patient recruitment

Adult patients with colorectal disease undergoing segmental resection via an open, laparoscopic or robotic approach will be approached for inclusion in the trial. Participants must be able and willing to comply with the terms of the protocol.

Such patients will be identified from referral letters, outpatient clinics, endoscopy lists and multidisciplinary cancer meetings. Patients will undergo standard preoperative work‐up and assessment of fitness for surgery as per standard practice. The optimal management of the patient will be determined on the basis of institutional practice and protocols.

Suitability for inclusion as per the selection criteria will be assessed and patients will be provided with verbal and written details.

Informed consent

A verbal explanation of the study along with the approved patient information leaflet (PIL)/informed consent form (ICF) (see Appendix S1) will be provided by a suitably qualified member of the healthcare team for the patient to consider. The PIL will provide detailed information about the rationale, design and personal implications of the study. Following the provision of information, patients will be given the opportunity to discuss the study with their families and healthcare professionals and will be given as much time as possible to consider their participation in the study: ideally, they will be allowed 24 h as a minimum. The right of the patient to refuse consent without giving reasons will be respected. Patients will then be formally assessed for eligibility and invited to provide informed, written consent for their participation in the study. Patients can withdraw their participation at any time without giving a reason; however, data that have already been processed will continue to be used where they have become integrated into the algorithm. A copy of the signed consent form will be given to the participant. The original signed form will be retained at the study site.

Baseline measurements

All baseline assessments performed will be part of standard preoperative procedures before surgery. Participants undergoing colorectal surgery are subject to a standard preoperative work‐up that may include colonic visualization by either colonoscopy or CT colonography, staging CT scan of the chest, abdomen and pelvis, MRI of the rectum and assessment of fitness for surgery. Haematological and biochemical blood testing is routinely performed prior to treatment.

Diagnostic intervention

Preparation

The colorectal resection will be performed according to the surgeon's usual technique, using an open, laparoscopic or robotic approach. Using standard clinical care, the colorectal segment for future anastomosis is prepared including vascular ligation and preparation of the mesocolon. For right hemicolectomy, where an ileal segment is also being prepared, these steps should also be completed for this segment.

Once presented for ICGFA, with the surgeon's confirmation of the level judged clinically appropriate for transection, the segment(s) of interest is positioned either intracorporeally or extracorporeally in either a horizontal (preferable) or vertical orientation.

With the camera held steady and the intestinal segment(s) in focus, either synchronous NIR–white light viewing or ICGFA overlay mode alone is selected and recording begun.

ICGFA is administered intravenously (the dose of which will be decided by the operating surgeon as per their routine clinical practice), and the imagery is recorded as the ICGFA signal acquires, maintaining camera stability. It is important for the observer not to disrupt the surgical scene (i.e. with movement or instruments, including in the event of a poor perfusion signal) until at least 60 s after the visualization of fluorescence inflow into the segment of interest. It is also important not to change the method of ICG visualization (NIR only/overlay) during the inflow period, although after this it can be changed to whatever method the surgeon feels most appropriate.

While still recording and maintaining the field of view, the site of the bowel for transection after ICGFA is indicated, ideally by placement of the stapler (but alternatives such as placement of a nylon tape or other indicator of bowel level are acceptable) so that it is evident in the same video recording. An explanatory video recording has been provided to each site.

Extra doses of ICG will be permitted (e.g. extracorporeal assessment, or after formation of the anastomosis) should the surgeon feel this to be beneficial. The specifics of each operation, including the decision to make a change to the planned anastomosis following ICGFA assessment, will be at the discretion of the operating surgeon.

Video analysis

Video recording from the intervention will be studied by local investigators and ensured to be free of any personal identifying information. The pseudonymized information (including clinical classifiers) will be shared with UCD in accordance with best General Data Protection Regulation (GDPR) practice and the data management plan.

Software analysis

The AI algorithmic process for analysing and classifying data in this study consists of four steps: (i) stabilization of the live stream, (ii) extracting fluorescence curves and features of said curves, (iii) creating a sequence of extracted features in a user‐defined region of interest, (iv) classifying the sequence. During video analysis the video stream is stabilized to counteract small movements of the camera. The movement between the current frame and a previous reference frame is calculated using tracking algorithms within OpenCV and an affine transform is applied to adjust for the movement. The reference frame is updated at regular intervals to prevent the image drifting over time. To generate features, each NIR frame is divided into grid regions and mean fluorescence intensity is extracted from each region. These values are collated to create a fluorescence curve for each region. A smoothing algorithm is applied to each curve to remove any noise. The four features for the classifier are then extracted from the smoothed curve for each region. In its current iteration, the user draws a line as the region of interest for classification. For each point on this line the median of the feature's five regions perpendicular to the line in both directions is taken to generate a sequence to be classified by the classification model. The classification model is a bi‐directional long short‐term memory model trained using the PyTorch Python library (Figure 2). Methods that allow automatic segmentation of the regions of interest and can present the AI representation to an observer with live masking are already in development.

FIGURE 2.

An example of an indocyanine green fluorescence angiogram of a colonic segment. Images 1–4 showing the progression of fluorescence over time with the superior part of each image showing the near‐infrared fluorescence and the inferior part of each image showing a green overlay of the fluorescence on a white light image. Image 5 shows the interpretation that the artificial intelligence system has made based on the inflow pattern of the indocyanine green fluorescence angiography signals from which images 1–4 are taken.

Sample handling

All video samples referred to in the protocol will be recorded to the local department with the results accessed by the research team. Video samples will be processed and stored locally prior to pseudonymization and sharing with UCD. Only defined researchers working on this project as part of the study team will have access to the samples.

Postoperative care

Postoperative management of each patient will be conducted as is standard for each clinical site. Adverse outcomes will be assessed and recorded during the patient's routine postoperative care and used to categorize the patient datasets.

Data collection and management

Clinicians at each clinical site will collect relevant patient data from both the patient and his or her healthcare records, which will be stored in a password‐protected database linked to the video. Video files will be transferred from the endolaparoscopic visual processing unit for secure transfer via an encrypted USB and/or digital operating platform. Video files will then be pseudonymized, with each clinical site retaining its own key. Pseudonymized files will then be transferred to UCD via the HEAnet FileSender platform.

Outcomes

The AUGUR‐AIM project will validate the concept that an AI algorithm can interpret ICGFA signals in real time to the level of an expert surgeon. This will include the results of the initial algorithm at the start of the project and also the new algorithms that are developed using new data generated from participating clinical sites. Accuracy will be defined according to whether the expert stapler placement by the operating surgeon falls within the boundaries of the ‘expert’ zone as predicted by the algorithm, at both object and pixel levels. Pixel‐level analysis will include accuracy, precision, recall (sensitivity), specificity, F1 score and Jaccard index/DICE coefficient to measure the overlap between the predicted and actual zones. Cross‐correlation analysis will be performed to examine the relationship between the actual stapler placement and the predicted zone. The predicted zone will be analysed for its distance from the actual stapler placement to quantify how far the predicted zone is from the actual placement zone. Usability and surgeon preferences regarding data display will be studied with participating surgeons by means of qualitative survey and debriefing questions.

Safety evaluation and reporting of adverse events

Patients will be monitored for any possible adverse events throughout the course of their care. Due to the excellent safety profile of ICG, and this being an observational study, monitoring will consist of clinical review only without any specific investigations. Any adverse events will be documented and reported directly to UCD.

Dissemination

Study outcomes will be disseminated among the academic community and will be led by UCD with input from each of the study partners. This communication will occur via publications and presentations in scientific forums. As this is a validation study, positive results will encourage and enable consideration of clinical translation via the appropriate regulatory pathways.

DISCUSSION

The use of AI in healthcare is rapidly expanding, but despite claims of boundless potential, uptake in surgery so far has been significantly slower than in other medical fields such as radiology or pathology [23, 24, 25, 26]. Surgeons themselves have varied opinions on the use of AI in surgery, with the strongest agreement that it would be best applied as a decision‐support tool rather than as an independent decision‐maker [27, 28]. Nonetheless, broadly, surgeons feel it will become perhaps the most dominant of currently foreseeable new/disruptive technologies over the next 10 years [29]. Naturally of course, given the permanency of intraoperative surgical decisions, any use of AI intraoperatively requires meticulous consideration, validation and implementation. Its introduction would seem to be best done in a graduated way, beginning with component decision support in a manner that allows easy recourse to the current gold standard of surgeon judgement.

The latest evidence demonstrates that the use of ICGFA for resectional colorectal surgery is associated with lower AL rates [2, 12, 13, 14], and so it seems likely that ICGFA could become standard of care in patients undergoing colorectal surgery. The technology and pharmacological agent are increasingly available, certainly in developed healthcare systems, and while cost‐effectiveness is unproven formally the high economic burden of AL facilitates the inclusion of additional methods to offset the risk. However, additional potential barriers exist with regard to concerns about variability in user interpretation, the need for experience to maximize yield of return from such investment and the need to allow new users to quickly gain the same benefit as experienced users have done for their patients [15, 30]. While efforts have been made to develop quantification methods to evaluate these ICGFA signals and negate the subjectivity of their interpretation [31, 32, 33] no validated systems yet exist, with a significant issue being the multifactorial nature of AL, meaning that clinically correlating quantitative findings to patient outcomes is difficult. The results of the aforementioned studies [2, 12, 13, 14] infer that experienced users can reliably interpret the ICGFA imagery with better patient outcomes, and so AI models, grounded in expert surgeon interpretation rather than quantitative figures, could provide a more useful method of computational support [34]. This concept is not just applicable to colorectal surgery. In thyroid and parathyroid surgery, it has been demonstrated in a RCT that experienced surgeons can reliably predict vitality of the parathyroid gland based on their interpretation of intraoperative ICGFA signals [35]. Subsequent work has demonstrated that machine learning methods can discriminate parathyroid gland vitality in concordance with these experienced surgeons [36]. Furthermore, such a method could help document not only that ICGFA was used, but that it was deployed and interpreted reasonably irrespective of the clinical outcome, supporting the intraoperative decision‐making.

While some studies have successfully quantified ICGFA signals with varying degrees of clinical correlation [31, 32, 33], the vast majority of this work has been done post hoc after completion of the surgery, as real‐time quantification has proven difficult, with few studies achieving it [18, 19, 20, 21]. This has been a significantly limiting factor in the development of any clinically usable systems, given that postoperative quantification can have no impact on clinical outcomes. Quantification software is not inbuilt into most fluorescence platforms, and so additional software on a separate device that can fit into the existing surgical workflow is required to analyse the signals generated from the fluorescence platforms. Our proof‐of‐concept work utilized a laptop (with 11th Gen Intel® Core i7‐1185G7, NVIDIA‐T500 2GB technical specifications) to test the algorithm in real time. However, this process was cumbersome and so the method has since been migrated to a graphics processing unit (NVIDIA Jetson) which enables much faster analysis that is incorporated more seamlessly into the established surgical workflow. This method will be employed at some but not all sites to enable the study's secondary outcome of assessment of algorithm integration by means of surveys and interviews with the involved surgical teams.

While our initial proof‐of‐concept work has shown that an AI model, grounded in expert surgeon interpretation, is both achievable and can accurately represent this interpretation of novel ICGFA imagery [21], it now needs to be assessed for repeatability and generalizability (and indeed user feedback). In this work, the model was trained using 20 patient videos, with five videos used for validation before unseen testing on 28 videos. From this dataset, analysis of model performance indicates that in fact ten video recordings is likely to be sufficient to train the model. The method utilized here is not raw video/image analysis for which very large datasets are needed, rather the fundamental component is time series analysis based on ICG inflow which generates a strong signal that experienced human users are capable of interpreting readily with high consistency and concordance [15]. The multicentre structure of this study will permit the inclusion of various expert surgeons’ interpretations, ICGFA imagery systems and patient populations to test the generalizability of both the AI method and the algorithm. A further interesting aspect of this work is whether experienced users are consistent with each other and, if so, how consistent. While our preceding work suggests that they are, and for RCTs to show benefit they also should be, this needs to be examined, and will be in this work. Generating an algorithm for each site will enable this and provide a useful working model for testing in the event that an ‘aggregated expert’ algorithm proves less accurate than the centre's own expert in any specific site. As much as variability may help with generalizability, there is also a chance it will worsen performance, so the trial design enables this to be assessed and proven.

Issues relating to data sharing and privacy need to be addressed in any such study, and it is important, and hopefully useful, here to present our approach to such considerations. GDPR first came into effect across the European Union in May 2018 [37] and governs the processing of personal data. Intraoperative surgical videos linked to patient details and used to developed and test the algorithm in AUGUR‐AI clearly fall within the remit of GDPR under the classification of personal data. However, the FAIR (findability, accessibility, interoperability, reusability) guiding principles for scientific data management and stewardship [38] require that data are made ‘as open as possible, as closed as necessary’—so finding a data sharing solution that adheres to both can be difficult. The two options that have been proposed are the zero‐risk or risk‐based approaches [39]. The zero‐risk method would involve irreversible anonymization of the data so that it would not be possible to link them back to the patient. While this seems the best option in terms of privacy, it would have a significant impact on the reusability of the data generated from the study. As previously discussed, AL is a multifactorial event and so having reference to specific patient and operative risk factors is important. The other issue is that the exact method of ensuring such anonymization (and indeed even if full anonymization is even actually possible) is unclear [38]. The alternative risk‐based method evaluates the likelihood of reidentification and establishes technical and organizational measures that can be utilized to minimize this risk. In consultation with data protection officers from both our clinical and university sites, a common ground interpretation was agreed to follow the latter approach with pseudonymization of the video files by the clinical sites prior to sharing, with the identification key retained at the clinical sites with specific patient consent as underlying grounds.

The recently enacted AI Act in the European Union permits notified authorities access to training, validation and testing datasets to verify compliance [40]. This means that data must be retained in a format that facilitates such assessment in the event of onward advance towards a clinical device. Discussion with the Irish national regulatory agency (the Health Products Regulatory Authority) also confirmed that this study design could allow results supporting the performance of the fundamental algorithm to be advanced in supported of a definitive trial of any first‐in‐human study of final device deployment as is required for CE marking. Importantly, too, the PIL specifically includes the intention to translate any positive outcome into a clinically approved, commercial product and allows opt‐out of the collected data to be used for this purpose.

In conclusion, the AUGUR‐AIM study, if successful, will validate the use of an AI model to interpret ICGFA imagery in real time to the level of an expert ICGFA user, building on our previous work [21] in a larger, more diverse patient and surgeon population. This could allow progression to future work to develop the AI model into a usable clinical tool that could provide decision‐support to novel/infrequent ICGFA users and documentary support of the decision made by experienced users.

AUTHOR CONTRIBUTIONS

Philip D. Mc Entee: Writing – original draft; data curation; methodology. Patrick A. Boland: Data curation. Ronan A. Cahill: Conceptualization; methodology; supervision; writing – review and editing.

FUNDING INFORMATION

Internal.

CONFLICT OF INTEREST STATEMENT

Philip D Mc Entee and Patrick A Boland's roles as research fellows are funded by the Horizon Europe CLASSICA study and the European Union. Ronan A Cahill is named on a patent filed in relation to processes for visual determination of tissue biology, receives speaker fees from Stryker Corp and Ethicon/J&J, consultancy fees from Arthrex, Astellas, Diagnostic Green and Touch Surgery (Medtronic), research funding from Intuitive Corp and Medtronic as well as recently from the Irish Government (DTIF) in collaboration with IBM Research in Ireland, and from EU Horizon 2020 in collaboration with Palliare and, currently, from Horizon Europe in collaboration with Arctur.

ETHICS STATEMENT

This project will be carried out in accordance with the World Medical Associated Declaration of Helsinki on the Ethical Principles for Medical Research involving human subjects. Ethical approval will be obtained from the local ethics committee prior to enrolment at each clinical site and has already been obtained from the Research Ethics Committee at Mater Misericordiae University Hospital (reference 1/378/2450). Participants may choose to withdraw early from the project at any time without the need to explain their decision.

Supporting information

Appendix S1.

Mc Entee PD, Boland PA, Cahill RA. AUGUR‐AIM: Clinical validation of an artificial intelligence indocyanine green fluorescence angiography expert representer. Colorectal Dis. 2025;27:e70097. 10.1111/codi.70097

DATA AVAILABILITY STATEMENT

Data sharing is not applicable to this article as no new data were created or analysed in this study.