Impact of Image Latency and Frame Rate on Simulated Remote Robotic-Assisted Neurovascular Procedures

Abstract

BACKGROUND AND PURPOSE:

The implementation of remote procedures represents the ultimate goal of the robotic development in the neurovascular field. Studies from remote cardiac interventions established a maximum latency threshold of 400 ms, however, no data are available for neurovascular procedures. The aim of this study was to define the maximum acceptable latency and minimum refreshment frame rate (RFR) for neuroendovascular procedures in a simulated remote setting.

MATERIALS AND METHODS:

Using a virtual simulator and an endovascular robotic arm, 7 operators performed 8 simulated aneurysm and stroke treatment interventions (4 manually and 4 robotic-assisted), during which video display of the intervention was randomly altered with different latencies (100, 250, 450, 600, 800 ms) and RFR (10, 15, 25, 30 frames per second [fps]). Operators rated the acceptability of each latency and RFR by using a modified acceptability score (mAS) and an independent observer recorded the number of dangerous uncontrolled movement (DUMs).

RESULTS:

Maximum acceptable latency (defined as a minimum mAS of 85%) was defined at 100 ms for manually performed procedures and at 250 ms by using robotic-assistance, whereas minimum acceptable RFR was defined at 15 fps. A total of 55 intracranial DUMs were recorded, most of which occurred at latencies ≥450 ms (49/51) and with RFRs of 10 fps (4/4). Time intervals were shorter for manual procedures, although not significantly, and for experienced operators.

CONCLUSIONS:

Latency during simulated neurovascular interventions influences operator performance, judgment, and confidence and maximum thresholds (250 ms) seem to be lower than those previously reported from remote cardiac interventions. In this experimental setting, RFR seemed to have a lower impact in terms of acceptance rates. Latency and RFR represent relevant parameters to define and monitor in remote environments to maximize safety.

SUMMARY

PREVIOUS LITERATURE:

Although remote robotic-assisted procedures may provide a faster access to stroke procedures in the future, data concerning the acceptable parameters in terms of procedural latency are poor. Indeed, only studies from remote interventional cardiology procedures and experimental setting are currently available, suggesting a threshold of acceptable latency at 400 ms. There is currently a lack of investigations focusing on the impact of connection parameters on remote neurovascular procedures.

KEY FINDINGS:

A maximum acceptable latency of 250 ms was defined under robotic assistance in mechanical thrombectomy and aneurysm embolization procedures. The minimum accepted RFR was 15 fps. Dangerous uncontrolled movements were observed with higher latencies (450 ms or higher) and the lowest RFR (10 fps).

KNOWLEDGE ADVANCEMENT:

Acceptable latencies seem to be lower for remote neurovascular procedures compared with cardiac interventions. These findings represent the first steps to define the acceptable parameters to develop remote robotic-assisted neurovascular procedures.

Since 2019, the feasibility of robotic neurointerventions has been demonstrated successfully and it has established promise for remote robotic-assisted procedures.^{1 -3} Endovascular treatment of brain aneurysms and acute ischemic stroke (AIS) have proved to dramatically change the outcomes of patients, but this expertise is often centralized in large centers leading to long hospital transfers and delayed treatment times.⁴ In particular, AIS could represent a meaningful target for remote treatments, since these would allow faster access to mechanical thrombectomy (MT) for patients in remote areas.⁵ Among the parameters to consider for remote robotics, the connectivity and, in particular, the latency may affect operators’ confidence and procedure safety while performing remote interventions. Historically, there have been a very limited number of remote surgeries performed in general across all specialties, and the connectivity requirements may vary between different interventions.^{6 -9} Furthermore, there are no specific data for neurovascular procedures. The aim of this study was to use simulated remote neurovascular interventions to safely assess the maximum acceptable latency and the refreshment frame rate (RFR) of video streaming for neuroendovascular procedures to provide reliable preliminary results and set expectations for future remote robotic-assisted settings. This article follows the Strengthening the Reporting of Observational Studies in Epidemiology checklist.

MATERIALS AND METHODS

We conducted a series of double-blinded experiments at our institution between May and September 2023. Using an experimental setup in a simulated remote environment, we evaluated the impact of various connectivity parameters on robotic-assisted MT and aneurysm treatment interventions. This initiative was formally reviewed by institutional authorities at the St. Michael’s Hospital, Toronto, and deemed to neither require research ethics board approval nor written informed consent from participants.

The Experiment Setup

We recreated a remote setting for neurovascular procedures by using a virtual procedure simulator (VIST lab G5 simulator; Mentice AB). The main monitor of the simulator (“real” monitor) was extended to a second, cabled monitor (FlexVision monitor; Philips) to create the “remote monitor” on which latency and RFR were changed randomly by a homemade algorithm for each step of the procedure. The operators were only able to view the remote monitor. Two independent, expert observers had access to both the real and remote monitor for external assessment of experimental parameters that were not communicated to the operator (Fig 1). All subjects (both operators and the independent experts) involved in the experiment were blinded to the artificial latency and RFR that were projected on the remote monitor during the execution of the procedure. The fluoroscopy frame rate of the Mentice simulator was set at 12 frames per second (fps), which is the standard fluoroscopy frame rate set on our angiographic systems for neurointerventional procedures.

FIG 1.

Experimental setup. Op = operator; HDMI = high-definition multimedia interface; DVI = digital visual interface; 1 and 2 = observer 1 and observer 2.

Operators

Seven neurointerventionists from various specialties (3 neuroradiologists, 3 neurosurgeons, and 1 neurologist) participated in this study. Operator assessments were categorized based on their robotic experience as expert if they had already performed at least 10 clinical procedures or 100 simulated procedures by using robotic assistance and as nonexpert when these criteria were not fulfilled. Nonexpert operators underwent preliminary robotic training on the simulator (15 procedures per operator) to familiarize themselves with the system. During the experiments, each operator performed 4 procedures twice: 4 procedures dedicated to latency assessment and a second group of 4 procedures dedicated to RFR, with a washout period of 1 month between the 2 sessions. From the 4 procedures in each group, 2 of the simulations were clot removal by using MT techniques (1 manual and 1 robotic-assisted) and 2 were aneurysm embolization procedures by using stent-assisted coiling techniques (1 manual and 1 robotic-assisted). The operators performed the procedures by using the remote monitor as reference where latency and RFR were changed randomly, to which they were blinded to the latency and RFR values. At the end of each procedure step, operators were interviewed on comfort and confidence by using a modified acceptability score (mAS), described later.

Definitions of Latencies and RFR

The intrinsic latency (IL) of the cabled monitor transmission and of the Mentice system was preliminarily calculated and the total latency resulted in an average of 30 ms. We defined 5 values of simulated latency (SL): 100, 250, 450, 600, and 800 ms. To provide a realistic assessment, the SL values included the IL (30 ms) and the amount of latency needed to reach the SL (ie, to reach a SL of 400 ms, 370 ms were added to the IL). We also tested 4 altered RFRs of the streamed images: 10, 15, 25 and 30 fps during simulated procedures.

Neurovascular Procedure Details

For the stroke treatment, we selected mechanical thrombectomy for a right-sided, ICA terminus occlusion that was performed manually by using a combined technique and a left M1-middle cerebral artery occlusion by using a stand-alone stent-retriever technique by using robotic assistance. This choice was done to simulate scenarios that are consistent with the current clinical practice and with the highest technical availabilities for both manual and robotic approaches. For the aneurysm procedure, we selected the treatment of a right-sided, unruptured, carotid-ophthalmic aneurysm by using stent-assisted coiling (further simulation procedure details including devices used are provided in the Supplemental Data). Each procedure was subdivided into the following steps: 1) the extracranial navigation (EN; from the femoral artery to the internal carotid artery, for both the aneurysm and stroke procedures), 2) the intracranial navigation (IN; including the navigation of the microcatheter for stent delivery, the access to the aneurysmal sac in the aneurysm treatment, and microcatheterization beyond the occlusion in the stroke treatment), 3) the stent deployment (with the purpose of stent-retriever ± contact aspiration positioning in stroke treatment and stent-assisted coiling in the aneurysm treatment), and 4) coiling (only for aneurysm treatment).

All robotic-assisted procedures were performed by using the CorPath-GRX system (Siemens Robotics) by using conventional (microcatheters and microwires) and simulated (coils, stent-retriever) devices that were registered in the Mentice simulator for the purpose of the study. Each of the aforementioned procedure steps were performed both manually and robotically, with the exception of the extracranial navigation which was excluded from the cases performed with robotic assistance, due to the known incompatibility of the 0.035-inch guidewire with the neuro GRX system.

Two precision tasks (PTs) were included in the aneurysm experiment to ensure consistency between operators. During PT 1, the operator was instructed to begin deployment of the stent precisely before the origin of the anterior choroidal artery and for PT 2, after coil deployment, it was instructed that the proximal marker of the microcatheter was to be placed precisely between the 2 markers of the detachment zone of the coil (Supplemental Data). Each step was not considered as completed until the precision task was completely successfully.

We performed an intraobserver agreement by comparing operator ratings for the same latency or RFR values during similar steps in different moments of the procedures. For example, we compared operator ratings for 2 separate navigation steps with the same latency, such as navigation of the microcatheter for stent delivery and navigation for the access to the aneurysmal sac.

Study Design, End Points, and Outcomes

The primary end point of the study was to define the maximum acceptable latency and minimum RFR values for remote, robotic-assisted neurovascular procedures. Latency/RFR were randomly altered during each step of the procedures by using a dedicated software, assuring that all the latencies/RFR values were sufficiently represented in each section. The operators performed each procedure step with different latencies and RFR and rated each of them before passing to the next latency/RFR. Some tasks were repeated with different latencies/RFR, such as the stent (or stent-retriever) deployment, the intracranial navigation in both aneurysm and stroke treatment, and the deployment of the first coil. The question that the operators were asked to answer was: “If you were performing this specific section of this procedure remotely, how confident would you feel with this latency/RFR?” For this purpose, a 4-point Likert scale (0–3), which is a modified version of the mAS utilized in the remote cardiac PCI study by Patel et al⁶ (Supplemental Data), was used to record the operators’ ratings. The scale was then dichotomized as acceptable or not based on acceptability scores 0–1 (= not acceptable) or scores 2–3 (= acceptable). All steps of the simulated stroke and aneurysm experiments were completed by all operators, and an acceptability rate (AR) was calculated for each latency/RFR value in each step of the procedures, by dividing the number of the operators rating 2–3 by the total number of the operators (ie, 5/7 observers rating 2–3; AR = 71.4%). The average AR, calculated by taking the average AR of all procedure steps, was then considered as the primary outcome. We defined an average AR of 85% as an overall accepted latency/RFR.

Secondary end points included the time spent to complete each procedure step with each latency/RFR, dangerous uncontrolled movement (DUMs), operators’ experience, and the clinical relevance.

For these purposes, all the simulated procedures were supervised by 2 observers. Observer 1 was responsible for blindly altering the latency and recording the time intervals, while observer 2 (also blinded) recorded the participant’s rating for each latency and RFR. Two independent experts recorded all the DUMs that occurred during the procedures. DUMs were defined as those involuntary maneuvers due to latency/RFR that could potentially result in a complication (mainly aneurysm or artery perforation; Supplemental Data) and these were registered only after consensus between the 2 observers and the operator. Clinical relevance was defined as the level of importance that the operator would give to the importance of low latency and high RFR in each specific procedural section. For example, we wanted to determine whether a longer latency may be more tolerable during navigation of the abdominal aorta for access compared with the moment one is navigating the microcatheter into the aneurysm dome for coiling. All the operators provided their personal rating by using a 4-point Likert scale ranging 0–3, where 0 was defined as not important and 3 as highly important.

Statistical Analysis

An average intraobserver agreement value was determined by taking the mean of each rater’s intraobserver percent consistency. The interobserver agreement was calculated by using Krippendorf α for both dichotomized and nondichotomized scales. mAS scores were considered to build the average AR, whereas DUMs were expressed as frequencies. Only DUMs occurring in comparable procedure sections (intracranial navigation, stent deployment, and coiling) were compared. When possible, χ² and Fisher exact tests were used to assess differences between groups. Student t test was performed to compare means between 2 groups and ANOVA was used for comparisons between multiple groups. The definition of the average AR and all the comparisons between manual and robotic procedures were performed excluding the frequencies and data of the EN section, since this was not performed in the robotic-assisted procedures. P values < .05 were considered statistically significant. Statistical analysis was performed by using SPSS (Version 19; IBM).

RESULTS

The overall results regarding the primary end point are summarized in the Supplemental Data.

A total of 840 procedure steps were rated to assess the maximum acceptable latency and RFR. For each stroke or aneurysm intervention, 175 steps were performed with robotic assistance, and 245 steps were done manually. Four operators were considered expert and 3 nonexpert (see Materials and Methods for classification criteria). Intraobserver agreement was excellent (κ = 0.84).

Latency

A maximum acceptable latency of 250 ms was observed for stent-assisted coiling performed with robotic-assistance, whereas operators deemed only 100 ms as acceptable for the same procedure performed manually, with a trend toward 250 ms latency (80.9%). Similar results were observed for stroke procedures for both robotic-assisted and manual simulations, however, 250 ms was considered as an acceptable maximum latency for both manual and robotic modalities (Supplemental Data).

Column averages in the Supplemental Data (average AR% per section) show no significant differences for average acceptable latencies between the specific procedural steps of MT performed under robotic assistance, nor between procedural steps for robotic-assisted stent-assisted coiling procedures. For manual procedures, there were also no differences observed in average acceptable latencies between the specific procedural steps of MT. On the other hand, the average overall acceptable latencies for the different procedure steps for the manual aneurysm treatment were higher for the stent deployment (71.4%), while the lowest rates were observed during the intracranial navigation and coiling (45.6% and 42.8%, respectively).

The average time needed to complete each section of the procedure was globally longer in robotically assisted procedures for both stroke and aneurysm stent-assisted coiling, although significant differences were not observed (Supplemental Data). A similar trend was observed comparing the average time spent for each specific step, however no significant differences were observed except for the stent deployment in aneurysm treatment (70.0 versus 44.7 seconds, P = .046). Furthermore, shorter time intervals were recorded for expert operators compared with nonexpert ones in both procedure types (aneurysm versus stroke) and modalities (manual versus robotic-assisted) (Supplemental Data).

A total of 67 DUMs were registered during the latency experiments, however, only 51 of these were included in the analysis (16 occurred in the EN section and were excluded) (Supplemental Data). These were more frequent during the stent-assisted coiling than in mechanical thrombectomy (37 versus 14, P < .001). Most of the DUMs occurred with latencies ≥450 ms (49/51; P < .001). The number of DUMs was slightly higher in the manual group compared with the procedures completed with robotic-assistance (29 versus 22, P = .56), although not statistically significant, whereas no significant differences were observed between the specific procedural steps.

RFRs

Contrary to latency, almost all RFR values were accepted in both procedures (Supplemental Data). An RFR of 10 fps was the least accepted frame rate, having not achieved an average minimum acceptability in manual procedures for aneurysm treatment nor in robotic-assisted stroke procedures.

Average acceptance rate was high in all sections of both aneurysm and stroke procedures, without significant differences according to the manual or robotic-assisted technique.

As previously observed for latency, we observed longer time intervals for each procedure step for both aneurysm and stroke treatment performed by using robotic-assistance, although not statistically significant. No relevant difference was observed when comparing different procedural sections (Supplemental Data).

Five DUMs occurred during experiments for RFRs (1 occurred during EN and was excluded from the analysis), with significantly less frequency compared with the latency-based experiments (4 versus 51, P < .001) and all of these occurred with a RFR of 10 fps (Supplemental Data).

DISCUSSION

We report for the first time a comprehensive assessment of the impact of 2 connectivity parameters in a remote simulated, experimental setting of robotic neurovascular procedures. With an acceptability score threshold set at >85%, our results suggest that connectivity parameters should aim for a maximum latency of 250 ms and a minimum RFRom of 15 fps for maximum safety during simulated remote acute ischemic stroke MT and brain aneurysm repair procedures.

Latency and RFRs in the Literature

Preliminary studies in other specialties of interventional radiology assessed these parameters by using either animal models or simulated environments. Legeza et al¹⁰ reported the results of a series of experiments performed by using the CorPath-GRX system on a simulator for percutaneous coronary interventions (telestent placement procedures) and on a swine model (15 consecutive attempts of navigation of a guidewire from the femoral artery to the coronaries). The control system was connected to the robotic arm via a virtual private network (VPN) and located at longer distances each day of testing (up to 103 miles away). The simulated fluoroscopy was streamed over the VPN. In another study, Madder et al¹¹ reported a series of experiments performed by 3 different operators (a vascular surgeon, a neurosurgeon, and a cardiologist) by using the CorPath-GRX on a porcine model. A series of 65 attempts of endovascular navigation of a guidewire from the femoral to the coronary/carotid arteries was performed by using different latencies ranging from 400–1000 ms. In both studies a tolerable latency was defined as ≤400 ms. The first endovascular procedure performed remotely on a patient (telestent placement) was accomplished by Patel et al.⁶ They reported a series of 5 remote robotic-assisted coronary stent placement procedures successful performed between Arkhadsham and Ahmedabad in India. In this study, procedures were performed by using an optic fiber Internet connection with a median latency of 53 ms, which the operator considered to be imperceptible.

Our study results show a lower acceptable latency threshold of 250 ms in simulated remote neurovascular procedures. This is likely because coronary interventions differ from neurointerventions in various aspects. There are anatomic differences including tortuosity, compliance, and pulsatility, which may influence the endovascular navigation, controllability of the devices, and operator’s confidence.¹² Furthermore, interventional cardiologists are accustomed to performing procedures based on images with intrinsic heart-beating motion artifact, which could increase their tolerance to latency, compared with the more fixed images that are observed in neurovascular procedures.

Contrary to latency, data regarding the RFR for remote video streaming are either not available or were not reported in the previous experiments or clinical series. In our experiments only 1 RFR value did not reach a minimum acceptable rating (10 fps) in both the stroke and aneurysm procedures. This is likely because this is the only frame rate tested that falls below the 13 fps typical maximum fluoroscopy frame rate used in image acquisition during neurointerventional procedures. Although not evaluated in the remote PCI studies above, we would hypothesize that the acceptable RFR threshold for PCI would have been higher than the acceptable threshold found in our study, as PCI procedures typically utilize a higher fluoroscopy frame rate of 30 fps to account for the high motion due to the contractility of the heart. Among the 4 different values of RFR tested in our study, 20 fps is considered the standard value for video streaming in the main live neurovascular conferences, while 2 of these represent better conditions of streaming (25 and 30 fps). To the best of our knowledge, no data comparing latencies or RFR in different types of neurovascular procedures are currently available.

DUMs and Time

One of the most feared situations in remote procedures would be the loss of the control of the devices by the operator due to excessive latency or low RFR of streamed images, which could potentially lead to an intraprocedural complication, such as a perforation or dissection.¹³ We presented a large data set of experiments (840 total maneuvers performed, 420 per each type of treatment). However, we included in the analysis only the experiments performed in the intracranial sections (700 maneuvers), excluding the EN where we observed a total of 55 DUMs with a rate of 0.78% (55 DUMs/700 procedure sections). The very low frequency of these events seems comparable to the reported clinical series of aneurysm and stroke treatments.^14,15 Interestingly, the number of events was higher during manually performed procedures rather than those performed by using robotic assistance, whereas the time spent per each section was longer in robotic-assisted procedures (Supplemental Data). These data, although not statistically significant, may underline an association between the time spent to perform each step, the number of events, and the technique of the procedure. It may be suggested that the lower number of DUMs during the robot-assisted procedures is due to the fact that robotic procedures were slightly longer, which seemed to provide the operator with more control of the devices during longer latencies. It is important to note, however, that the rate of DUMs may have been higher in our study due its double blind design. We recommend that for clinical procedures, that the latency be constantly monitored by the system and presented to the operator so that they may adapt to the condition and operate with more caution, which will likely result in less DUMs and improved procedure safety.

Operators’ experience was relevant in this experiment as the experts performed a lower number of DUMs, although they needed longer time intervals to complete the different procedure sections for both aneurysm and stroke treatments (Supplemental Data). The impact of RFR seemed to be less relevant. Indeed, the operators felt that it was easier for them to adapt to RFR than to latency and that the latter could have a slightly higher impact on robotic-assisted procedures compared with manual ones. However, although RFR was associated with a significantly lower rate of DUMs and no differences in terms of time intervals, a potential impact on remote procedures could not be overlooked, particularly if the RFR drops below the fluoroscopy acquisition FR. Indeed, several operators underlined how RFR may produce scattered or “jumpy” images, which could pose a safety concern as streamed procedural imaging represents the main feedback for the operators during remote procedures. The frame rate selected for signal transmission can impact the volume and size of live video data that would need to be transmitted to the operator, thus lowering the RFR can significantly reduce the bandwidth requirements for signal transmission during the procedure which may in turn improve network stability by reducing data packet loss and network jitter. Further investigations will be necessary also to assess the frequency of DUMs in more difficult cases, which could potentially increase the risk of uncontrolled movements due to anatomic and technical complexity.

Procedure Type and Sections and Treatment Technique

We observed no overall differences in acceptability rates between the different procedure steps during mechanical thrombectomy and stent-assisted coiling procedures (Supplemental Data). However, the stent deployment steps, for both stent-retriever and intracranial stent placement, resulted in the highest average acceptance rates for both latency (100–850 ms) and RFR (10–30 fps). According to the operators’ feedback, the task of stent deployment is the one where the maneuvers are more cadenced, and movements are mainly based on small strokes on the pusher wire with simultaneous controlled pull back of the microcatheter both for manual and robotic-assisted interventions. This observation would explain the higher acceptance rates, because during both latency and RFR experiments, the operators felt to have more control on this specific task also with longer latencies and lower RFRs. Extracranial navigation was the section in which the highest number of DUMs occurred, however, this section was completed only manually and most of the operators felt that the maneuvers performed in this section were less risky than those performed intracranially. As expected, intracranial navigation was the procedural step in which operators were more demanding in terms of average acceptance and where interval times were longer, particularly for aneurysm procedures during latency experiments (Supplemental Data). This is likely because these steps included navigation of the microwire and microcatheter beyond the clot and the intracranial ICA beyond the aneurysmal neck eventually into the sac, which were considered higher risk maneuvers compared with other procedural tasks.

Limitations

This study has limitations. This study used a simulation-based environment and, therefore, it is affected by the intrinsic limitations of the simulation system. Indeed, for example, some operators acknowledged that for stroke treatment, the resistance to cross the occlusion was not realistic, and this could have had an impact on the assessment of the acceptable latency/RFR, although all the operators were asked to take this into account. Furthermore, some procedural complications, such as the thromboembolic events, could not be evaluated. Additionally, we tested the 2 network parameters separately, whereas the connectivity of such remote systems will have to consider multiple parameters at the same time. Indeed, image resolution, throughput, package loss, connection type (optical fiber, 5G, satellite) and jitter would also have an impact on the operator’s experience and are important parameters to test and set acceptability standards for to ensure a proper connection between the remote site and the operator. Jitter, for example, which is defined as the range of latencies that can be measured in a timeframe, was assessed for online video game performance and, interestingly, was more impactful when assessed together with latency compared with latency alone.^16,17 Moreover, we have used a cable connection for the purpose of the study, whereas in the future remote procedures would use different types of connections, such as Wi-Fi. Also, it has to be acknowledged that we have tested the impact of latency based on the visualization of the simulated radiographic images, however we might expect latency in sending and executing commands. Finally, we also acknowledge that a further intrinsic latency of the x-ray generating system might be added to the total amount of the maximum acceptable latency.

CONCLUSIONS

In a simulation-based remote environment, image latency seems to influence the operator judgement and confidence. Maximum acceptable latency was ≤250 ms for both aneurysm and stroke procedures performed by using robotic assistance and ≤100 ms for the same procedures performed manually. Refreshment frame rates were considered acceptable at 15 fps or higher. Dangerous uncontrolled movements were more frequent in manual procedures and were inversely associated with time spent to complete procedural sections. These preliminary observations may serve as a baseline for more comprehensive experiments to establish the minimum requirements for remote neurovascular clinical procedures.

ABBREVIATIONS:

AIS

acute ischemic stroke

AR

acceptability rate

DUMs

dangerous uncontrolled movements

EN

extracranial navigation

fps

frames per second

IL

intrinsic latency

IN

intracranial navigation

mAS

modified acceptability score

MT

mechanical thrombectomy

PCI

Percutaneous Coronary Intervention

PT

precision task

RFR

refreshment frame rate

SL

simulated latency

VPN

virtual private network