The digital heart–lung unit: applications of exponential technology

Abstract

Exponential technologies such as virtual reality (VR), computational modelling, and additive manufacturing have emerged in the field of cardiology and cardiothoracic surgery. An increasing number of publications that evaluate the clinical role of these technologies are becoming available. Moreover, there is an increase in the number of hospitals and departments that have implemented digital and exponential solutions in clinical workflow. In our centre, we have adopted various exponential technologies in order to improve clinical pre-procedural workflow, patient care, and training and education. In order to provide our view and approach on the implementation of these technologies, in this article, we provide an overview of the currently applied modalities including immersive VR, three-dimensional (3D) computational modelling, VR-based simulations, and additive manufacturing (3D printing). Moreover, we present the potential of these applications in cardiovascular and cardiothoracic medicine, and additionally, we will provide key facilitators, challenges, and recommendations to adopting these technologies in clinical practice.

Introduction

Exponential and advanced technologies such as additive manufacturing [three-dimensional (3D) printing], computational modelling, and virtual reality (VR) have emerged in the field of cardiology and cardiothoracic surgery. These innovative technologies can help create an innovative ecosystem, which enables personalized healthcare delivery through digital health and interdisciplinary collaboration. Herein, we describe our digital heart–lung unit that combines these principles for interventional and surgical planning, training, and education. Furthermore, we will present our view and approach on the clinical use and potential of these technologies in the future, and additionally provide key facilitators, barriers, and directions/recommendations to adopting and implementing these technologies in other hospitals.

Computed tomography-derived 3D computational modelling

Established work-up for a broad range of interventions in structural heart disease include echocardiography and computed tomography (CT).^1–4 Although echocardiography provides essential information on severity and location of functional defects (i.e. stenosis/regurgitation/leaks), its limitations in spatial resolution could lead to suboptimal measurements of corresponding geometries (i.e. annuli/orifices/calcium). Two-dimensional CT evaluation allows sizing, but measurement of asymmetrical shapes in x, y, and z planes are simplified and structural interactions may be left obscured. Conversion of a contrast-enhanced, electrocardiogram-gated, CT into a personalized 3D computational model (3DCM), could provide additional insights. In our practice, we have implemented the use of 3DCM in preprocedural planning of complex adult cardiac surgery and complex transcatheter interventions in structural heart disease.⁵ Specifically, 3DCMs are currently used for anatomical challenging situations, requiring additional spatial insights. For instance, in the adult population, these models are applied to plan for transcatheter mitral valve replacement (TMVR), transcatheter paravalvular leak (PVL) and valve leaflet perforation closures, and left atrial appendage closure (both percutaneously and thoracoscopically). Over the course of 2.5 years, 28 such cases have been performed in our centre (although numbers have been growing initially, we have seen quite a decrease in the number of these procedures due to the COVID-19 pandemic and the decreased intensive care-capacity for post-operative monitoring). In transcatheter aortic valve replacement (TAVR), 3DCMs with pressure contact simulations and computational fluid dynamics are used to evaluate the post-procedural risk of conduction disturbances and PVLs.⁶ However, these models are currently not used routinely, as more advanced simulations require increased processing time. Prospective research is ongoing to define the impact of these models on procedural strategy and outcome. For instance, For TAVR planning, we are currently carrying out a study in which we are creating preoperative simulations of TAVR procedures to study the size and implantation depth of the valve and also to predict and analyse the risk of PVLs and compression on the electrophysiological conduction system. Besides clinical endpoints (e.g. mortality, stroke), other important outcomes in this study are the incidence of PVL and conduction abnormalities and need for pacemaker implantations.

Essential for 3DCM implementation is the availability of high-quality CT images acquired in predetermined cardiac phases (Table 1). Computed tomography images are exported in Digital Imaging and Communications in Medicine (DICOM) files and loaded into dedicated segmentation software which allows creation of a volume rendered or mesh-like computational model^5,7 (Figure 1). Segmentation is a computational technique which allows labelling of anatomic structures by clustering regions with similar radiographic properties. With this, the user is able to create an in-colour 3D model of any anatomical structure. Segmentation is currently available by using both semi- and fully automated artificial intelligence-based techniques and requires a technician familiar with cardiovascular (or any other) anatomy. After segmentation, evaluation of 3D anatomy and measurement of defects/orifices can be performed with most segmentation software packages or after exporting 3DCMs into secondary modalities (i.e. 3D pdf). Several open-source segmentation tools such as 3D Slicer and itk-SNAP^8,9 are available and can be used for 3D labelling (segmentation) and post-processing. In addition, commercial segmentation services and software platforms (i.e. Materialise Mimics™ Enlight) are available that also enable semi-automatic and automated segmentation to create 3DCM that can be used for procedural planning.⁵ Other software packages allow simulation of contact pressure and/or computational flow dynamics.⁶ (Figure 1). Enhanced 3D appreciation is obtained by exporting the 3DCMs to 3D pdf files that also permits instant, preprocedural, projection in the cathlab and the operating theatre monitors. This road-map increases operators’ understanding in the procedure but can also facilitate heart-team discussions to evaluate procedural feasibility/safety.¹⁰

Table 1.

An overview of facilitators (green boxes) and barriers (red X, representing factors that will become barriers when not available/present) in the implementation of digital health solutions for patient care, education, and training

	3DCM	3D-printing	VR	Digital simulations
Facilitators
Multidisciplinarya and dedicated team	☑	☑	☑	☑
Off the shelf hardware components (i.e. HMD, 3D printers, printing ink, physical simulation models)	☑	☑	☑	☑
Software developers	☑	☑	☑	☑
(integrated) Access to Patient Archiving and Communications System (PACS) on workstation	☑	☑	☑
Strategic (public/private) partnerships	☑b	☑b	☑	☑
Financial provision	☑	☑	☑	☑
Validation strategies	☑	☑	☑	☑
Education specialists	☑	☑	☑	☑
Resident training programme		☑	☑	☑
Dedicated CT acquisition protocols	☑	☑	☑
Standardized cut-offs for visualization and fitting	☑	☑	☑
Easy-access sharing platform for virtual models	☑	☑	☑	☑
Software platforms allowing segmentation, creation of surface models, virtual measurements and geometrical fitting	☑		☑
Materials for bench testing		☑	☑	☑
Scientific evidence (see references)	☑	☑	☑	☑
CE/FDA certification	☑	☑	☑
Barriers
Interdisciplinarity	X	X	X	X
Large scale evidence (lack of clinical trials/randomized trials)	X	X	X	X
Simulation capabilities (tissue/device)	X		X	X
Funding	X	X	X	X
Local production facilities		X
Industrial partners for outsourcing software/hardware development/production	X	X	X	X
Decentralized data (i.e. imaging) sources	X	X	X
Attitude towards new technology from clinicians	X	X	X	X

CT, computed tomography; HMD, head mounted display; MDR, medical device regulation.

^aEngineers, software specialists, physicians, researchers, surgeons, cardiologists, policy advisors, MDR-specialists, radiology technicians, imaging specialists, biomedical engineers, machine learning specialists.

^bDepending on local or outsourcing

Figure 1.

Examples of use-cases of various digital health solutions in clinical practice at the Thoraxcenter. (A) 3D print of the aortic root (translucent) after transcatheter aortic valve implantation (pink), aortic view: a paravalvular leak, located near the left coronary, is closed with a cardiovascular plug (black). Calcium is coloured blue. (B) 3D print of the left heart, surgical view: mitral annular calcification shows anchoring possibilities for transcatheter mitral valve replacement. (C) 3D computational model of the left heart and modelling of transcatheter mitral valve replacement. (D) A virtual transcatheter heart valve (yellow) is fitted in a previously implanted mitral ring. (E) 3DCM of the left heart. Surgical view: two paravalvular leaks are shown adjacent to a mechanical mitral prosthesis (blue). (F) An immersive virtual reality simulation of cardiopulmonary resuscitation. (F) An immersive virtual reality simulation of an operating theatre with surgical instruments and scrub nurse, providing an opportunity for resident training of surgical situations. (G) Screen captures of preoperative 3D volume rendering (in virtual reality) of segmented computed tomography scans of a patient (left image) with a left ventricular assist device (grey) with an aortic valve biological prosthesis (purple), a paediatric patient with double outlet right ventricle and ventricular septal defect (middle two images), and of 3D anatomy of the anatomical lung segments, bronchi, and pulmonary vasculature (right image). 3D, three-dimensional; LV, left ventricle; PA, pulmonary artery; RV, right ventricle; VSD, ventricular septal defect

Immersive virtual reality

Virtual reality is an emerging exponential technology that facilitates review of computer rendered digital images in an artificial 3D environment. In the past decade, there has been an increase of investigations using various forms of extended reality (such as virtual and augmented reality) for clinical applications in cardiology and cardiothoracic surgery.^11,12 In the currently available literature on VR in cardiovascular interventions and cardiothoracic surgery, VR is mostly used in preoperative planning of congenital heart surgery.¹² Incidentally, also some preliminary case studies have been reported in the field of lung surgery and transcatheter cardiac interventions.¹² Generally, VR is used to provide 3D representations of CT(A) images in an immersive environment. Some of the following advantages of VR for CT review have been mentioned repetitively in literature: more awareness of spatial orientation, increased realism, minor distractions from surroundings, enhanced depth perception, more accurate determination of surgical margins, better interpretation of morphology and anatomical abnormalities, and better accuracy in localization of small tumours. In our centre, we have already implemented VR for procedural planning of more complex cardiac and pulmonary surgery procedures, education of medical students, and anatomy training of residents and fellows.^7,13

CardioVR platform [developed at our centre together with MedicalVR (Amsterdam, The Netherlands)] can help review and understand (congenital) cardiac anatomy of newborns with complex intracardiac anatomy (Figure 1).⁷ PulmoVR platform is a VR and artificial intelligence-based platform and can be used for surgical planning of segment resection of the lung parenchyma for early-stage lung cancer (Figure 1).¹⁴ In this feasibility study, PulmoVR was successfully applied as a preoperative imaging tool to facilitate segementectomy planning. Interestingly, in 40% of the cases, the surgical strategy (e.g. an extended resection or another target segment resection) was adjusted based on the 3D-VR evaluation of the patient’s digital twin.

In both of the aforementioned VR tools, artificial intelligence-based as well as semi-automated segmentation of intrathoracic structures (i.e. pulmonary or coronary arteries) are used to enhance 3D in-colour visualization of anatomic target structures. In a multidisciplinary team consisting of VR developers, technicians, surgeons, and resident physicians, specific procedures are planned in VR. Commonly, surgeons request VR reconstructions for specific preoperative questions in the context of complex surgery. Subsequently, after loading CT DICOM files from our local Patient Archiving and Communications System (PACS), segmentations are prepared (by a technical team) and the combination of both DICOM and segmentation files are rendered in our VR planning software. The rendering of images in the software platform usually takes no more than 5 min. However, preparations and segmentations can take longer, depending on complexity and necessity of additional segmentation analyses. Currently, PulmoVR platform has already been used in a clinical scientific setting in 41 patients undergoing thoracoscopic pulmonary segmentectomy (showing critical surgical strategy changes, resulting in better surgical performance for both patient and surgeon), 1 patient with a large axillary tumour undergoing four-quarter amputation of the left arm with extended chest wall resection,¹⁵ 1 patient with a large mediastinal tumour encased between the great vessels, 1 patient undergoing tracheal repair, and 2 patients undergoing lung transplantation. With regard to CardioVR, we have recently published two manuscripts on the application of CardioVR in adult patients (n = 7) undergoing cardiac surgery.^7,13 In these studies, CardioVR showed that it was a valuable tool for surgeons in determining ideal surgical access, cardiopulmonary bypass cannulation strategies, and for preoperative awareness of certain anatomical details. Since the introduction of the CardioVR platform, also complex paediatric heart surgery [double outlet right ventricular repair (n = 2)] has been planned by using CardioVR. Currently, we are scientifically exploring the role of CardioVR and PulmoVR in other paediatric cardiac and pulmonary surgery, adult aortic surgery, transcatheter mitral valve intervention, and percutaneous transcatheter PVL closure. For instance, in context of the latter, we have studied the feasibility and applicability of 3DCMs and immersive VR models for preprocedural planning of percutaneous PVL closure by vascular plugs.¹⁶ Based on the results of this study, we found that the eventually placed vascular plug dimensions mostly corresponded with the dimensions measured on the 3D modalities such as 3DCM and VR when compared to 2D-CT and conventional ultrasonography.

We are currently awaiting CE certification of both PulmoVR and CardioVR platforms. After CE certification, these platforms will become commercially available and can be used clinically in other European countries as well. Until then, these platforms will be used in clinical-scientific settings as, for example, we are settings up a multicentre clinical trial to evaluate the benefits of PulmoVR in larger patient populations.

With regard to costs, initial investments in essential software and hardware (head mounted display, high performance computer, VR software platform, segmentation tools) might vary between €2000 and €8000. However, after purchasing essential elements, relatively small funding is needed to render images in VR and implement the technique in clinical practice. We believe that in the near future more VR software applications and VR hardware will become available to support clinical decision making in preprocedural planning. At last, in our experience, when a dedicated team is on site and all relevant hardware/software elements are physically available, time investment from clinicians can be minimized and is not more than common preprocedural planning using conventional techniques.

Future applications of VR may include more advanced preprocedural planning of catheter based cardiac interventions and navigational virtual bronchoscopy based-planning of pulmonary nodule sampling.^5,12,17 Finally, patients could also leverage the VR technology to dissociate from a procedure under local anaesthesia which may reduce pain sensation and enhance procedure satisfaction. An overview of the essential components and barriers to setup a successful VR team/unit for preprocedural planning assistance are described in Table 1.

3D printing

A patient-specific physical model generated with 3D-printing can promote medical teaching, exploration of valve function, preprocedural planning and dry testing to help determine device size selection (Figure 1).^18,19 Computational models converted into stereolithographic (STL) files serve as the blueprint for a 3D-printed models (3Dpm). Subsequently, adjustments are made (i.e. smoothing, colour-coding) to generate a file suitable for physical printing. It is vital to determine the goal/use of the 3D-print prior to STL-generation and subsequent printing, as different techniques yield different types of models.¹⁹ For example, a rigid model printed with fused deposition modelling (low-cost) or stereolithography (increased cost) could suffice if the purpose of the model is only educational, whereas 3Dpm including various parts with different degrees of flexibility printed with PolyJet (high cost) would be required if the models were used to fit actual devices. For the first two techniques, it could be cost-effective to produce 3Dpm on-site, especially in a high-volume clinic. Additionally, by producing on-site, time to completion can be reduced to 1 day. However, production of 3Dpm with the latter technique would be too costly to perform onsite as associated printers alone currently require investments of over >€20 000. Prior to the implementation of 3D-printing in daily practice, one should decide whether to outsource or to invest in local production. Currently, there are multiple commercial manufactures of medical 3Dpm which provide patient-specific cardiovascular models suitable for bench-testing. In our practice, 3Dpm complement 3DCMs in understanding a patient’s anatomy and performing virtual device implantation. In TMVR and PVL, 3Dpm are obtained for every patient of whom the virtual 3DCM reinforced procedural feasibility. A bench-test, implanting the intended device, is performed to confirm virtual work-up. The interventionalist and team review this model directly prior to the procedure. Preprocedural work up using bench testing in (accurately sized) 3Dpm in TMVR and transcatheter PVL closure is illustrated in Figure 1. Important factors in implementing 3D printing in clinical practice are outlined in Table 1.

Digital clinical and surgical simulations

For cardiothoracic surgery residents, the operating room (OR) and surgical ward represents the main area to acquire key surgical and clinical skills as well as developing other ‘non-surgical’ skills such as communication and decision making. However, unlike in the aviation world, no advanced or widely available training simulators are available to facilitate skills training outside the OR. It is already accepted that skills training outside the OR results in improved performance of residents.^20,21 Moreover, the recent COVID-19 pandemic has drastically reduced hands-on and clinical exposure of surgical residents, which stresses the need for more efficient and innovative training methods of learning. By leveraging serious gaming and extended reality technology, these challenges could be overcome in the future. Accelerated by the COVID-19 pandemic and stimulated by emerging VR technology, several groups, including ours, are currently developing tools to facilitate training and education for clinically relevant scenario’s (Figure 1).²² For example, with our technical partners we have developed various clinically relevant simulations to allow resident training in immersive VR. Currently, we have developed (together with our partners; Zan Mitrev Clinic and Distant Point LLC, Skopje, North Macedonia) a cardiopulmonary VR resuscitation simulator and VR extracorporeal circulation simulator in order to facilitate training of junior physicians, surgeon residents in training, and junior clinical perfusionists (Figure 1). These proof-of-concept simulators are now being validated scientifically, which is essential for structural clinical and educational implementation. Currently, we are also engaging with several other stakeholders in developing novel augmented reality-based simulators to enhance surgical skills training outside the OR in a realistic and safe environment. When interested in developing simulations or serious gaming applications, a correct and stepwise approach in validating the simulations and digital scenarios should be taken into consideration. For instance, face validity, content validity, construct validity and concurrent validity are essential in the steps towards implementation of simulations for practical use.²³ In Table 1 other facilitators, but also barriers/challenges in creating digital tools for simulation and training are presented.

Challenges in advancing exponential technology in clinical practice

Although the implementation of 3D technology in cardiothoracic surgery and cardiology practice has benefits, several shared barriers, limitations, and challenges need to be overcome. In Table 2, an overview of important differences between VR, 3DCM, and 3D-printing for patient-specific modelling are presented.

Table 2.

A comparison of the features/factors of three different exponential visualization technologies to review 3D models of patient-specific models

	In-depth view	On-site use	Slicing planes	Model fabrication time	Initial investment	Costs per model	Multi- component models	Immersiveness within model	Remote sharing/multi-user	Quantification/measurements	Simulation possibilities
3DCM	−	+	+	+	€€	€	++	−	+/−	+	+
VR	++	+	+	++	€€	€	++	++	+/++	+	+
3D printing	+	+/−	−	−−	€€€a/€b	€a/€€b	−	+/−	−/++	−	+/−

− disadvantage; + advantage; 3D printing, three-dimensional printing; 3DCM, 3D computational model; VR, virtual reality.

^aIn-house facilities for 3D printing.

^bOutsourcing

Currently, costs associated with generation of 3DCM, 3D printing or VR assessment are generally considered high (Table 2). However, as the application of these modalities outside of the medical field is advancing, the number of facilitators increases. For software, several accessible open source (minimal cost) providers are available which could serve as an introductory platform. The current number of 3D-printer and VR-device manufacturers is growing and competition is expected to reduce costs in the forthcoming years.

In the context of 3DCM, several limitations are also worth mentioning: first, as current 3DCMs represent one cardiac phase per model, repeated segmentation is required to understand full cycle 3DCMs (Table 1). This can be laborious as it takes approximately 8–10 min to create a 3DCM of one cardiac phase (depending on available computational power and anatomical complexity). However, when this process is implemented in the clinical workflow and a dedicated multidisciplinary and well-trained team is available, the process can become more efficient and seamlessly fit into daily routine. Second, 3DCMs are static geometries. Incorporation of tissue/device mechanical properties and flow-dynamics (i.e. simulation) could further enhance the prediction of geometrical and hemodynamic effects after device implantation. Although a 3DCM provides valuable insights on 3D spatial interactions, 3D appreciation on a 2D computer-screen can be challenging, but transforming the model in a physical (i.e. 3Dprint), holographic, or immersive VR form could improve perception.

With regard to clinical and surgical simulators, already a large body of evidence is available that has demonstrated the benefits of these technological solutions for training purposes.^12,21,24 However, in the field of cardiothoracic surgery, not many clinically validated or commercial VR-based clinical/surgical simulators are available. A potential barrier/challenge that can be identified in creating reality-like surgical simulators in the field of cardiac surgery, is the lack of suitable materials that can mimic the tissue properties of a human heart and parts of it (such as coronary arteries, valvular structures, great vessels). In addition, for the implementation of simulators in training programmes of residents and junior trainees, there is a need for well-defined protocols and guidelines (for trainers) on how to assess and to keep track of the progress of trainees in these novel platforms. Another interesting challenge is the current lack of advanced VR-based systems that are capable of providing adequate haptic feedback, relevant to surgery.

Time associated with analysis of 3DCMs or VR assessment is dependent on available computational power, anatomical complexity, and operator training. Due to increased automation in the segmentation and analysis process, time to generate 3DCMs has been reduced significantly and expected to further decline. VR assessment of anatomic models commonly does not take longer than reviewing conventional images. Interestingly, in some small case studies, VR has even shown to reduce the time to interpret CT angiography images.²⁵ Our VR platforms allow instant volume rendering of all types of DICOM formats (e.g. echocardiography, CT, magnetic resonance), which can be reviewed immediately in an immersive environment. However, it needs to be taken into account that there is a learning curve for learning to handle the controls of the VR workstation. We are constantly working on improving the user interface and control settings to make the platforms accessible for all clinicians. In addition, an important challenge in structural implementation of immersive VR in clinical practice (relevant to cardiology/cardiac surgery) is the lack of a large number of CE (Conformité Européenne)/FDA (Food and Drug Administration) approved platforms. Moreover, integration of VR workstations within the local network of hospitals can be quite challenging as well, since most hospitals and academic centres do not yet have a well-established infrastructure (and technical support) available to support VR hardware and software platforms. In our centre, we are currently working together with the information technology (IT), IT-security, and radiology departments to create a network-platform in which our PulmoVR/CardioVR systems can be embedded and, after CE certification, can be seamlessly integrated within the clinical workflow.

Expanding the use of 3D applications in clinical practice often occurs simultaneously in different disciplines. Isolated development of a digital program might drive up costs and effort. Interdisciplinary cooperation could reduce these problems. The multidisciplinary heart team discussion, already an essential part of the work-up of an intervention in structural heart disease, could serve as the starting point for the adoption of 3D applications. By integrating 3DCMs, 3D printing, and VR assessment in the standard work-up (echocardiography, coronary angiography, CT) for complex structural disease, both cardiothoracic surgeons and the interventional cardiologists would benefit. Current evidence is limited to case reports, expert option and validation studies.^5,11,12,25 Research describing benefits on procedural outcome are warranted, but require international consensus on definitions of the new modalities. Additionally, cut-offs for important 3D parameters (i.e. asymmetrical calcium distribution, device fitting) need to be established.

Setting up strategic partnerships with industrial parties for outsourcing software/hardware development and production has been key in the advances of our digital platforms. Through the Erasmus medical centre’s technology transfer office, several partnership agreements have been setup to clarify the role and responsibilities of all parties involved. We would highly recommend strategic partnerships since this reduces the workload significantly, and additionally, creates a highly interdisciplinary environment where one can learn from each others experiences and knowledge.

The attitude towards new technology will improve as the practical/clinical benefits become evident and barriers are overcome. In the early days, senior clinicians were reluctant to using these technologies. However, we found that, in challenging structural heart cases, 3DCM/3D printing/VR assessment was increasingly more supported as it provides additional insights not generated by conventional imaging modalities. Currently, these technologies are adopted by both departments of cardiology and cardiothoracic surgery and more and more requests for 3D planning are coming in from other departments as well.

Conclusion

In this point of view article, we have provided an overview of the currently applied exponential technologies to support cardiothoracic surgery and cardiovascular interventions at our centre. In addition, we have described facilitators, barriers, and directions in implementing these technologies at our hospitals and medical centres. Exponential technology tools can enable personalized and patient-tailored interventions. Advanced computation modelling, immersive VR, 3D printing, and digital virtual simulators can facilitate preprocedural planning and training, intraoperative guidance, and education. Close collaboration with dedicated multidisciplinary teams, strategic (industrial) partnerships, and proper technical support (for instance from Radiology departments) and an open attitude towards the implementation of new technology are key components in setting up digital health units within clinical departments.