Exploring the value of three-dimensional printing and virtualization in paediatric healthcare: A multi-case quality improvement study

Abstract

Background

Three-dimensional printing is being utilized in clinical medicine to support activities including surgical planning, education, and medical device fabrication. To better understand the impacts of this technology, a survey was implemented with radiologists, specialist physicians, and surgeons at a tertiary care hospital in Canada, examining multidimensional value and considerations for uptake.

Objectives

To examine how three-dimensional printing can be integrated into the paediatric context and highlight areas of impact and value to the healthcare system using Kirkpatrick's Model. Secondarily, to explore the perspective of clinicians utilizing three-dimensional models and how they make decisions about whether or not to use the technology in patient care.

Methods

A post-case survey. Descriptive statistics are provided for Likert-style questions, and a thematic analysis was conducted to identify common patterns in open-ended responses.

Results

In total, 37 respondents were surveyed across 19 clinical cases, providing their perspectives on model reaction, learning, behaviour, and results. We found surgeons and specialists to consider the models more beneficial than radiologists. Results further showed that the models were more helpful when used to assess the likelihood of success or failure of clinical management strategies, and for intraoperative orientation. We demonstrate that three-dimensional printed models could improve perioperative metrics, including a reduction in operating room time, but with a reciprocal effect on pre-procedural planning time. Clinicians who shared the models with patients and families thought it increased understanding of the disease and surgical procedure, and had no effect on their consultation time.

Conclusions

Three-dimensional printing and virtualization were used in preoperative planning and for communication among the clinical care team, trainees, patients, and families. Three-dimensional models provide multidimensional value to clinical teams, patients, and the health system. Further investigation is warranted to assess value in other clinical areas, across disciplines, and from a health economics and outcomes perspective.

Introduction

At BC Children's Hospital (BCCH) and worldwide, surgical care is an increasingly important part of complex treatments, with over 230 million major operations carried out worldwide every year.¹ Evidence suggests that in Canada 7.5 per 100 hospital admissions have an adverse event, and that 20.8% of these people die as a direct consequence.² Notably, 36.9% of adverse events are preventable and related to surgery.² Advanced visualization techniques, such as three-dimensional (3D) printing and virtualization, are emerging technological approaches that can assist healthcare teams in increasing the quality of care.

3D printing, also known as additive manufacturing, is a technology process that enables the creation of a physical object from a digital file. As an alternative to visualizing anatomy through traditional medical imaging techniques, 3D models enable the translation of two-dimensional images into accurate and comprehensive virtual or physical models. These techniques are currently being used across several clinical settings and for various purposes, including in surgery, education, medical device development, tissue engineering, and research.^3–9

Some review articles have also highlighted the promising value of 3D models for improving patient experience, advancing healthcare outcomes, and improving the provider experience. 3D models have been reported to be accurate, demonstrating anatomy and pathologies with little difference from the true patient anatomy.¹⁰ Surgeons and trainees have reported that the models better prepare them for the realities of operations.¹¹ Patients report a better understanding of their disease process and surgical treatment when explained using 3D printed models of their anatomy.¹² However, evidence remains relatively limited in the paediatric context, and there is a lack of comprehensive studies evaluating the multidimensional effects of this technology. It is critical for decision-makers to have a better understanding of value in order to invest the capital required to develop capacity and infrastructure.

In 2018, a 3D technologies (3DT) programme was established at BCCH. The 3DT programme aims to improve the delivery of healthcare to paediatric populations through digital innovation and technology. In this paper, we conceptualize 3D modelling as a training event using Kirkpatrick's Four Levels of Training Evaluation,¹³ which suggests that proper training provides relevant knowledge, and confidence for participants to apply learning on the job. Learning consists of reaction, learning, behaviour, and results. Reactions are self-reported and measure if the learners have found the learning to be relevant to their role, favourable, and engaging.¹³ Learning is measured typically through post-test evaluations and focuses on the degree to which the learner has acquired the knowledge, skills, attitude, confidence, and commitment based on their participation in the programme.¹³ Behaviours are often recorded by on-the-job observations of performance and seeks to identify if the learners are taking what they learned in training and applying it to their job.¹³ Results are the degree to which target outcomes occur as a result of the training and the support and accountability package.¹³ Investigators suggests that four conditions must be met for behaviour change to occur: the person must have the desire to change, the person must know what to do and how to do it, the person must work in the right climate, and the person must be rewarded for changing.¹⁴ They also suggests that barriers to knowledge and skills on the job include: the lack of the opportunity to use one's learning, the lack of the personal capacity to try out the learning, a belief that the effort exerted will not change performance, a belief that the desirable performance will lead to outcomes the learner values, the extent to which the supervisor or manager actively inhibits the use of the new knowledge and skills, and the support or resistance that peers provide when using new approaches.¹⁴ Using a globally recognized framework to present our findings will increase the transferability of findings across studies to better inform policy and practice decisions.

This study aims to illustrate how the programme was integrated into a paediatric environment and also adds to the literature on what value the technology brings to a paediatric healthcare system. While there are an abundance of studies focusing on case reports or specific application areas,^{10–12,15–43} this study uniquely demonstrates multidimensional areas of potential value from the perspectives of clinicians across different specialties and applications. The results intend to provide insight into the current situation and catalogue identified programme costs and benefits to inform the future development of a cost-benefit framework.

Methods

This project was a quality improvement project conducted at BCCH, Vancouver Canada and is reported using the Revised Standards for Quality Improvement Reporting Excellence (SQUIRE 2.0).

The 3D modelling intervention

Model development process

Physicians can formally request an anatomical model (virtual rendering or physical 3D printed replica) using a standardized requisition form submitted via email. After submitting the requisition form, the requesting physician, the radiologist involved with the image acquisition, and an engineer meet to discuss feasibility, timelines, and create a plan for next steps. An overview of the anatomical modelling and 3D printing workflow is described in Figure 1.

Figure 1.

Anatomical modelling and three-dimensional (3D) printing workflow. We follow a four-phase process: medical imaging, segmentation and processing, model creation, and 3D printing and post-processing. Engaging the clinical care team, including the requesting physician and radiologists, ensures continuous quality checks.

Relevant computer-tomography (CT) datasets of the referred cases are used to create the anatomical models, Figure 2(a). It is customary for the programme to offer two types of physical models for cardiology-related cases [1]: a cast model of the blood pool inside the structures that are readily visible on the CT scan; and [2] a shell model representing the surface that is created by graphically adding a shell of arbitrary thickness on the surface of the aforementioned cast model, using a 3D modelling software such as 3-Matic (Materialise NV, Leuven, Belgium), Figure 2(c). To best mimic the mechanical properties of the different anatomical structures, we used a combination of proprietary print materials; this included TangoPlus and Agilus for creating softer models and Vero materials (white, cyan, yellow, magenta, black) for hard structures, Figure 2(d). Each anatomical model goes through an iterative design process, whereby relevant members of the clinical care team qualitatively assess the model and provide feedback. When errors are encountered, the models are adjusted accordingly and rechecked. The process is similar to those described by other investigators.⁴⁴

Figure 2.

A graphical presentation of the technical anatomical processing processes. The process starts by importing the computer-tomography (CT) dataset.

The results of segmentation and 3D modelling were distributed among the clinical care team in a 3D portable document format which allowed the user to download the model onto their personal computer and independently review and interact with the 3D content using Acrobat Reader (Adobe Inc., California, USA). When requested, a physical replica of the anatomical 3D model was created using a 3D Printer (Objet 260 or J720, Stratasys Ltd, Minnesota, USA).

Study of the intervention

The primary objective of this study was to identify the scenarios for which 3D models would be most helpful within the context of paediatric medicine and surgery. Secondarily, we aimed to evaluate the impact of 3D modelling through its uptake, perceived usability, and any incurred risks when using the technology for planning complex surgeries and interventions. Thirdly, we present the findings with an emphasis on Kirkpatrick's model to explore the effectiveness of using 3D modelling as a pedagogy for approaching complex paediatric surgical cases.

Measures

The cases received through the 3DT programme are rare and heterogenous, so it was not possible to plan, predict, or control the variables required for highly controlled trial designs (eg. randomized clinical trials). We used a guide proposed by Burns et al.⁴⁵ for designing and conducting surveys with clinicians and adapted the Health Belief Model⁴⁶ to explain and predict individual changes in the behaviours of clinicians using the models. In principle, we assume that clinicians would be empowered to use 3D modelling for two main reasons: (a) the desire to improve a patient's health and (b) the belief that the 3D model will provide value. Indicators, considerations, and questions were generated and refined through multidisciplinary expert consultation and a literature review conducted by our team. The survey was piloted with a small sample of staff (N  =  3) at the hospital before implementation to identify unusual, redundant, irrelevant, or poorly worded questions and responses. Conducting factor analysis (eg, reliability, validity) was not feasible.

Administration and ethical considerations

The survey was administered at a large tertiary care hospital located in Vancouver, Canada and covers 19 cases. Ethics approval was not required by our Research Ethics Board. Data were collected and are presented from 2019 to 2021. The sampling strategy was deliberate (nonprobability based), aiming to evaluate the experiences of radiologists, physicians, and surgeons for all (100%) of the 3D models produced during the selected period. Therefore, participants who had already completed a survey on a past model were allowed to engage in more than one survey as long as the models being evaluated were unique. Respondents were consented and invited to participate in the voluntary survey in-person with the programme manager (SZ) as an extension of existing debrief and closing processes.

Analysis

Statistical analysis was performed using the Statistical Package for the Social Sciences Version 22. Data was analyzed according to the full analysis set. Descriptive statistics are reported.

Chi-square test was applied to test the independence of association between categorical variables. ANOVA (for normal distribution) or Kruskal–Wallis (for non-parametric distribution) was applied for one-way analysis to compare outcomes across cases, groups of expertise (physicians, specialists, surgeons), or departments. Post hoc Bonferroni analysis was applied to statistically significant findings to confirm differences between groups. If equal variance assumption was not met during the ANOVA process, pairwise comparisons were based on the statistics of Dunnett's T3.⁴⁷ To test for relationships between two ordinal variables, we used Spearman Correlation.

The results of the qualitative open-ended responses were coded using a rapid analysis framework.⁴⁸

Results

Demographics of 3D printed model users

There were 12 total new recruits to the programme following the programme launch, ultimately requesting a total of 22 anatomical models. One case was transferred to another hospital, and outcomes could not be tracked. We could not engage the clinical team in the survey for two other cases. Therefore, the final analysis includes the surveys evaluating 19 models.

A baseline survey was sent to each new recruit and completed by 12 participants (100% response rate). The follow-up survey was completed by 37 of the possible 59 respondents (63% response rate). Participant characteristics are shown in Table 1 for both surveys.

Table 1.

Characteristics of survey respondents for the 3D technologies programme evaluation.

		Baseline survey (programme intake)	Follow-up survey (post case)
Total respondents		12 (100%)	37 (63%)
Experience, n (%)
	< 10 years	5 (50%)	11 (30%)
	> 20 years	5 (50%)	26 (70%)
Location
	Canada	10 (100%)	37 (100%)
Patient group
	Paediatric	12 (100%)	37 (100%)
Discipline
	Cardiology	6 (50%)	14 (37%)
	Radiology	2 (17%)	13 (35%)
	Cardiac surgery	1 (8%)	6 (16%)
	General paediatric surgery	2 (17%)	3 (8%)
	Otolaryngology	1 (8%)	1 (3%)

The 19 cases ranged in diagnostic severity, complexity, and resources (Table 2). The median amount of material used to create the models was 431 g (interquartile range (IQR) 426 g). The median engineering labour was 13 h (IQR: 9 h).

Table 2.

3D model diagnosis, application, and resources required.

Diagnosis	Team evaluated/Total, (%)	Provider experience level (n)	Total cost	Material (g)	Material type	Engineering labour (h)
Coarctation of the aorta	2/2 (100)	<10 years (1) > 20 years (1)	$ 122	316	Tango + , SUP705	1
Heterotaxy  +  unbalanced AVSD + total anomalous pulmonary venus return	2/2 (100)	> 20 years (2)	$ 590	137	Tango + , SUP705	9
Tetralogy of Fallot  +  pulmonary atresia	2/2 (100)	<10 years (1) > 20 years (1)	$ 1408	485	Tango + , SUP705	21
Tetralogy of Fallot  +  RVOT obstruction	1/3 (33)	> 20 years (1)	$ 1270	878	Tango + , SUP705	17
Heterotaxy	1/1 (100)	> 20 years (1)	$ 1050	-	-	18
Tetralogy of Fallot	2/2 (100)	<10 years (1) > 20 years (1)	$ 1308	725	VeroWhite, Tango + , SUP705	18.5
CCTGA + DORV + VSD	3/3 (100)	> 20 years (3)	$ 335	345	VeroWhite, Tango + , SUP705	4
CCTGA + VSD	4/4 (100)	<10 years (1) > 20 years (3)	$ 473	416	VeroWhite, Tango + , SUP705	8
Complex systematic venus anatomy	2/2 (100)	> 20 years (2)	$ 1500	182	Tango + , SUP705	24
Right bronchial atresia (absent right lung) + horseshoe left lung  +  esophageal atresia  +  distal fistula	2/2 (100)	<10 years (1) > 20 years (1)	$ 300	-	-	5
Coarctation of the aorta  +  ventricular septal defect  +  tricuspid valve dysplasia	2/2 (100)	> 20 years (2)	$ 1094	1153	Tango + , SUP705	12.5
Tetralogy of Fallot variant  +  MAPCA	3/3 (100)	> 20 years (3)	$ 810	-	-	13.5
Tetralogy of Fallot variant  +  2 VSD	2/2 (100)	> 20 years (2)	$ 943	446	Tango + , SUP705	13.5
Double outlet right ventricle  +  large inlet VSD	3/3 (100)	> 20 years (3)	$ 905	410	Tango + , SUP705	13
Transposition of the great arteries  +  pulmonary atresia  +  inlet VSD	1/1 (100)	> 20 years (1)	$ 898	173	Tango + , SUP705	14
Congenital short trachea	1/1 (100)	<10 years (1)	$ 420	-	-	7
Profound bilateral hearing loss	1/1 (100)	<10 years (1)	$ 567	409	Agilus, VeroWhite, SUP706	7.5
Large muscular VSD	2/2 (100)	<10 years (1) > 20 years (1)	$ 601	466	Agilus, VeroWhite, SUP706	9
Large epithelioid sarcoma of perineal/ischiorectal region	1/1 (100)	<10 years (1)	$ 1410	1741	Agilus, VeroWhite, VeroCyan, VeroYellow, VeroBlack, VeroMagenta, SUP706	18.5

AVSD: atrioventricular septal defect; RVOT: right ventricle out flow tract; CCTGA: congenitally corrected transposition of the great arteries; DORV: double outlet right ventricle; VSD: ventricular septal defect; MAPCA: major aortopulmonary collateral arteries

Kirkpatrick level 1: Reaction

Reactions of the radiologists, surgeons, and specialists were measured through self-reported questions on satisfaction and relevancy. The models received positive reactions from the survey respondents. In total, 94% of respondents recommended 3D printing to their colleagues, 93% thought the 3D model answered the clinical questions they hoped it would, and 91% thought the model provided at least some value to the clinical case (Table 3).

The models were also thought to be relevant, as 92% of the models were accurate compared to the preoperative medical images and 83% compared to the intraoperative findings. In terms of the inaccurate models, in one congenital heart disease case, a ventricular septal defect extended from inlet to the peri-membranous region, but the model only demonstrated the peri-membranous portion of the defect. For the other model (ear model for simulating cochlear implantation), the surgeon indicated that the tactile drilling experience during simulation was different than the intraoperative experience, due to differences in the composition and density of materials in the model. In this specific case, tactile feedback during drilling was critical for identifying anatomical landmarks.

Kirkpatrick level 2: Learning

The extent of learning by radiologists, surgeons and specialists was measured though self-reported questions on knowledge, attitude and confidence. The models provided additional clarity to some aspects of severe and complex conditions when used as an adjunct to other imaging modalities. In total, 67% of the respondents thought the model improved their own understanding of the patient's condition and the anatomical structure (Table 3). There were significant differences found between the staffs' expertise and how helpful the models were perceived to be [F(2,36)  =  11·846, p  =  0.003]. Physicians and surgeons reported the models to be more helpful than radiologists and may benefit most from the programme (p_bonferroni < .05). The respondents indicated that:

We chose a different stent based on the simulation procedure… [when we] put one stent in, [we found] a different type and size would be better suited to the anatomy.

It helped me decide this was a surgical patient and not an interventional case.

The models also helped to anticipate procedural difficulties (68%) and improved confidence (84%; Table 3). Respondents that used a model as a visual aid during intraoperative orientation or to assess the likelihood of success or failure of the surgical strategy generally reported more confidence with the overall diagnosis and clinical care plan (r_s(21)  =  0.434 p  =  0.05, r_s(21)  =  0.470 p  =  0.03, respectively).

Table 3.

Responses in Kirkpatrick levels. Responses to questions according to the four levels for Kirkpatrick's Model of Evaluation.

		Count, n/N	Frequency, %
Kirkpatrick level 1: Reaction
Would you recommend three-dimensional (3D) printing to your colleagues?
	Yes	30/32	94%
	No	2/32	6%
Did this 3D model answer the clinical question(s) you were hoping it to?
	Yes	25/27	93%
	No	2/27	7%
In your opinion, how much value did the 3D model add to this clinical case?
	Substantial value	19/32	59%
	Some value	10/32	31%
	Little value	1/32	3%
	No value	2/32	6%
Does this 3D-printed model accurately display the anatomy as portrayed by the preoperative medical images?
	Yes	11/12	92%
	No	1/12	8%
Did you notice any difference(s) between this 3D printed model and intraoperative findings?
	Yes	2/12	17%
	No	10/12	83%
Kirkpatrick Level 2: Learning
Do you think your own understanding of the patient's condition and the anatomical structure was enhanced by this 3D
	Yes	12/36	33%
	No	24/36	67%
The model helped me to anticipate procedural difficulties
	Agree	13/19	68%
	Neutral	4/19	21%
	Disagree	2/19	11%
The model improved my confidence with the overall diagnosis and clinical care plan
	Agree	16/19	84%
	Neutral	2/19	11%
	Disagree	1/19	5%
The model offered an opportunity to explore a different treatment or a treatment that would not have otherwise been pursued.
	Agree	4/19	21%
	Neutral	2/19	11%
	Disagree	12/19	63%
Kirkpatrick Level 3: Behaviour
Please indicate who you shared this 3D model with:
	Patients and families	10/37	27%
	The clinical care team	33/37	89%
	Trainees	23/37	62%
	Other	4/37	11%
Did you use this 3D model as part of preoperative planning?
	Yes	21/37	57%
	No	16/37	43%
Please indicate how you used the model for planning
	For surgical or interventional simulation	12/21	57%
	As a visual aid for intraoperative orientation	8/21	38%
	For assessing the likelihood of success or failure of the surgical strategy	16/21	76%
	Other	9/21	43%
Kirkpatrick Level 4: Results
Patient and family's understanding of the disease and surgical procedure
	Increased	8/10	80%
	No effect	2/10	20%
	Decreased	-
Consultation time with the patient and family
	Increased	3/10	30%
	No effect	6/10	60%
	Decreased	1/10	10%
Clinical care team's understanding of the pathology and clinical care plan
	Increased	30/32	94%
	No effect	2/32	6%
	Decreased	-
Length of time to define the individual's surgical or interventional plan
	Increased	6/21	29%
	No effect	13/21	61%
	Decreased	2/21	10%
From your perspective, how do you think this 3D model impacted fluoroscopy/radiation exposure?
	Increased	-
	No effect	9/12	75%
	Decreased	3/12	25%
From your perspective, how do you think this 3D model impacted operational time?
	Increased	1/12	8%
	No effect	5/12	42%
	Decreased	6/12	50%
From your perspective, how do you think this 3D model impacted blood loss?
Increased		-
No effect		11/12	92%
Decreased		1/12	8%
From your perspective, how do you think this 3D model impacted anaesthetic and other medication usage?
	Increased	-
	No effect	8/12	67%
	Decreased	4/12	33%
From your perspective, how do you think this 3D model impacted number of major complications?
	Increased	-
	No effect	8/12	67%
	Decreased	4/12	33%
From your perspective, how do you think this 3D model impacted human resources used (ie, operative assistance and other support staff)?
	Increased	-
	No effect	11/12	91%
	Decreased	1/12	9%
From your perspective, how do you think this 3D model impacted time spent on preoperative planning?
	Increased	17/21	81%
	No effect	2/21	9%
	Decreased	2/21	9%

There was also some indication that skills were improved by using the models. For 21% of respondents, the model offered an opportunity to explore a different treatment or a treatment that would not have otherwise been pursued (Table 3). Respondents that used the models to assess the likelihood of success or failure of the surgical strategy reported being better able to anticipate procedural difficulties (r_s(21)  =  0·573 p  =  0·01) and having opportunities to explore different treatments or treatments that would not have otherwise been pursued (r_s(21)  =  0·463 p  =  0.03).

Kirkpatrick level 3: Behaviour

To provide multidimensional value, models must be shared with various users to make the greatest impact. All (100%) of the models were shared with either the clinical care team (89%), the patients and families (27%), trainees (69%), or other groups (11%; Table 3). Other groups included future intentions to share the models with trainees, patients, and families; and to use the models for research or publication. We found significant differences between who was sharing the models with patients and families [F(2,37)  =  9·298, p  =  0.01], and trainees [F(2,37)  =  8·287, p  =  0.02]. Physicians were more frequently sharing the models with patients and families than radiologists (p_bonferroni  =  0.005]. We also found significant differences between physicians and surgeons sharing the models with trainees (p_bonferroni  =  0.01).

To drive uptake, 3D models must be used within a process or system that reinforces value. Models were often used for preoperative planning (57%; Table 3). During preoperative planning, the models were used for assessing the likelihood of success or failure of the surgical strategy (76%), for surgical or interventional simulation (57%), as a visual aid for intraoperative orientation (38%), and other purposes such as optimizing stent placement, to determine if surgery was required, or to localize various anatomical structures (43%). We found that the models were thought to be most helpful when they were used as a visual aid for intraoperative orientation (r_s(36)  =  0.378, p  =  0.02) and to assess the likelihood of success or failure of the surgical strategy (r_s(10)  =  0.514, p  =  0.001).

Kirkpatrick level 4: Results

There was a belief from the respondents that the models changed the performance and perioperative outcomes for themselves, their clinical teams, patients and families, trainees, and other user groups.

In total, 94% thought the models shared with the clinical care team improved understanding of the pathology and clinical care plan. There was generally thought to be no effect (61%) on the length of time to define the individual's surgical or interventional plan. Respondents commented:

We knew what we wanted [to achieve], but we used the model to convince [the rest of the team]

…I think it enhanced the discussion, enhanced communication. Being able to show all the care team members the relationship of the blood vessels to airway [helped].

Surgeons, compared to physicians in the same case, indicated an increased time to define the plan because the model was used to determine the feasibility of the proposed management approach.

When models were shared with trainees, the length of time to define the individual's surgical or interventional plan was shortened (r_s(23)  = −0·493, p  =  0.02). Models shared with trainees were often also used for surgical or interventional simulation (r_s(37)  =  0.421, p  =  0.01).

Models shared with patients and families increased understanding of the disease and surgical procedure (80%) and the respondents thought that the model had no effect on consultation time (60%).

During the operation, models in our study made the most impact on decreasing operational time (50%), and increasing time spent on preoperative planning (81%; Table 3). One respondent said:

It increased the planning time but that saved us in the operating room.

Other measures, such as fluoroscopy/radiation exposure, blood loss, anaesthetic and other medication usage, number of major complications, and human resources were more commonly reported to not be affected by the models (Table 3).

Discussion

Summary

This study illustrates how 3D printing can be integrated into paediatric environments for trainingand adds to the literature on potential areas of impact within the healthcare system. We explored the perspective of clinicians adopting 3D printing and how they make decisions about whether or not to use models in patient care. The respondents in our study had positive reactions, learning, behaviour and results on Kirkpatrick's levels of evaluation when using the 3D models in the management of complex paediatric surgical cases. While there are many studies focusing on case reports or specific application areas,^{10–12,15–43} this study uniquely demonstrates multi-dimensional areas of potential clinical value from the perspectives of clinicians across different specialties and applications.

Interpretation

Several of the models in our study were used as visual aids for intraoperative orientation by surgeons, and the models were perceived as more helpful when used like this. Reviews have also reported models to be effective when used for intra-operative orientation.^49,50 Since the models were largely perceived as accurate and true-to-size, they eliminated the need to mentally reconstruct the three-dimensional relationships in the operating room. Other investigators have found that this allows the team to avoid improvisation, save intraoperative time, and improve outcomes.^43,50,51 A majority of the research evaluating the intraoperative use of 3D printed models focuses on congenital heart diseases, orthopaedics and craniomaxillofacial surgery,⁴ and our research shows that other disciplines such as otolaryngology and oncology could also benefit. For example, Lau et al.⁵² used a 3D patient-specific model to provide realistic visualization of brain anatomical structures and tumour which enhanced their understanding of pathology in relation to the surrounding structures during tumour resection. Our research shows that exploring the impacts of intraoperative use in these disciplines, as well as other untouched areas, and on health systems as a whole, would be a beneficial target for further research studies.

Our study found that 3D models must be shared with various users to make the greatest impacts, and they must be used within a process or system that reinforces value. We found surgeons and physicians considered the 3D printed models to provide more benefit than radiologists. An explanation for this could be the limited involvement of radiologists in the clinical applications observed,¹⁴ primarily partnering with the engineer in defining the factors that affect accuracy and reproducibility.⁵³ Alternatively, due to the nature of their work, it is possible that radiologists have developed enhanced spatial analysis skills that lessen the incremental benefits of the 3D model compared to clinical peers. Physicians and surgeons were able to utilize the models in a variety of ways, choosing to share the models with their teams, trainees, patients, and other users. Our study shows that the same model can be deployed in multiple contexts, having multiple levels of influence on the outcomes of a patient's healthcare.

For example, in addition to using the model for team communication, clinicians who chose to share the models with patients and families in our study thought the model generally increased their patient's understanding of the disease and surgical procedure, and it had no effect on their consultation time. Other investigators have also found 3D models to improve communication quality and efficiency with patients, especially those with relatively poor anatomical knowledge^42,54–58 A review by Traynor et al.⁵⁹ found that limited studies looked at using 3D models during consultations and no studies looked at how 3D models could be used in communication between patients and families and their friends. Studies using patient-specific 3D printed models in paediatric consultations typically focus on the parent perspective only.^60,61 Alshomer et al.⁶⁰ found that when using patient-specific 3D printed models in craniosynostosis consultations, parents did not feel the need to read any more about the condition and had more agreement for surgical management. Entezami et al.⁶¹ used 3D modelling in paediatric plagiocephaly consultations with parents and reported improved understanding from traditional counselling. Some authors have noted that patients may experience an emotional response as a negative effect of confronting a physical model of their disease process.⁶² Adverse emotional responses were not mentioned in our study results, however, this could be a bias in our processes as it was up to the physician to choose if, when, and how to share the models. While our survey responses provide some subjective and indirect indication of the patient and family experience, further research should be conducted on the benefits and barriers to using 3D models during consent and consultation from the patient and family perspective, especially in contexts where health literacy is low.

Our results further showed that the models were more helpful when the team could use them to assess the likelihood of success or failure of management strategies. Reviews have also reported models to be effective when used for surgical decision-making.^49,50 Notably, the models in our study did not always need to change the surgical decision to be considered helpful. In some cases, the models helped to define or confirm a hypothesis about the management approach or improve confidence with the overall diagnosis and clinical care plan. Paediatric surgeries tend to be complicated by the range of sizes of each patient, and a review by Meyer-Szary et al.⁶³ suggested that 3D printed models might be most beneficial in paediatric surgery when used to confirm a hypothesis about the medical equipment that can ideally serve certain individuals, as well as custom print medical equipment (e.g., stents) when patients fall between the specifications of readily available tools. While our results provide promising evidence in support of using 3D modelling in surgical decision-making, our findings are subjective and should be further validated using objective investigations.

Overall, our results agree with the high dimensional accuracy of 3D models previously reported.⁵³ Accuracy is a complex issue because no single variable will be representative of the comprehensive accuracy of medical models for all indications.⁵³ In two of the cases described in our study, the models we created were reported to be inaccurate compared to the intraoperative findings. In one congenital heart disease case, a ventricular septal defect extended from the inlet to the peri-membranous region, but the model only demonstrated the peri-membranous portion of the defect. The accuracy of the printed model did not appear to be a limitation in the setting of limited blood flow shunting based on equal resistance but could be a result of the acquisition technique or image resolution described by other investigators.⁵³ Accuracy is vital for clinical applications, especially in preoperative planning and treatment, because an inaccurate representation of patient anatomy and pathology can impact the success of the management plan.⁶⁴ However, in our study, the inaccurate model was reported to still be useful in combination with the echocardiogram (ECHO) findings, and provided added benefit to visualizing the great vessel relationships. This provides evidence to support that 3D models add to the information that can be obtained from other imaging modalities (eg, ECHO, magnetic resonance imaging (MRI), CT) in complex cases, but should only be used to aid in decision-making and enhance a clinician's judgement. It also supports the notion by George et al.⁵³ that workflow modifications, for example, changes in imaging protocols or segmentation, should be frequently reviewed when modelling new dimensions and morphologies.

The model used for profound bilateral hearing loss was also perceived to be inaccurate compared to the intraoperative findings within our study. In this case, the surgeon indicated that the tactile drilling experience during simulation was different from the intraoperative experience. This model used multiple print materials to recreate the harder structure of bone and the softer cartilaginous tissues of the ear, however, the model still did not adequately mimic the mechanical properties of human tissue. The challenges of recreating anatomical fidelity using print materials for surgical simulation have been documented by other investigators.^65–67 Similar to Rose et al.⁶⁸ who studied the use of 3D printed models for paediatric mastoid surgery, the support material necessary for the 3D printing process contributed to inhibiting the realism of the simulation. Our results indicate the importance of selecting appropriate materials when models are used for simulation in clinical contexts, especially where tactile feedback is a critical component. Further research is needed into potential solutions for these types of design challenges.

Additionally, the results of our investigation demonstrate that 3D-printed models are thought to improve perioperative metrics. Notably, we found that while models increased planning time, they may reduce operational time. Other investigators have found that time and cost savings may be related to the way models are used by individuals.⁶⁹ For example, Rogers-Vizena et al.⁶⁹ suggested that models used for visualizing and studying the spatial relationships of anatomic structures provided the most value. 3D printed models may save the healthcare system up to $ 3720 USD per case.^69,70 Further economic studies evaluating the costs and effects of these models may be of interest to healthcare systems for establishing fee guides.

Limitations

This study describes a survey on 3D printing through a set of organized questions about opinions, perceptions, and behaviours at one site. The reliability of the data is very dependent on the structure of the survey, which was developed via a thorough literature review and in consultation with experts. However, we did not statistically test for reliability and validity. Furthermore, the responses were subjective for a number of questions on impact. It is worthwhile to note that the results may not necessarily reflect objective measures.

The Kirkpatrick model describes 4 levels of evaluation, and our evaluation was largely subjective. We realize there are more rigorous analysis methods beyond our analysis of self-reported responses by which to gauge the quality of 3D printed models. This paper does not assess whether the resources expended on the model construction have acceptable multi-dimensional benefits. To this point, our team has the future aim to use the impacts found within this study to evaluate the cost-benefit of 3D printing.

This study is also subject to several biases. The study is subject to information bias as we recruited participants through the 3DT programme, but not everyone participated. Additionally, our study is subject to response bias because respondents answered questions with an engineering manager in a leadership role with the programme. The respondents may have answered in a way that supports our questions, rather than objectively.

Conclusion

We demonstrate the reaction, learning, behavioural, and result benefits of 3D modelling in paediatric preoperative planning and communication with the clinical care team, trainees and patients and families. The findings warrant further investigation into the value of 3D printing across disciplines, and from the health economics and outcomes perspective.

Appendix 1: Baseline intake survey

https://www.dropbox.com/s/nzvz8zoq6cw95 mt/Baseline%20Intake%20Survey.pdf?dl = 0

Appendix 2: Clinical impact survey

https://www.dropbox.com/s/gm5o1lvod9lv0c0/Clinical%20Impact%20Survey%20Followup%20Case%20Reports.pdf?dl = 0