Optically tracked and 3D printed haptic phantom hand for surgical training system

Abstract

Background

For surgical fixation of bone fractures of the human hand, so-called Kirschner-wires (K-wires) are drilled through bone fragments. Due to the minimally invasive drilling procedures without a view of risk structures like vessels and nerves, a thorough training of young surgeons is necessary. For the development of a virtual reality (VR) based training system, a three-dimensional (3D) printed phantom hand is required. To ensure an intuitive operation, this phantom hand has to be realistic in both, its position relative to the driller as well as in its haptic features. The softest 3D printing material available on the market, however, is too hard to imitate human soft tissue. Therefore, a support-material (SUP) filled metamaterial is used to soften the raw material. Realistic haptic features are important to palpate protrusions of the bone to determine the drilling starting point and angle. An optical real-time tracking is used to transfer position and rotation to the training system.

Methods

A metamaterial already developed in previous work is further improved by use of a new unit cell. Thus, the amount of SUP within the volume can be increased and the tissue is softened further. In addition, the human anatomy is transferred to the entire hand model. A subcutaneous fat layer and penetration of air through pores into the volume simulate shiftability of skin layers. For optical tracking, a rotationally symmetrical marker attached to the phantom hand with corresponding reference marker is developed. In order to ensure trouble-free position transmission, various types of marker point applications are tested.

Results

Several cuboid and forearm sample prints lead to a final 30 centimeter long hand model. The whole haptic phantom could be printed faultless within about 17 hours. The metamaterial consisting of the new unit cell results in an increased SUP share of 4.32%. Validated by an expert surgeon study, this allows in combination with a displacement of the uppermost skin layer a good palpability of the bones. Tracking of the hand marker in dodecahedron design works trouble-free in conjunction with a reference marker attached to the worktop of the training system.

Conclusions

In this work, an optically tracked and haptically correct phantom hand was developed using dual-material 3D printing, which can be easily integrated into a surgical training system.

Introduction

To stabilize and fix broken bones in the hand, within a minimally invasive surgery so-called Kirschner-wires (K-wires) are drilled into bone fragments. K-wires are long, sharp, stainless steel pins, which can be seen in Figure 1A (1-4). Since no bone bore dust is transported to the outside as with a twist drill, the entire bore dust remains in the bore hole. Thus, K-wires get stuck in the bore hole due to the considerable friction between the bone bore dust and the wire surface. This is sufficient to hold bone fragments strongly together (3). Positioning of these K-wires must be carried out carefully and accurately, since surgeons must not hurt any underlying risk structures such as nerves, tendons or vessels (1,2). Within the scope of minimally invasive surgery, these risk structures cannot be spotted through the skin. Thus, to determine the ideal drilling angle, surgeons relay only on their practical and theoretical knowledge of the human anatomy to locate risk structures by feeling bony protrusions through the skin. Currently, surgical training is restricted to drill on synthetical or defrosted cadaver bones, which may lead to limited reality (5). Additional “hands-on” training in the operating room with surgical guidance for young surgeons is very time consuming for an expert surgeon, can endanger the patient condition (6,7) and arises ethical problems. The ethical concerns relate to the use of patients as medical simulators, as, according to Gates [1997], the central ethical practice of medicine is to avoid harm to patients (5,8). Thus with HaptiVisT, a virtual reality (VR) based training system (Figure 1B) is being developed to teach especially young surgeons the interplay between accurate K-wire drilling and protection of risk structures (9,10). A haptic device (Virtuose 6D Desktop, Haption GmbH, Aachen, Germany, www.haption.com) for drilling with force feedback and an autostereoscopic three-dimensional (3D) monitor to visualize the drilling process (SeeFront GmbH, Hamburg, Germany, www.seefront.com) is used. As an additional essential haptic component, an optically tracked haptic phantom hand in order to feel bony protrusions for localizing risk structures is required to complete the overall system. To develop a haptically correct phantom hand, dual-material 3D printing for rapid prototyping was chosen.

Figure 1.

HaptiVisT training system and metamaterial from previous work. (A) Long, sharp and stainless steel Kirschner-wire (K-wire) for minimally invasive bone drilling. (B) HaptiVisT training system with real-time tracking camera (b1), force feedback device with drill handpiece (b2), touchscreen user interface (b3), autostereoscopic 3D monitor for drilling visualization (b4), haptic phantom hand with dodecahedron marker (b5) and reference marker (b6). (C,D) Wireframe of a unit cell and corresponding cuboid lattice structure for (C) cube edge (CE) and (D) cube face (CF) design.

Soft tissue imitation and previous work

Currently, in 3D printing the most elastic rubber like material on the market is TangoPlus (TP, FLX930) Stratasys Ltd. (Minnesota, USA, www.stratasys.com) (11,12). The hardness of 3D print materials is measured in a scale named shore hardness, which is the measured hardness of an elastomer (13). TP or TBP with shore hardness 26–28 scale A are still too firm to mimic soft tissue according to its material properties (14). In comparison, a bovine aorta achieves a shore 00 hardness of 41 (15). This corresponds to a value less than five on the shore A scale (16). Thus, in previous work a support-material-filled (support-material, SUP) metamaterial to soften raw TP and imitate human soft tissue with its haptic and elastic properties was introduced (17). Metamaterials in 3D printing allow to change the haptic features assembling small-scaled structural components (18-20). Maier et al. [2019] used a basic wireframe, which consists of a repeated arrangement of cubic unit cells with iterative cube edge (CE) or cube face (CF) design to create a stabilizing lattice structure. The size of a unit cell is determined by its rule size, which is the side length of a single unit cell. CE lattice consists of thickened wireframe struts (Figure 1C) and CF of additionally closed base areas (Figure 1D). This creates a thickened lattice structure, which can be printed with TP material inside the volume of a surface body. The space between the lattice is automatically filled with blowy SUP to stabilize the model during 3D printing. The SUP is not removed from the volume after the printing process and crumbles under pressure. By reducing the amount of TP in the inner volume through SUP, an object becomes significantly softer. This makes it possible to soften the raw TP material to reproduce a realistic replica of human hand soft tissue (17). The CF design convinced with its elasticity and softened the TP material. However, an expert evaluation found shiftability of the skin and bone palpability poorly implemented, leading to the conclusion, that soft tissue is still too firm. In addition, until now only finger samples were printed and not a whole phantom hand for a training system.

In this work, the CF metamaterial is improved to manufacture softer 3D print volumes and a whole phantom hand. Additionally, a sliding skin component is introduced by transferring the real hand anatomy. Finally, a 3D printed marker is connected to the phantom hand to enable optical real-time tracking. Our novel phantom hand design with phantom tracking is evaluated by two separated expert surgeon case studies. Our purpose: the development of a 3D printed phantom hand for a VR based training system, which is haptically correct, has a relocatable skin and gives young surgeons in a training environment the possibility to locate risk structures sensing bone protrusions haptically. Our hypothesis and main goal: the relocatability of the skin and a new softer metamaterial improve the bone palpability and the tissue softness for a realistic phantom hand. The paper is organized as follows: (I) in the “Methods” section a new metamaterial design is introduced, describing the phantom hand assembly by imitation of the human anatomy. Additionally, the phantom hand real-time tracking and the expert surgeon study designs are explained. (II) In the “Results” section the findings obtained from several 3D test prints, the fabrication of whole phantom hands and the outcomes obtained from expert surgeon studies regarding phantom hand haptic and tracking quality are presented. (III) Finally, in the section “Discussion” conclusions and future works are stated.

All parameters regarding 3D sample prints are compactly listed in Table 1 and important technical terms or abbreviations in Table 2.

Table 1. List of all relevant 3D printed samples.

ID	R	RS(CEFO) (mm)	tcutis (mm)	tfat (mm)	tboundary (mm)	tlattice (mm)	tperiost (mm)	dpores (mm)	SUP (%)	Study
A01	CF	3.5	1.5	–	–	0.5	1.0	–	35.19	No
A02	CEF	3.5 (0.50)	1.5	–	–	0.5	1.0	–	39.51	No
A03	CEF	3.5 (0.75)	1.5	–	–	0.5	1.0	–	45.96	No
A04	CEF	3.5 (0.25)	1.5	–	–	0.5	1.0	–	36.20	No
C09	–	–	2.0	2.0	–	–	–	–	27.78	No
C14	CF	3.5	2.5	–	–	0.5	–	–	37.08	No
C18	CEF	3.5 (0.50)	2.5	2.0	0.5	0.5	–	1.00	53.10	No
C19	CEF	3.5 (0.50)	2.5	2.0	0.5	0.5	–	0.25	51.75	No
C20	CEF	3.5 (0.50)	2.5	2.0	0.5	0.5	–	0.50	51.35	No
H01	CF	1.5	2.0	–	–	0.5	1.0	–	49.81	Yes
H02	CEF	3.5 (0.50)	1.0–2.0	0.5–2.0	0.5	0.5	1.0	0.50	65.35	Yes

Every sample is determined by its geometry ID (forearm A, cuboid C, hand H), unit cell rule R (CF, CEF), cubic rule size RS [side surface opening of CEF (CEF_o) proportional to the rule size in brackets], thickness of cutis t_cutis, subcutaneous fat t_fat as well as subcutaneous boundary t_boundary, lattice thickening t_lattice, pore diameter d_pores and SUP ratio. The table also shows, whether or not the sample is included in the phantom hand expert study design. Missing sample print ID’s were lack prints. CEF, cube edge face; CF, cube face; SUP, support-material.

Table 2. Listing of all relevant technical terms with accompanying description.

Technical term	Description
SUP	Blowy support-material for supporting overhanging features of complex 3D printed objects
TP	TangoPlus: most elastic rubber like 3D print material
VBP	VeroBlackPlus: stiffest 3D print material
CE	Cube edge metamaterial with thickened wireframe struts
CF	Cube face metamaterial with thickened wireframe struts and base areas
CEF	Cube edge face metamaterial as a combination of CE and CF design
Cutis	Outer skin layer of a 3D printed object consisting of TP
Subcutaneous fat	SUP layer under the cutis representing fatty tissue for skin shiftability
Subcutaneous boundary	TP layer holding the subcutaneous fat between cutis and subcutaneous boundary
Soft tissue	Metamaterial for imitating human soft tissue with SUP filled lattice structures
Periosteum	TP layer surrounding the bones
XPoint	Chessboard like tracking point

VBP, VeroBlackPlus; TP, TangoPlus; CEF, cube edge face; CF, cube face; SUP, support-material.

Methods

CEF metamaterial design

In order to make a CF lattice structure even softer, the amount of SUP must be further increased. For this purpose, a new metamaterial unit cell was developed, which is a combination of CE and CF. The closed side surfaces of CF are opened by a negative applied CE structure (Figure 2A). However, the side surfaces of this cube edge face design (CEF) are only opened to such an extent that the amount of SUP is increased, but still guarantees a local limitation of the SUP. Compared to the already published lattice generation workflow in (17), nothing changes except for the unit cell design. The change in unit cell design from CF to CEF is illustrated by the unit cells in Figures 1D,2A for CF and CEF, respectively. The size of the side surface opening (CEF opening: CEFo) is given in relation to the rule size (RS) with CEFo = x · RS, where x ∈ {0.25;0.50;0.75}.

Figure 2.

Cube edge face (CEF) metamaterial design. (A) Cube edge face (CEF) unit cell with constant side length. Compared to the cube face (CF) unit cell, the side surfaces are opened by a negative CF structure. Thus, the amount of SUP can be increased. (B) Model of a forearm section consisting of surface models for the skin, CEF lattice structure, periosteum and bone. (C,D) CEF lattice prints of forearm sections with SUP partially (C) and completely kneaded from the outside (D). SUP, support-material.

Three forearm cuts (forearm sample prints ID: A) are printed in CEF design (A02-A04, see Table 1) with rule size RS =3.5 mm to test the metamaterial within large volumes. Each sample has an outer skin thickness (cutis, see Section Human anatomy transmission or Table 2) of t_cutis =2.0 mm and a CEF lattice structure thickness of t_lattice =0.5 mm in TP material. Bone parts are printed with the stiffest material VeroBlackPlus (VBP, RGD875, shore hardness 83–86 scale D) from Stratasys and have a surrounding TP bone layer (periosteum, see Section Human anatomy transmission or Table 2) of thickness t_periost =1 mm (14). A02-A04 differ in their side surface openings (A02: CEFo =0.50·RS, A03: CEFo =0.75·RS, A04: CEFo =0.25·RS). For comparison, a forearm cut in CF design (A01) is printed with the same parameters as the other sample prints. Figure 2B presents the CEF model of A03. After kneading a 3D printed forearm cut, the SUP within the lattice structure crumbles. Figure 2C shows the partial kneaded A04 and Figure 2D the full kneaded A02.

Human anatomy transmission

The skin anatomy (Figure 3) is primarily divided into epidermis, dermis and subcutaneous tissue (22-24). Epidermis and dermis are usually combined as cutis. The subcutaneous tissue consists mainly of fatty tissue, which makes it possible for skin layers to be shifted against each other like a sliding film (24). The thickness of the skin layers changes depending on the part of the body (25,26). According to a sonographic thickness measurement, the average skin thickness on phalanges bones is t_skin =0.811 mm for women and t_skin =1.080 mm for men. On the forearm the average skin thickness is t_skin =1.041 mm for women and t_skin =1.300 mm for men. The maximum value on the forearm for men is t_skin =1.700 mm (25). The soft tissue of the hand is located between skin layers and bones. Soft tissue consists of fat, muscles, tendons and connective tissue including the fasciae put through arteries, veins and nerves. Connective tissue holds the cells in place and is responsible for the elasticity (24). Bones are located underneath the soft tissue and are surrounded by a thin periosteum. Epidermis and dermis are pierced with skin pores. Skin pores are openings of hair follicles, oil and sweat glands (27), which are located unevenly over the skin.

Figure 3.

Human anatomy transmission. (A,B) Transfer of the human anatomy to the 3D printing model (21). The individual layers of the skin (A) are sequentially transferred to the hand model (B): skin pores, cutis, subcutaneous tissue, soft tissue, periosteum and bone. In the model, the subcutaneous tissue consists of subcutaneous fat and a rubber-like subcutaneous boundary. (C) shows the cross-section of the entire hand model. Adapter plate and bone made of VBP, cutis and subcutaneous boundary of TP, metamaterial of a combination of TP and SUP and the subcutaneous fat of SUP. VBP, VeroBlackPlus; TP, TangoPlus; SUP, support-material.

The human tissue anatomy is transferred to the haptic phantom hand model as follows: In order to achieve a certain robustness and thickness, epidermis and dermis are combined to form the outer skin (cutis), which is printed with pure TP material. Subcutaneous tissue is imitated with a layer of pure blowy SUP—like fatty tissue—and a limiting subcutaneous layer to hold SUP of the subcutaneous fat in place. In the following: the skin of the hand is termed cutis, the subcutaneous tissue is divided into subcutaneous fat and subcutaneous boundary, the tissue between subcutaneous boundary and periosteum is referred as soft tissue. See Figure 3B to get an impression of the 3D print phantom hand layers. If SUP crumbles due to external pressure applied to the cutis, it forms a sliding film. Thus, the cutis can slide past the subcutaneous boundary over crumbled SUP of the subcutaneous fat. Depending on the metamaterial, the soft tissue consists of a CF or CEF structure. The lattice structure of the metamaterial and the periosteum consist of pure TP. All spaces between the lattice struts are automatically filled with SUP during 3D printing. As the lattice structure extents into the subcutaneous boundary and periosteum, it is firmly attached to them. Bones are printed with hard VBP material. Since the entire hand model is filled with material, there would be a vacuum in the volume preventing the model to be kneaded deeply. Therefore, cylindrically shaped holes (skin pores) protrude through cutis, subcutaneous fat and the subcutaneous boundary into each outer unit cell of the lattice. After kneading, air flows into the subcutaneous fat and into the lattice structure dissolving the vacuum in the volume and also allowing layers to be shifted easily.

Several samples are printed for subcutaneous fat and pores. The 3D print of C09 (cuboid sample prints ID: C) can be seen in Figure 4A. This cuboid with height h =6 mm has a cutis thickness t_cutis of t_cutis =2 mm which is separated by a t_SUP =2 mm thick SUP layer. A subcutaneous fat layer with skin pores is tested with three cuboids. All cuboids are built with a cutis thickness of t_cutis =2 mm, subcutaneous fat of t_fat =2 mm and a CEF lattice structure. The pores only differ in diameter, which are: d_pores =0.25 mm (C19), d_pores =0.50 mm (C20) and d_pores =1.00 mm (C18). The 3D model and resulting print of C20 is shown in Figure 4B, its cross-section in Figure 4C.

Figure 4.

Test prints for skin shiftability, fat and pores. (A,B) Test print for the displaceability of two skin layers against each other, separated by SUP. Two TP layers can be moved past each other after SUP crumbled. (B) Test print of a cuboid with CEF lattice structure, subcutaneous fat and pores. The pores allow air to penetrate into the subcutaneous fat and the inner volume. Due to the small diameter of the holes, air can flow inside, but hardly any SUP reaches the outside. (C) The structure of the test cuboid can be seen as an illuminated cross-section. The two skin layers are separated from each other by the SUP and therefore allow displacement. TP, TangoPlus; SUP, support-material; CEF, cube edge face.

Hand assembly

The phantom hand assembly is composed of several surface models. Figure 3C shows a cross-section of this assembly. Segmented bone structures are stiffened with bone connectors. A stiffening of the joints is necessary as the HaptiVisT training system is not made for simulating joint movements. Bone connectors consist of simple cylinders, which are manually positioned between two bones. Bones and bone connectors are combined via Boolean operations using blender software (blender foundation, Amsterdam, Netherlands, www.blender.org). Pores, cutis, periosteum, lattice, subcutaneous fat and boundary are created with nTopology’s Element Pro V1.22.0 (nTopology, New York, USA, www.ntopology.com). The periosteal layer is an extent of the bone mesh. Since the thickness of the cutis varies in reality, we need a variable thickness extrusion. Starting at the segmented hand surface from combined computer tomography (CT) and magnetic resonance imaging (MRI) scans (11,28), the variable cutis thickness is defined using two parameters: a minimum offset t_cutis (min) at distal phalanges and a maximum offset t_cutis (max) at the forearm. The offsets between t_cutis (min) and t_cutis (max) are linear increased along the hand axis and visualized using a heatmap (Figure 5A). Thus, the cutis thickness is defined by the segmented hand surface and the variable extruded hand surface. The subcutaneous fat is located next to the cutis. Since all empty areas are automatically filled with SUP during 3D printing, the thickness of the subcutaneous fat is determined by the interspace between cutis and subcutaneous boundary. Like the cutis, the subcutaneous fat has a variable thickness, too. The subcutaneous boundary with a constant thickness completes the subcutaneous fat. Creating the pores, separately generated wireframe beams protrude from all directions through the cutis and just under the subcutaneous boundary. Boolean differences of cutis and subcutaneous boundary between the thickened ray paths produce holes through all models and thus skin pores can be imitated. Since the creation of an entire lattice model in one calculation step would exceed the computer’s memory, smaller lattice parts are created using auxiliary models. The individual lattice models are loaded separately during the printing process. Auxiliary models are constructed in such a way that the lattice parts are overlapping. Thus, they are firmly connected during printing. Phantom hands can be printed with or without subcutaneous fat. Subcutaneous tissue allows the cutis to be shiftable over the subcutaneous fat. This is of clinical significance as this gliding around bone protrusions enables surgeons to sense bones and locate risk structures. If subcutaneous fat is present, soft tissue is located immediately under the subcutaneous boundary and reaches up to the periosteum. If no subcutaneous fat is modelled, the soft tissue starts immediately next to the cutis.

Figure 5.

Heatmap and 3D printed phantom hands. (A) Heatmap visualizing a linear increase in thickness extrusion with a minimum offset at distal phalanges and a maximum offset at the forearm. Thus, skin layers of different thickness e.g., a variable thickened cutis can be created. (B) Completely 3D printed phantom hands without (H02) and with (H01) screwed on hand marker. Compared to H01 with CF structure, H02 has a CEF design with skin pores, subcutaneous fat and boundary. The bones in H02 cannot be seen through the subcutaneous fat of opaque SUP. After the printing process, the SUP of subcutaneous fat appears black (B, H02) and after kneading it greys up to white (C). CF, cube face; CEF, cube edge face.

Two left hands (hand sample print IDs H01 and H02) with a length of l =30 cm are printed up to the forearm base with a closing VBP adapter plate to mount a tracking marker later (Figure 5B,C). The hands are based on the same segmented MRI and CT data of a 42 years old male person, who has given his informed consent. However, their inner structure differs. Since H01 is printed without subcutaneous fat, it has no variable thickened cutis and no subcutaneous boundary. The selected soft tissue CF rule size of RS =1.5 mm and t_lattice =0.5 mm lattice thickness are chosen as recommended in (17). The thickness of periosteum and cutis is t_periost =0.5 mm and t_cutis =2.0 mm, respectively. In total, the H01 assembly consists of 28 surface models: Cutis, 24 overlapping lattice models, periosteum, bones and adapter plate. In comparison to H01, H02 has a t_lattice =3.5 mm CEF lattice structure, skin pores, subcutaneous boundary and variable thickened cutis as well as subcutaneous fat. The variable cutis thickness ranges from t_cutis (min) =0.5 mm (phalanges) to t_cutis (max) =2.0 mm (forearm) and the subcutaneous fat from t_fat (min) =0.5 mm (phalanges) to t_fat (max) =2.0 mm (forearm). Skin pores have a diameter of d_pores =0.50 mm. The H02 assembly consists of only nine meshes, caused by the bigger rule size: Cutis and subcutaneous boundary with pores, four lattice models, periosteum, bones and adapter plate.

The skin relocatability on phantom hand H02 is measured in comparison to real hands using an experimental setup. One end of a string is placed on a piece of double-sided adhesive tape with size s =2×1 cm². A weight of 500 gram is attached at the other end of the string. The opposite side of the tape is centrally glued onto the backs of the hands of participants and the phantom hand H02. If the hand is positioned vertically at a standardized measuring position in front of a camera, the vertical translation of the skin can be measured via a pixel difference before and after skin displacement. The transfer of pixels to length units is achieved by measuring a checkerboard pattern with known length scales in the same measuring environment. Pixel measurements are performed with the ImageJ software (open source, www.imagej.net). The camera lens distortion is corrected using the Computer Vision Toolbox from MATLAB (The MathWorks, Inc., Massachusetts, USA, www.mathworks.com).

Optical tracking

To use the haptic phantom hand in a VR-based training its 3D orientation must be transferred to the system in real-time and without interference. An optical tracking camera MicronTracker model Sx60 from Claronav (Claron Technology Inc., Ontario, Canada, www.claronav.com) with a field of measurement FOM =115×70×55 cm³ is used to transmit the position and rotation of the 3D printed phantom hand to a computer. This real-time stereo camera tracks so-called XPoints in a 48 Hz rhythm (29). XPoints are identified by a chessboard-like transition from black to white. By a suitable combination of three or four XPoints, two vectors in L- or X-shape can be formed (Figure 6A). This arrangement is called a facet. Two vectors are needed to calculate the position and rotation of a facet in 3D. At least one facet forms a marker (30). Since in our special case not only the translation of the phantom hand needs to be transmitted, but also the rotation around all axes up to 360 degree, a rotation-symmetrical marker shape in dodecahedron design was developed and 3D printed. For the 3D printing, one side of the dodecahedron is left open in order to remove SUP. This significantly reduces the weight of the marker. Afterwards, the dodecahedron is closed. The marker consists of 11 independent hand facets (HF1–HF11, Figure 6B) and is connected by screws to the phantom hand adapter plate. Thus, the marker is exchangeable and can be used for several phantom hands. The tracking point is located at the intersection of the two vectors of the first facet in X-design. If the first facet exceeds the camera field of view and hence, cannot be detected, the tracking point is recalculated via other visible facets (30).

Figure 6.

Tracking marker design. (A) A marker consists of at least one facet. This in turn consists of two vectors, formed from three or four chessboard-like XPoints in L- or X-arrangement (29). (B) The 3D printed dodecahedron marker consists of 11 independent facets (HF1–HF11). The tracking point is located in the center of the vectors of facet one. (C) HaptiVisT working area with dodecahedron and reference marker from the tracking camera’s view. The hand marker is tracked relative to the reference marker. (D) The reference marker design consists of five facets (RF1–RF5) and is located at the upper left corner of the working area, so that not all facets of the marker can be covered by the phantom hand at the same time. The red cross marks the origin of the reference marker.

The 3D hand position is calculated relative to a reference marker for basic calibration. The reference marker consists of five individual reference facets (RF1–RF5) with large vector lengths to enhance stability (Figure 6C). As we choose an L-shape for the first facet the reference markers’ origin lays on the center of its longer vector. All facets are located at the upper left corner of the training systems working area. Thus, the phantom hand rarely covers single marker points. The position of the reference facets is determined empirically. As for the phantom hand marker, a template for the reference marker must be created. The distance of the camera to the worktop is not set to the maximum of the FOM, as the measurement error increases proportionally to the square of the distance (30). The maximum detection distance (MDD) for an XPoint is:

MDD =100·r_XPoint,

where r_XPoint is the radius of an XPoint. For stability reasons, a radius of r_XPoint =1.1 cm is selected for the reference and phantom hand marker design. The dodecahedron and reference marker design from camera perspective is shown in Figure 6D.

For stable tracking, the boundary of the XPoints must be sharp and the contrast between black and white high with a matt finish. For comparison, the XPoints are attached to the dodecahedron in three different variants: First, using printed self-adhesive paper (Figure 7A), second separately 3D printing of black and white XPoint components (Figure 7B) and a combined print, in which black and white material are simultaneously loaded into the printer (Figure 7C). The second variant allows an individual post-processing with lack for each component.

Figure 7.

XPoints of the marker are placed in different variants. (A) Dodecahedron with printed self-adhesive paper. (B) 3D printed dodecahedron with cavities for the white areas. The white XPoint parts are printed separately, painted and subsequently inserted into their corresponding forms. (C) The transition between black and white is blurred by simultaneously printing black and white materials. This leads to an insufficient tracking result.

Evaluation

Two evaluations were performed to analyze the phantom hands haptic and tracking as well as the phantom hand implementation in the training system. Both study designs are prepared in cooperation with the Fraunhofer Center for Applied Research on Supply Chain Services (SCS) and analyzed using the software SPSS 25.

Evaluation 1: phantom hand

The evaluation was carried out in July 2019 at the Trauma Surgery & Plastic, Reconstructive and Hand Surgery Department from the University Hospital Regensburg. A standardized questionnaire using a 6-level Likert scale was developed for an expert survey, which serves to evaluate the phantom hand 3D prints (H01 and H02, see Table 1) regarding haptics, elasticity, softness, palpability of the bones and skin relocatability in comparison to a human hand. Taking Likert scales, a participant must represent his agreement with a statement in the questionnaire on a metric scale by ticking (31). The evaluation ranges from very good [6], good [5], rather good [4] to rather bad [3], bad [2] and very bad [1]. On basis of a 6-level Likert scale, the participants could not adopt a neutral position. In addition, a 6-level Likert scale is more reliable than a standard 5-level scale (32). After filling a demographic questionnaire regarding participant age, gender, work experience and 3D print experience, both phantom hands were kneaded for several minutes by each participant and were then evaluated in the following five categories:

1. Haptic: how do you rate the haptic feeling of the finger in comparison to human soft tissue?

2. Bones: how do you rate the palpability of bones through soft tissue?

3. Elasticity: how do you rate the tissue elasticity of the hand compared to human soft tissue?

4. Softness: how do you rate the softness compared to human soft tissue?

5. Relocatability: how do you rate the skin’s relocatability compared to reality?

Evaluation 2: phantom hand tracking

The second expert evaluation was carried out in June 2018 at the international FESSH conference in Copenhagen. The FESSH is the Federation of European Societies for Surgery of the Hand and unites hand surgical societies from 27 European countries (33). The HaptiVisT system was set up in the exhibition room of the conference and was tested regarding phantom hand tracking and system implementation. Unfortunately, no structured survey of the participants could be carried out, as several people were standing in front of the prototype at the same time. In addition, after filling a demographic questionnaire, the short familiarization phase of five to eight minutes was already mixed with questions and comments from participants. Thus, the participants gave their evaluation in an openly conducted interview. With permission of the participants, the interviews were recorded on tape. Beside a survey on system simulation, three relevant questions were asked relating to the phantom hand:

1. How do you rate the positioning of the haptic phantom using the tracking system?

2. When you compare the training system with the real situation in the operating room: how do you rate the implementation of the haptic phantom in terms of: flexibility, size, general manageability?

3. When you compare the training system with the real situation in the operating room: does the handling of the haptic phantom correspond to what you would expect from the screen display?

The HaptiVisT prototype can be operated in right- or left-handed mode, with default setup right-handed. Therefore, in contrast with Evaluation 1, the participants’ handedness was asked in the demographics.

Results

CEF sample print

As to be expected, the amount of the SUP increases steadily with increasing side surface opening of the CEF unit cell (A02–A04, Table 1). In comparison to the A01 sample in CF design the SUP share within CEF is constantly higher. Thus, using the CEF design, amount of SUP can be increased from 35.19% up to 45.96%. A volume with CEF opening of CEFo =0.25·RS is too hard whereas CEFo =0.75·RS provides less elasticity and geometric stability. To avoid SUP displacement after repeated kneading, a side surface opening of CEFo =0.5·RS with SUP share of 39.51% (see A02 in Table 1) is used for further experiments. This results in an 4.32% SUP increase in comparison to A01. By increasing the SUP share, the amount of TP is automatically reduced and the volume becomes softer. Thus, after several minutes of kneading, underlying bones can be felt through tissue (Figure 2D).

Skin shift sample print

SUP separates the TP skin layers of the cuboid C09 from each other (Figure 4A). By kneading the cuboid, SUP within the subcutaneous fat crumbles completely as supposed. The upper and lower skin layers glide past each other through the crumbly SUP. However, the sliding effect is small. Reasons could be that the dimensions of the cuboid are limited and the vacuum inside the cuboid restricts easy motion. Shifting skin layers is possible, but difficult. After rupturing the skin through frequent strong kneading, the skin layers can easily glide past each other and are no longer restricted by vacuum.

Skin pores sample prints

For a controlled dissolution of the vacuum inside the subcutaneous fat and the lattice structure, holes protrude through cutis and subcutaneous boundary (Figure 4B). After kneading, air rises into the volume. It was seen, that the skin layers glide easily past each other and that SUP in contact with air also disintegrates faster. With cuboid C19, the holes are too small, so that almost no air can penetrate to the inner layers. However, with cuboid C18 the holes are too large. SUP gets outwards in large quantities. Cuboid C20 offers a good trade-off. Air flows into the volume and hardly any SUP reaches the surface. Thus, C20 creates a haptically and closed 3D printed surface, which is opened by only small and hardly noticeable holes with d_pores =0.5 mm.

Hand sample prints

Both assemblies (H01, H02) are 3D printed within about 17 hours each (Figure 5B). It turns out that the SUP inside small CF unit cells with RS =1.5 mm of H01 can only be crumbled partially when applying external pressure. Hence, SUP softens the raw TP only slightly. Furthermore, a skin relocatability is not possible without subcutaneous fat. The H02 rule size enlargement from RS =1.5 mm to RS =3.5 mm ensures, that the entire SUP can be crumbled even in large volumes up to the bones (Figure 2D). The CEF design significantly softens the raw TP material and bones can be felt under the soft tissue mantle. Further the cutis can be freely moved because of the subcutaneous fat. Cutis and subcutaneous boundary glide past each other due to the penetration of air through numerous skin pores. Additionally, the skin layers are not locally limited as with C09.

The human skin relocatability of five participants (N =5) was compared with the skin relocatability of the phantom hand H02. All participants were male with an age between 23 and 29 years (M =26.0, SD =2.45). For each participant the vertical skin displacement was measured three times and averaged over all five participants. The skin relocatability for the phantom hand was measured with five different pieces of the same adhesive tape three times and averaged. The mean relocatability of human hands δ(human) results in δ(human) =13.9 mm (SD =0.80) whereas the mean of the phantom hand H02 is δ(H02) =7.6 mm (SD =1.67). Therefore using H02, we achieve a 54.86% coincidence compared to reality. Figure 8A shows the skin relocatability measurement of the phantom hand, Figure 8B of a male participant and Figure 8C the checkboard pattern for pixel-length transmission and correction of lens distortion.

Figure 8.

Skin relocatability measurement of the phantom hand H02 (A) and of a male participant (B). The images are captured from a camera and show the superposition of images before and after displacement (δ). A 500 gram weight is attached to one end of a string, whereas the other end is centrally glued with a double-sided adhesive tape onto the back of the hands. (C) Checkerboard pattern with known length scales for a pixel-to-length unit transmission and camera lens distortion correction. (D) 3D printed phantom hands evaluation: Expert surgeon results grouped in percental positive (mean of rather positive, positive and very positive; blue) and negative (mean of rather negative, negative and very negative; red) participant feedback according to haptic, bone palpability, elasticity, softness, and skin relocatability. The evaluation contained the phantom hands H01 and H02 (see Table 1).

Object tracking

The rotationally symmetric dodecahedron marker offers the ability, that up to three facets are tracked simultaneously by the camera. Thus, raw data transmission and calculation of the tracking point is extremely stable. Even with fast changes in position or rotation, the camera never loses the marker. Due to the dodecahedron’s steep angle of 30 degrees, the attached phantom hand can be positioned almost perpendicular to the camera without interrupting tracking. The phantom hand never covers all facets of the reference marker at the same time, resulting in a robust reference coordinate system. Transmission of the 3D position of the phantom hand to the training system is stable with 48 Hz. Since the raw data must be filtered for a jitter-free transmission, the hand visualization of the training system moves slightly slower.

Printing the dodecahedron marker with its connection to the haptic phantom proceeds error-free. The glued-on facets are detected by the camera without any restriction. However, occasionally detaching of the paper is a disadvantage (Figure 7A). Printing black and white XPoints simultaneously, materials are automatically blended into each other by the 3D printer. This small overlap blurs edges of the XPoints (Figure 7C). In addition, the contrast of the white material is low. This results in an unstable tracking process. The camera loses tracking even with small phantom rotations as well as in a distance of about 50 centimeters without any rotation. Separately printed black and white XPoints are finished with black or white matt paint and then placed one inside the other (Figure 7B). Precise operating is necessary in order to not damage the edges of the XPoints. Finally, the tracking result is accurate and reliable without restriction.

Expert surgeon evaluation

Phantom hand evaluation

Ten participants (N =10) assessed both phantom hands in terms of haptic feeling, bone palpability, tissue elasticity, softness and skin relocatability compared to human soft tissue. All participants could be included in the analysis without exception, as the clinical background and knowledge of human anatomy were given in a Trauma Surgery & Plastic, Reconstructive and Hand Surgery Department. Four participants were female and six male with an age ranged from the youngest participant 26 to the oldest 40 years old (total: M =33.9, SD =4.93; women: n =4, M =34.5, SD =6.56; men: n =6, M =33.5, SD =4.18). They had in average 2.2 years of surgical experience (SD =3.11) in the field of trauma surgery (n =5, M =0.6, SD =0.51), surgery and plastic surgery (n =3, M =6.3, SD =2.52) or were medical students (n =2) without surgical experience. The 6-level Likert test answers and their means are shown in Table 3. Similar to Kozlowska and Rehman [2017], the evaluation is divided into positive (mean of rather positive, positive and very positive) and negative feedback (mean of rather negative, negative and very negative) (34). H02 outperformed H01 in each category queried (haptic, bone palpability, elasticity, softness and relocatability). Figure 8D shows a boxplot with the summarized evaluation results. The participants were grouped in positive and negative feedback according to each category and phantom hand in percent. The percentages describe the ratio between positive and negative scores given from all participants. Experts evaluated all categories of H01 negatively in haptic (60% negative feedback), bone palpability (90% negative feedback), elasticity (70% negative feedback), softness (90% negative feedback) and relocatability (80% negative feedback). On the other hand, H02 convinced with its elasticity (80% positive feedback), softness (80% positive feedback), bone palpability (70% positive feedback) and relocatability (60% positive feedback). The H02 phantom’s haptic was evaluated neither positive (50% positive feedback) nor negative (50% negative feedback). For descriptive statistics we use the non-parametric rank-based Mann-Whitney-U test to compare the phantom hands H01 and H02 in categories haptic, bone palpability, elasticity, softness and skin relocatability (35,36). In contrast to our assumption, we suppose in our null hypothesis (H₀) that H01 outperforms H02 and use a one-tailed Mann-Whitney-U calculation with equally distributed participant populations (H01: n₁ =10; H02: n₂ =10). H₀ can be rejected if the U-statistic (U) is smaller or equal than the critical value U_crit of probability P(U ≤ U_crit│H₀,n₁,n₂) ≤α. The critical value for a 5% significance level (α =0.05) and 1% level (α =0.01) is 27 (P(U≤27│H₀, n₁ =10,n₂ =10) ≤0.05) and 19 (P(U ≤19│H₀,n₁ =10,n₂=10) ≤0.01), respectively. At the 5% level tested U-statistic the H02 bone palpability (U =20.5), elasticity (U =25) and skin relocatability (U =23.5) is significantly better than H01 as we can reject H₀. Tissue softness (U =10) is highly significant at a 1% level, whereas the haptic category shows no significant difference at all (U =32.5).

Table 3. Evaluation of 3D printed phantom hands H01 and H02 according to haptic, bone palpability, elasticity, softness and relocatability.

Category	Very good [6]	Good [5]	Rather good [4]	Rather bad [3]	Bad [2]	Very bad [1]	Mean
H01
Haptic	0 [0]	0 [0]	4 [16]	5 [15]	1 [2]	0 [0]	3.3
Bones	0 [0]	1 [5]	0 [0]	5 [15]	4 [8]	0 [0]	2.8
Elasticity	0 [0]	1 [5]	2 [8]	6 [18]	1 [2]	0 [0]	3.3
Softness	0 [0]	0 [0]	1 [4]	5 [15]	4 [8]	0 [0]	2.7
Relocatability	0 [0]	1 [5]	1 [4]	3 [9]	4 [8]	1 [1]	2.7
H02
Haptic	0 [0]	5 [25]	0 [0]	5 [15]	0 [0]	0 [0]	4.0
Bones	0 [0]	3 [15]	4 [16]	2 [6]	1 [2]	0 [0]	3.9
Elasticity	0 [0]	2 [10]	6 [24]	2 [6]	0 [0]	0 [0]	4.0
Softness	0 [0]	2 [10]	6 [24]	2 [6]	0 [0]	0 [0]	4.0
Relocatability	0 [0]	3 [15]	3 [12]	3 [9]	1 [2]	0 [0]	3.8

The rating is, according to a 6-step Likert scale, divided into very good (6 points), good (5 points), rather good (4 points), rather bad (3 points), bad (2 points) and very bad (1 point). The table contains both, the number of answers given and the multiplication of the answers given by the rating score (in square brackets). The mean value is the sum of all points divided by the number of participants (N =10).

Phantom hand tracking evaluation

Expert interviews are restricted to participants with practical work experience. Therefore, 28 out of 29 interviews are included in the analysis (N =28), as one participant was a medical student. Ten interviews were conducted with women and 18 with men. The average age of the participants was 41.9 years (SD =10.95), with the youngest participant 27 and the oldest participant 67 years old (women: n =10, M =41.4, SD =12.80; men: n =18, M =42.4, SD =10.17). On average, the participants had 13.8 years of practical work experience (SD =10.69) in hand surgery (n =17, M =15.8, SD =10.74), in traumatology (n =6, M =11.2, SD =10.72) and in other medical fields (research: n =2, M =13.0, SD =7.07; plastic surgery: n =3, M =8.3, SD =14.43). Twenty-seven participants were right-handed and had no experience with hand surgical computer training simulators before the interview. As the survey was not structured but openly designed using voice recording, the interviews were transcribed using MAXQDA software (VERBI Software GmbH, MAXQDA—The Art of Data Analysis, Berlin, Germany, www.maxqda.de). Individual statements of the participants are selected for evaluation. Twenty-one participants gave positive feedback about positioning of the phantom hand within the tracking system. The positive statements fluctuated between “okay” and absolute enthusiasm, as the following selected voices prove: Single participants gave feedback like “It’s really fast, yeah, it’s good”, “This seems perfect” and “It’s amazing. I’ve never seen something like that”. Only two participants expressed themselves critically with the statements “This has a minimal delay. [...] But I think you’re extremely flexible in positioning” and “It’s good, but you have to get familiar to it. I mean, if you are using it the first time, it’s difficult.” In contrast to the last answer one person mentioned “It takes getting used to, but it’s not a problem. You need to be able to adapt to something during training.” The minimal delay between the visualization of the hand and the position transmission observed is caused by filtering of the raw data. The remaining five people adopt a neutral position on phantom hand tracking.

Discussion

In this paper, we have expanded and improved on the findings of imitating human soft tissue gained from previous work (17). The result is a haptically correct and optically tracked whole phantom hand for a medical training system. The final phantom hand (H02) has a movable skin, subcutaneous fat and soft tissue, so that underlying protrusions of the bones can be palpated. This was confirmed by an expert surgeon study, in which participants continuously rated the phantom hand H02 better than H01 regarding haptic, bone palpability, tissue elasticity as well as softness and skin relocatability. Statistical analysis shows that skin relocatability of a phantom hand (H02) provides significant enhancement in bone palpability compared to a phantom hand without subcutaneous fat (H01). In comparison to reality we achieve a 54.86% agreement regarding the skin relocatability. Thus, the friction generated by the SUP between cutis and subcutaneous boundary forms a limiting factor. In our skin relocatability experiment, only the skin shiftability of young men was evaluated. Therefore, it can be assumed that the skin shiftability of older men is even higher due to reduced elasticity and skin strength (37). The new CEF-metamaterial-design and skin pores resolving the vacuum within the volume provide significant improvement in tissue elasticity and highly significant refinement in tissue softness. Unfortunately, although TP is extremely resistant, it begins to break when exposed to heavy kneading stress. An improvement in durability could provide a final finish. However, a silicone or PlastiDip liquid rubber (Plasti Dip International, Minnesota, USA, www.plastidip.com) coating has not yet provided sufficient results. Using 3D printing, casting moulds according to Brichacek et al. [2018] could be created to pour the phantom hand in a layer of flexible silicone improving phantom hands life periods (38). Since there is currently no connection between cutis and subcutaneous boundary, it is possible to lift the cutis from the subcutaneous fat. By imprinting fascial tissue using biotensegrity mechanics (39), this lift-off could be reduced to an appropriate level. The implementation of the phantom hand in our HaptiVisT training system using real-time optical tracking works stable with 48 Hz and generates a jitter-free hand simulation. The XPoint based rotational symmetric dodecahedron marker is mounted via an adapter plate to the phantom hand. The tracking was rated as “[…] really fast […]”, “[…] perfect” and “[…] amazing […]” in course of an expert surgeon evaluation. The visualization moves slightly slower due to data filtering. This was only stated negatively from two study participants only.

The transfer of human anatomy and metamaterial design to imitate soft tissue can be applied not only to a phantom hand but also to various other parts of the body. Realistic phantoms are becoming increasingly important in surgical training and offer great potential for future training (40). Haptic phantoms can be used beside training simulators also in surgical planning, device development, diagnosis and orthopedics. The phantom hand tracking can be implemented easily in a surgical training system but is not limited to medical training.

In future work, the hand model can be further extended to include nerves, vessels, tendons and muscle strands. Muscle strands with differently thickened CEF produce various soft tissue shore hardness and tendons with higher TP shore hardness could be felt through the cutis. To further soften the soft tissue and to print vessels, new printing material with lower shore hardness has to be developed.

The holes in cutis and subcutaneous boundary with connection to the CEF lattice structure may be used to wash SUP out of the object volume. However, this requires washable SUP (e.g., SUP706), which unfortunately cannot be loaded by the printer and was thus not available for this research.

Currently, the XPoints of the reference marker are printed on paper and glued upon the wooden HaptiVisT working plate. The next setup will include laser-cutted XPoints to meet maximum precision.