Development and Quantitative Evaluation of a Novel Autonomous In situ Bioprinting Surgical Robotic Framework For Treatment of Volumetric Muscle Loss Injuries

Abstract

In situ bioprinting has been identified as a promising tissue engineering technique for treating volumetric muscle loss (VML) injuries. However, the success of this procedure significantly depends on the uniform and precise deposition of cells contributing to the regeneration of muscles. To address this critical need, in this work, we present design and quantitative evaluation of a novel autonomous in situ bioprinting surgical robotic framework that can be used with a generic bioprinting material. The proposed framework consists of three main components: (i) a bioprinting tool integrated with a seven-degree-of-freedom robotic manipulator to perform a precise autonomous bioprinting procedure; (ii) a unique 3D visual measurement framework comprised of a high-accuracy structured light camera with complementary 2D/3D computer vision algorithms- to enable online and accurate measurement and reconstruction of the bioprinted constructs; and (iii) a quantitative evaluation module with novel assessment metrics- to characterize and evaluate the performance of the bioprinting process toward finding optimal bioprinting parameters. To ensure the biological functionality of a printed construct using our robotic system, we performed 90 experiments and identified optimal bioprinting parameters using the proposed novel assessment metrics.

Note to Practitioners–

This paper was motivated by the problem of volumetric muscle loss treatment using an in situ bioprinting procedure but it also can be applied for treatment of skin and cartilage injuries. Existing approaches to perform in situ bioprinting is limited to either manual handheld bioprinting devices– that suffer from poor manual control and inaccurate printing constructs– or robotic systems– that have been developed without (i) considering a realistic surgical workflow and (ii) quantitatively evaluating the quality of printed constructs. To collectively address these issues, in this paper, we propose a novel autonomous in situ robotic bioprinting framework. We also introduce unique and complementary quantitative assessment metrics to characterize and evaluate the performance of the bioprinting process. Experiments suggest that the proposed framework can robustly identify optimal bioprinting parameters to ensure the biological functionality of a printed construct using our robotic system.

I. INTRODUCTION

SEVERE musculoskeletal injuries, resulting from traumatic events such as car accidents and the combat injury, can lead to volumetric muscle loss (VML) with permanent loss of function [1], [2]. To overcome the existing limitations for surgical and prosthetic treatments of VML injuries, many researchers worldwide have been developing nano- and micro-fabrication technologies to engineer native-like tissues for muscle regeneration [2], [3]. Typically, in these tissue engineering procedures, cells from patients or donors are manually seeded into porous scaffolds to form artificial muscle tissues for implantation [3], [4]. However, these manual procedures often result in a set of randomly organized cells that can poorly contribute to the regeneration of muscles, whereas the alignment and maturation of muscular cells are critical factors for engineering functional muscle tissues [4], [5]. Although many tissue engineering strategies have been utilized to fabricate muscle tissues and their corresponding extracellular matrix, the capability for the development of a three-dimensional (3D) spatial cell organization that favors cell maturation and differentiation cascade is still lacking [4], [6]. An emerging strategy to overcome these barriers is to directly printing a scaffold into the defect sites in situ [2], [4].

In situ bioprinting is an innovative technique that combines 3D printing technology with biological materials to create complex, functional tissues and organs directly within the body [7], [8]. This technique can circumvent fluid loss and enables fabrication of customized implants that can meet the clinical requirement for tissue repairs. Compared with in vitro bioprinting that is performed on ex vivo substrates, this novel approach reduces the risk of introducing infections to the host or involuntarily damaging the printed constructs during the transplantation [9], [10]. Moreover, it achieves a low-cost and efficient fabrication on account of the successful organotypic differentiation in the natural and growth-factor-rich cellular environment [2], [4], [10].

To perform in situ fabrication of tissues, manual handheld in situ bioprinting devices have recently been proposed to directly deliver bioinks into the injury site to repair skin, cartilage, and muscle defects [4], [11]–[13]. Despite the benefits of using these devices and obtaining promising results, these systems suffer from various limitations including (i) poor manual control of the handheld bioprinter directly affected by the user’s muscle fatigue and hand tremor during the printing procedure of large injuries; and (ii) heavily relying on the user’s expertise and experience on using the bioprinting device and controlling the printing procedure [4], [14]. These shortcomings, therefore, lead to a long and hard-to-use printing procedure for large 3D injuries, non-uniform deposition, and inaccurate stacking of multi-layer constructs. Most importantly, they cause the lack of precision in the reconstruction of complex and large 3D curved and injury shapes that leads to random and unorganized deposition and poor maturation of cells contributing to the regeneration of muscles [4], [5], [15].

To address the aforementioned challenges, bioprinting robotic systems have also been recently proposed in the literature (e.g., [16]–[19]). However, a review of the literature demonstrates that these systems often (i) are either bulky and do not provide sufficient degrees-of-freedom (DoFs)– mostly limited to X-Y-Z motions– to print complex 3D structures on different anatomies; and (ii) do not consider a realistic surgical setting and workflow in the operating room (e.g., [4], [16]. For instance, some systems (e.g., [16]) cover the whole patient’s body and does not consider inadvertent patient movement during the surgery to automatically adjust robotic pre-planned printing motions, thus endangering patient’s safety and reducing the printing accuracy; and (iii) most importantly, do not quantitatively evaluate and characterize the performance of the printed constructs and limit their evaluations solely to qualitative assessments (e.g., [18]–[20]). For example, Fortunato et al. [18] designed and assembled a 5 DoF robotic arm to enhance this procedure by enlarging the working space to non-planar and non-smooth surfaces, which are common in the human body. Besides, Zhao et al. [19] developed a multi-DoF autonomous robot to process in situ bioprinting for skin repairs. Although the predefined motions are performed with high precision and repeatability in these works, the chosen bioprinting parameters (e.g., injection and robot speeds) and assessment of the printed constructs’ geometric parameters are limited to qualitative visual comparisons, which are highly subjective and exclusive to the used experimental setup. Particularly, geometric bioprinting parameters (i.e., uniformity and thickness of printed constructs) are only evaluated by measuring the robot motion accuracy or from follow-up evaluations through offline in vivo/vitro experiments using sophisticated microscopes. Of note, the former evaluation only ensures the precision of robotic motions, not the bioprinting procedure as it ignores quantitatively evaluating the thickness and uniformity of the printed hydrogel constructs. Moreover, the latter evaluation is performed offline after the printing procedure and is both time- and money-consuming. Nevertheless, without a thorough quantitative evaluations of the bioprinted constructs and particularly evaluating their geometric uniformity, providing desired conditions for cell growth and, therefore, the biological function of the formulated new tissue cannot be guaranteed [4], [6], [10]. Overall, despite the progress made in the quantitative evaluation of in vitro/situ bioprinting constructs, a review of the existing literature demonstrates that these evaluations still demand a framework for (i) an accurate and online segmentation and quantitative evaluation of the bioprinting constructs right after the procedure; and (ii) the development of novel scoring and assessment metrics that can thoroughly reflect the influence of the geometric parameters on the quality of bioprinting procedure with any bioprinting material to ensure further cell growth and the biological function after the printing procedure.

To collectively address the above-mentioned limitations and as our main contributions, this work presents the design, thorough characterization, and performance evaluation of a novel autonomous in situ bioprinting surgical robotic framework (shown in Fig. 1 and Fig. 2) developed for VML treatment using a generic bioprinting material. The proposed robotic framework has been developed, for the first time, to holistically and systematically address the development and quantitative evaluation of a unique and generic in vivo robotic bioprintig framework that can be used with a generic bioprinting material (i.e., bioink). In other words, to comprehensively address this problem, as opposed to the existing methods in the literature, focusing either on the hardware or algorithmic components of the robotic system, we are introducing a systematic procedure and system to characterize the optimal bioprinting parameters of any bioprinting material to ensure cell growth and biological functionality. The proposed framework consists of three main components: (i) a bioprinting tool integrated with a seven-DoF robotic manipulator to precisely perform a bioprinting procedure (Fig. 2(a)); (ii) a unique and novel 3D visual measurement framework and process that includes a high-accuracy structured light camera with complementary 2D/3D computer vision algorithms to enable an accurate measurement and reconstruction of the geometric parameters of the bioprinted constructs (Fig. 2(b)); and (iii) a quantitative evaluation module with novel quantitative assessment metrics to characterize and evaluate the performance of the bioprinting process toward finding optimal bioprinting parameters (Fig. 2(c)) with a generic bioprinting material. To completely characterize the proposed framework, we performed various experiments and quantitatively evaluated its performance using a set of novel metrics. To our knowledge, the proposed robotic system and quantitative evaluation framework are the first of its kind for performing an autonomous in situ bioprinting procedure for the treatment of VML injuries.

Fig. 1.

The proposed autonomous in situ bioprinting surgical robotic system composed of a bioprinting tool integrated with a seven-DoF robotic manipulator, and a high-accuracy structured light camera. Figure also shows a calibration board used to calibrate the proposed measurement framework.

Fig. 2.

The proposed multi-step procedure for autonomous bioprinting and quantitative characterization of the printed filaments. As shown, first a bioprinting tool integrated with a seven-DoF robotic manipulator performs a bioprinting procedure (Step a). Next, a 3D visual measurement system together with complementary 2D/3D computer vision algorithms are used to reconstruct the geometric parameters of the bioprinted constructs (Step b). Finally, a quantitative evaluation module with novel assessment metrics are used to quantify the bioprinting performance by measuring the local and global uniformity of the printed filament (Step c).

II. The In situ Robotic Bioprinting Framework

As shown in Fig. 1 and Fig. 2, the overall components of the proposed robotic system for in situ bioprinting comprises of a bioprinting tool, a seven-DoF robotic manipulator (LBR iiwa, KUKA) for precisely controlling the position of the bioprinting tool, and a highly-accurate 3D structured light visual measurement system (i.e., Zivid Two M70, Zivid Inc.) for quantitatively evaluating and analyzing the printing constructs (Fig. 1). All components are connected and controlled through a host computer and using Robot Operating System (ROS) [21]. Of note, as opposed to the previous literature (e.g., [18], [19]), solely focusing on a robotic bioprinting process without quantitatively assessing the quality of bioprinted constructs, our proposed system takes advantage of the utilized 3D structured light visual measurement system and the complementary measurement algorithms developed in this paper to completely characterize the proposed framework. The following sections describe these components and developed algorithms in detail.

A. Mechanical Design of Bioprinting Tool

As shown in Fig. 3, the overall mechanical structure of the bioprinting tool comprises a 50 ml plastic syringe containing the bioprinting material with an 18 gauge needle, a linear stepper motor (P8-50-165-3-ST, Actuonix Inc.), a stepper motor driver (TIC T825, Pololu Inc.), a joystick controller for controlling the bioprinting injection, and a micro-controller (Arduino micro, Arduino). Of note, based on the previous literature (e.g., [22], [23]), the selected linear stepper motor provides a maximum force of 25 N, maximum speed of 15 mm/sec, and a stroke of 50 mm for the proposed bioprinter. The used syringe and its needle can readily be changed if needed. Also, the bioprinting procedure can either be controlled manually using the joystick or autonomously by controlling the stepper motor using the ROS interface.

Fig. 3.

The overall mechanical structure of the bioprinting tool.

B. Three-Dimensional (3D) Visual Measurement Process

Uniform thickness and consistency of the printed hydrogels (i.e., filaments) play a fundamental role in scaffold formation during a bioprinting procedure. It is worth mentioning that non-uniformity in the filament’s thickness may decrease the pore size of the scaffold, which can result in a degraded cell growth and biological function [24], [25]. Therefore, an optimal bioprinting procedure and complementary measurement and evaluation metrics are necessary to fabricate and assess the uniformity of printed filaments regardless of the used bioprinting material. Figure 2 and Fig. 4 illustrate the proposed multi-step procedure to address this important need and, particularly, quantitative characterization of our bioprinting robotic system and the printed filaments. In Step I, as shown in Fig. 4(a), the proposed workflow starts with a pre-operative 3D scanning (i.e., before starting the printing procedure) of the printing site (e.g., injury) in which the normal map of the bioprinted scaffold is acquired via a 3D visual measurement system followed by printing a scaffold using the robotic system. Next, in Step II, as shown in Fig. 4 (b)), using post-operative scanning (i.e., after printing the filaments), the pixels/sub-pixels of the filaments are masked in the colored images (i.e., 2D segmentation). Using the segmented 2D masks, the 3D point cloud of filaments is then acquired (i.e., 3D segmentation). Finally, In Step III, the actual sizes of the filaments will be calculated using the 3D point cloud data obtained from pre- and post-operative site scanning, (Fig. 4 (c)). The following sections describe these steps in detail.

Fig. 4.

The overall process for quantitative characterization of filament thickness: (a) Step I: Pre-operative Site Scanning: The normal map of the pre-printed site is acquired through preoperative site scanning; (b) Step II: 2D/3D Segmentation: Using the obtained colored images from the post-operative scanning of the bioprinted scaffold, a 2D/3D segmentation procedure is performed to mask filament pixels and their corresponding 3D point cloud data, respectively; and (c) Step III: Point Cloud Measurement and Filament Thickness Evaluation: The actual sizes of the printed filaments is quantified and calculated.

1). Step I: Pre-operative Site Scanning:

To assist the thickness calculation, a normal map of the printed site is pre-operatively captured using the utilized Zivid camera and the provided software (ZIVID SDK 2.9) at the pre-operative site scanning stage (Fig. 4(a)). Note that this scanning is only performed at the beginning of the workflow. The acquired pre-operative normal map is then used in each post-operative scanning performed in Step III, to determine a projection surface parallel with the thickness dimension (see Fig. 4(d) and Fig. 6). Notably, the obtained point cloud data is highly accurate and noise-free as the camera SDK automatically performs a denoising step after obtaining images.

Fig. 6.

Conceptual illustration of the proposed approach to determine the thickness direction $\stackrel{\rightharpoonup }{v_t}$ and the thickness value $f_t^i$ of the $i$-th element along a printed filament. In this figure, $\stackrel{\rightharpoonup }{n_s}$ denotes the normal vector of the surface and $\stackrel{\rightharpoonup }{v_l}$ represents the filament direction, which is defined by connecting the shown median points. $\stackrel{\rightharpoonup }{p_{m^{\text{*}}}}$ and $\stackrel{\rightharpoonup }{p_{k^{\text{*}}}}$ represent projected points leading to the maximum and minimum values along the thickness dimension, respectively.

2). Step II: 2D/3D Segmentation:

In this step, as illustrated in Fig. 4(b), using the obtained colored images from the post-operative scanning of the bioprinted scaffold, a 2D/3D segmentation procedure is performed to mask filament pixels and their corresponding 3D point cloud data, respectively. It is worth mentioning that segmentation of a hydrogel filament is a challenging procedure and depends on the characteristics of the utilized structured-light 3D camera including: (1) its spatial resolution (e.g., 0.18 mm for the Zivid Two M70, Zivid Inc.), which makes measurement of filament’s small thickness very difficult and highly error-prone; and (2) the minimum required working distance of the camera to the filaments (e.g., 300 mm for the Zivid Two M70, Zivid Inc.), which may degrade the image resolution and thus limit the acquisition of filament boundaries with fine details.

To address these issues and develop an interactive 2D/3D segmentation framework, we propose utilizing the GrabCut algorithm [26]. This algorithm is selected as it only requires sparse scribbles as the annotations and uses texture (color) and boundary (contrast) information of the image to minimize the amount of user interaction for obtaining a better segmentation performance [26]. With the segmentation map attained by this algorithm, an interactive correction step is then performed to correct the mis-segmented regions, which increases the accuracy of this algorithm compared with automatic segmentation procedures [27]. These two procedures (i.e., segmentation and interactive correction) are conducted iteratively until the segmentation map achieves a gold-standard-level segmentation accuracy, which means no obvious errors can be visually found in images. Typically, this loop only needs to be repeated 1~2 iterations for each image to obtain the required segmentation accuracy and time.

Although, as shown in Fig. 4(b), pixel-level perfect segmentation can be achieved via the GrabCut interactive segmentation algorithm (i.e., “Interactive segmentation” step in Fig. 4 (b)), boundary regions still suffer from aliasing due to the mentioned resolution limitation of the used 3D camera, which can impair the point cloud measurement. Therefore, following the interactive segmentation step, we propose performing an additional Sub-Pixel-level Refinement (SPR) where a region-based level set method [28], [29] is leveraged to refine the segmentation predictions. In this method, the mask boundary of 2D pixel-level segmentation map is used as the initial contour and actively is adjusted to approach the filament boundary. In this process, the contour implicitly evolves toward the filament boundary by minimizing the following energy function [28]:

$${\epsilon }_s\left(\varphi \right)=\mu {\int }_{\Omega }p\left(\right|\nabla \varphi \left|\right)dx+\lambda {\int }_{\Omega }g{\delta }_s\left(\varphi \right)|\nabla \varphi |dx+\alpha {\int }_{\Omega }g{H}_s(-\varphi )dx$$ (1)

where $\varphi$ denotes the contour function, $\Omega$ represents the domain of the image, while $p(\cdot )$ denotes potential function and $g$ denotes the edge map which takes smaller values at boundaries than at other locations in the image. The Dirac delta function $\delta$ and Heaviside function $H$ are defined by:

$$H_s\left(x\right)=\left\{\begin{array}{ll}\frac{1}{2}\left(1+\frac{x}{s}+\frac{1}{\pi }\text{sin}\left(\frac{\pi x}{s}\right)\right),& \left|x\right|<s\\ 1,& x>s\\ 0,& x<-s\end{array}\right.$$ (2)

and

$${\delta }_s\left(x\right)=\left\{\begin{array}{ll}\frac{1}{2s}\left(1+\text{cos}\left(\frac{\pi x}{s}\right)\right),& \left|x\right|\le s\\ 0,& \left|x\right|>s.\end{array}\right.$$ (3)

Moreover, $s,\mu ,\lambda$ and $\alpha$ are fixed parameters. For this study, according to [28], we selected 1.5, 0.2, 5, and 1.5 for these parameters, respectively.

Optimizing the continuous energy function (1), the level set function can generate a sub-pixel-level continuous boundary for the segmentation mask to further mitigate the errors caused by bad resolution, obtained during the segmentation using the GrabCut algorithm. Given the initial segmentation mask reaching the pixel-level accuracy, the second and third regularization terms in (1) can further constrain the length and areas to prevent complex and unrealistic boundaries during the contour evolution. Comparative examples are presented in Fig. 5 demonstrating an observable aliasing effect in the pixel-level segmentation map (i.e., “w/o SPR”) that can be mitigated using the proposed SPR (“w/ SPR”). As shown, with the sub-pixel-level segmentation map, the algorithm can accurately mask the 3D points belonging to printed filaments. Of note, to further improve the performance of the segmentation step, it might also be necessary to assign values for a few points in the measured point cloud data that often miss the depth information. To remedy this issue, we used a bilinear interpolation [30] to estimate the missing depth values of the sub-pixel contour points and make the contour continuous.

Fig. 5.

Visualization of the segmentation boundaries with or without SPR. The red curve and arrows indicate the segmentation boundaries and the aliased regions, respectively.

3). Step III: Point Cloud Measurement and Filament Thickness Evaluation:

Point cloud measurement (Fig. 4(c) and Fig. 6) calculates the filament thickness using the normal vectors obtained from the pre-operative site scanning in Step I and the 3D point cloud of the segmented filament in Step II. To perform the thickness calculation, the point cloud of printed filaments will first be projected to the pre-operative site plane along the corresponding normal vector, calculated in Step I.

Next, from the projected 2D point clouds, a skeleton extraction algorithm, proposed in [31], is adopted to acquire the median points, as shown in Fig. 6. After obtaining a large number of median points, the printed filament is split into a sequence of elements $i$ determined by adjacent median points. Using the two adjacent median points, defining element $i$, and by subtracting them, the filament direction $\stackrel{\rightharpoonup }{v_l}$ can be calculated (shown in Fig. 6).

For each element $i$, by calculating the cross-product of the plane normal vector $\stackrel{\rightharpoonup }{n_s}$ and the filament direction $\stackrel{\rightharpoonup }{v_l}$, the thickness direction is obtained as $\stackrel{\rightharpoonup }{v_t}$. Using these orthogonal directions, a corresponding coordinate frame with an origin located at the middle point of each element $i$ can then be defined. Considering this coordinate frame, an arbitrary projected point with index $j$ located inside an element $i$ can be represented by a vector $\stackrel{\rightharpoonup }{p_j}$ (Fig. 6). The projection of vector $\stackrel{\rightharpoonup }{p_j}$ to vector $\stackrel{\rightharpoonup }{v_t}$ is then used to calculate the thickness value $f_t^i$ of the $i$-th element. To this end, we first find the indices $m^{*}\in j$ and $k^{*}\in j$ denoting the projected points $\stackrel{\rightharpoonup }{p_j}s$ with the maximum and minimum projected length (i.e., thickness) along vector $\stackrel{\rightharpoonup }{v_t}$, respectively, as follows:

$${\mathit{\text{m}}}^{*}=\underset{j}{\mathit{\text{argmax}}}\stackrel{\rightharpoonup }{v_t}\cdot \stackrel{\rightharpoonup }{p_j},k^{*}=\underset{j}{\mathit{\text{argmin}}}\stackrel{\rightharpoonup }{v_t}\cdot \stackrel{\rightharpoonup }{p_j}$$ (4)

where (⋅) represents the inner-product between vectors $\stackrel{\rightharpoonup }{v_t}$ and $\stackrel{\rightharpoonup }{p_j}$.

Using the obtained vectors $\stackrel{\rightharpoonup }{p_{m^{\text{*}}}}$ and $\stackrel{\rightharpoonup }{p_{k^{\text{*}}}}$, then thickness value $f_t^i$ of the $i$-th element can be calculated:

$$f_t^i=\frac{\stackrel{\rightharpoonup }{v_t}\cdot \left(\stackrel{\rightharpoonup }{p_{m^{*}}}-\stackrel{\rightharpoonup }{p_{k^{*}}}\right)}{‖\stackrel{\rightharpoonup }{v_t}‖}$$ (5)

where ∥·∥ denotes the two norm of a vector.

C. Quantitative Evaluation Metrics

As opposed to the existing qualitative evaluation metrics (e.g., [18], [19]), to quantitatively evaluate the uniformity of the printed filaments, in this study, we define two new metrics representing the element-wise Local $e_l^i$ and Uniformity $e_u^i$ errors of each element $i$ along a printed filament. Of note, as shown in Fig. 2(c), the local error $e_l^i$ reflects the accuracy of the printing thickness in the $i$-th element (i.e., $f_t^i$) along the printed filament as compared with a desired printing thickness $f_t^{des}$, whereas the Uniformity error $e_u^i$ indicates a local thickness uniformity of the printed filament on two neighboring elements $i$ and $i-1$.

$$e_l^i=‖f_t^i-f_t^{des}‖,e_u^i=‖f_t^i-f_t^{i-1}‖$$ (6)

Moreover, to quantify the overall performance of the procedure in printing the whole filament, the Overall Global Error $E_g$ and the Overall Uniformity Error $E_u$ are defined as follows:

$$E_g=\frac{\sum _i{e}_l^i}{N},E_u=\frac{\sum _i{e}_u^i}{N}$$ (7)

where $N$ denotes the total number of considered elements $i$ in a printed and measured filament. In other words, this coefficient determines the level of discretization of the printed filaments for our analysis and calculating the local error $e_l^i$, the Uniformity error $e_u^i$, the Overall Global Error $E_g$, and the Overall Uniformity Error $E_u$. A smaller $N$ causes missing important information and thickness uniformity due to the large discretization; while a larger $N$ might be unnecessary and adds to the computational and processing time. Overall, this parameter can be tuned depending on the required resolution and accuracy of the printing procedure and application.

It is worth highlighting that the proposed local and global metrics enable quantitative evaluation of the printed filaments from both continuous (i.e., global) and element-wise (i.e., local) perspectives, which are used for characterization and tuning of bioprinting parameters of the proposed robotic system.

III. Evaluation Experiments and Results

To thoroughly evaluate the performance of the proposed bioprinting system (shown in Fig. 1) while using a specific printing bioink (hydrogel), we first need to obtain the accuracy of the used measurement system and the proposed complementary segmentation and thickness evaluation framework (illustrates in Fig. 4). Knowing the performance of the measurement framework, we can characterize the important parameters affecting the bioprinting procedure [16], shown in Fig. 7, including the robot velocity $v_r(mm/s)$, extrusion rate $r_e(\mu L/s)$, and the distance between the needle tip and printing surface $D_{tip}\left(mm\right)$. This analysis together with the proposed quantitative evaluation metrics will aid us in determining the optimal combinations of autonomous bioprinting parameters that result in a uniform and consistent printed filament. The following sections describe these steps in detail.

Fig. 7.

Figure defines the considered important bioprintig parameters including the robot velocity $‘v_r’(mm/s)$, extrusion rate ${‘r}_e’(\mu L/s)$, and the distance between the needle tip and printing surface $‘D_{\mathit{\text{tip}}}’\left(mm\right)$.

A. Measurement Accuracy Validation

To achieve the above-mentioned aims, using the setup shown in Fig. 1, we first evaluated measurement system’s accuracy (i.e., the Zivid Two M70, Zivid Inc.) and the proposed complementary segmentation and thickness evaluation framework. Aside from the described bioprinting robotic system, the key component of this setup is a precise and customized 3D printed (Raise3D E2, Raise3D) calibration board containing a series of grooves with different widths of $1mm,2mm$, and $3mm$, identical lengths of $90mm$, and depths of $2mm$. Of note, the known width of grooves was used to evaluate the accuracy of the measurement framework. Considering the sensitivity of the utilized 3D structured light visual measurement system (i.e., Zivid Two M70, Zivid Inc.) to the light reflection at the injury site, after evaluating various filament colors, we realized that any filament colors with high contrast (e.g., red, blue, and black) can be used in our bioprinting and evaluation process. Of note, in real application, the utilized bioink can readily be dyed with these colors. Therefore, without loss of generality, a blue-dyed ultrasound gel (Aquasonic 100, Parker Laboratories Inc.) was used to simulate the printing bioink and conduct the accuracy validation experiments. As illustrated in Fig. 1, to perform the experiments, the grooves of the calibration phantom were first carefully filled with the printing ultrasound gel. Next, the 3D point cloud of the calibration board was acquired using the Zivid Camera attached to the robotic manipulator. Of note, this scan was obtained when the camera had the recommended minimum distance of 300 mm to the calibration board. The obtained point cloud was then processed by the proposed measurement framework described in Section II-B to calculate the calibration board grooves thickness. With the known ground-truth thickness of the grooves, as the desired filament thickness $f_t^{\mathit{\text{des}}}$, we used the error metrics defined in (6) and (7) with $N=140$ (i.e., element length of 0.66 mm) to evaluate the performance of the measurement framework. Notably, we selected this element length after performing various experiments and analyzing the results. An element length larger than this resulted in missing important information and thickness uniformity due to the large discretization, while smaller length unnecessarily increased computational time and processing without enhancing the results. Moreover, for each ground-truth thickness value, we also performed a comparison study for SPR by comparing the performances between adding sub-pixel level refinement (w/ SPR), and only using the pixel-level segmentation map (w/o SPR).

Table I summarizes the average and standard deviation of the Overall Global Error $E_g$ and the Overall Uniformity Error $E_u$ between the measured and ground truth thickness of the grooves. Note that in Table I, each experiment was repeated three times and the global error was calculated while the desired thickness $f_t^{\mathit{\text{des}}}$ (i.e., the known grooves’ thickness) was set as the corresponding ground truth values for both cases of “w/ SPR” and “w/o SPR”. A comparison between “w/ SPR” and “w/o SPR” cases are visualized in Fig. 5, where smoothed boundaries can be observed after the refinement.

TABLE I.

The average and standard deviation of the Overall Global Error $E_g$ and the Overall Uniformity Error $E_u$ between the measured and ground truth thickness of the grooves. ‘SPR’ denotes the sub-pixel-level refinement.

Ground truth	Method	${\mathit{E}}_{\mathit{g}}\left(mm\right)$	${\mathit{E}}_{\mathit{u}}\left(mm\right)$
$3mm$	w/ SPR	0.11±0.05	0.03±0.02
	w/o SPR	0.23±0.11	0.13±0.02
$2mm$	w/ SPR	0.10±0.06	0.04±0.02
	w/o SPR	0.42±0.09	0.11±0.03
$1mm$	w/ SPR	0.12±0.05	0.02±0.01
	w/o SPR	0.41±0.08	0.13±0.03

B. Quantitative Characterization of Bioprinting Parameters

After validating the accuracy of the measurement framework, to determine the optimal combinations of important autonomous bioprinting parameters [16] (i.e., robot velocity $v_r$, extrusion rate $r_e$, and the distance between the needle tip and printing surface $D_{tip}$) that result in a uniform and consistent printed filament, we performed a set of autonomous bioprinting experiments. For these experiments, we used the mentioned bioprinting system with 18-gauge needle and printed filaments with 50 mm length using the blue-dyed ultrasound gel (Aquasonic 100, Parker Laboratories Inc.) on a planar surface. It is worth highlighting that the obtained optimal combination of printing parameters through this process can readily be used while reconstructing a flat or 3D non-planar geometries. This is achievable due to the fact that any complex 3D non-planar geometry and shape can be reconstructed through a layer by layer printing of planar constructs and slices of the original geometry (similar to the procedure performed in the current commercialized 3D printing machines). Particularly, to find the optimal combination of the bioprinting parameters, we first performed experiments using the following values suggested in the literature (e.g., [19], [32], [33]): $v_r\in \left\{\mathrm{1,2},\mathrm{3,4},\mathrm{5,6}\right\}mm/s,r_e\in \left\{\mathrm{2.4,3.2,4},\mathrm{8,12}\right\}\mu m/L,D_{tip}\in \left\{\mathrm{0.8,1.8,2.8}\right\}mm$. We repeated each experiment three times and reported the average value for each metric. In total we performed 90 experiments. After performing these experiments, we used the validated measurement system and the proposed metrics, defined in (6) and (7) with $N=80$, to find the printing combinations that lead to the best printing performance (i.e., resulting in the minimum global uniformity errors). We selected $N=80$ to have a consistent element length of 0.66 mm with the one selected during the measurement accuracy validation step, described in the previous section. Analysis of each printed filament took less than 60s. Of note, from now on, we use the index ${\mathit{\text{Case}}}_{v_r,r_e}^{D_{\mathit{\text{tip}}}}$, to represent a specific autonomous bioprinting experiment corresponding to the selected bioprinting parameters $v_r,r_e$ and $D_{\mathit{\text{tip}}}$.

The case-wise characterization results are illustrated in Fig. 8 using the stacked bar charts, where the overall global error $E_g$ and the overall uniformity error $E_u$ are calculated for each combination of $v_r,r_e$, and $D_{\mathit{\text{tip}}}$. For each stacked bar chart, smaller values for $E_g$ and $E_u$ represent a high accuracy and high uniformity for the printed filaments, respectively. Additionally, Fig. 9 qualitatively and quantitatively visualizes a few exemplary success and failure cases for the printed filaments that are segmented using the algorithm described in Section II-B. Moreover, this figure demonstrates the corresponding normalized continuous element-wise local $e_l^i$ and uniformity $e_u^i$ errors along the printed filament geometry. Using min-max normalization [34], local $e_l^i$ and uniformity $e_u^i$ errors for each case were scaled to [0, 1] values and represented using heat maps. The heat maps clearly indicate locations along the printed filaments experiencing a non/uniform printing as compared with a desired printing thickness. Particularly, Figs 9(a–c) shows the continuous errors for ${\mathit{\text{Case}}}_{\mathrm{4,8}}^{2.8},{\mathit{\text{Case}}}_{\mathrm{1,2.4}}^{2.8}$ and ${\mathit{\text{Case}}}_{\mathrm{6,12}}^{1.8}$ from which large global errors and low uniformity can be observed. Figures 9(d–f) also show the failure cases with disconnected printed filaments. Nevertheless, Figs. 9(g–i) visualize several well-performed cases indicating similar performances and overall global errors in the case-wise characterization (Fig. 8). The overall global errors $E_g$ for ${\mathit{\text{Case}}}_{\mathrm{3,3.2}}^{1.8},{\mathit{\text{Case}}}_{\mathrm{5,4}}^{0.8}$ and ${\mathit{\text{Case}}}_{\mathrm{4,3.2}}^{2.8}$ are $0.34mm,0.47mm$, and $0.22mm$, respectively, which their performance is difficult to be visually differentiated. Overall, we select ${\mathit{\text{Case}}}_{\mathrm{4,3.2}}^{2.8}$ as the the optimal combination of parameters with the smallest $E_g$ of $0.23mm$ and $E_u$ of $0.02mm$. More details about the calculated errors for the success cases are also summarized in Table II. Overall, this table together with Fig. 8 and Fig. 9 quantify the importance of the selected autonomous bioprinting combinations in finding the optimal parameters that result in a locally- and globally-uniform bioprinting filament.

Fig. 8.

The bar charts illustrate the overall global error $E_g$ (top row) and the overall uniformity error $E_u$ (bottom row) for the considered combination of the bioprinting parameters (i.e., $v_r,r_e$, and $D_{\mathit{\text{tip}}}$). Yellow, green, and gray bars denote overall global error, overall uniformity error, and failure (discontinuous) cases, respectively.

Fig. 9.

Figure qualitatively and quantitatively visualizes a few exemplary success and failure cases for the printed filaments that are segmented using the algorithm described in Section II-B. Moreover, this figure demonstrates the corresponding normalized continuous element-wise local $e_l^i$ and uniformity $e_u^i$ errors along the printed filament geometry using heat maps.

TABLE II.

The measured overall errors ($E_g\text{and}{E}_u$), maximum local and uniformity errors ($e_l^{\mathit{\text{max}}}\text{and}{e}_u^{\mathit{\text{max}}}$) for the exemplary success cases shown in Fig. 9.

Case	${\mathit{E}}_{\mathit{g}}\left(mm\right)$	${\mathit{E}}_{\mathit{u}}\left(mm\right)$	${\mathit{e}}_{\mathit{l}}^{\mathit{\text{max}}}$	${\mathit{e}}_{\mathit{u}}^{\mathit{\text{max}}}$
${\mathit{\text{Case}}}_{\mathrm{4,8}}^{2.8}$	1.12	0.08	1.99	0.31
${\mathit{\text{Case}}}_{\mathrm{1,2.4}}^{2.8}$	2.01	0.10	2.45	0.28
${\mathit{\text{Case}}}_{\mathrm{6,12}}^{1.8}$	1.23	0.04	1.63	0.21
${\mathit{\text{Case}}}_{\mathrm{3,3.2}}^{1.8}$	0.28	0.02	0.47	0.15
${\mathit{\text{Case}}}_{\mathrm{5,4}}^{0.8}$	0.43	0.03	0.61	0.09
${\mathit{\text{Case}}}_{\mathrm{4,3.2}}^{2.8}$	0.23	0.02	0.40	0.15

IV. Discussion

As mentioned, uniform thickness and consistency of the bioprinted filaments play a fundamental role in scaffold formation during a bioprinting procedure. It is worth mentioning that non-uniformity in the filament’s thickness may result in a degraded biological function [24], [25]. The proposed novel autonomous in situ bioprinting surgical robotic framework can quantitatively evaluate the printing performance and find optimal parameters $\left(v_r,r_e\right.,\text{and}\left.D_{\mathit{\text{tip}}}\right)$ that minimize the global and local uniformity error metrics $\left(e_l,e_u,E_g\right.\text{and}\left.E_u\right)$ defined in this study. Combined with the 2D/3D computer vision techniques, the developed system enables a systematic quantitative characterization method for a generic robotic bioprinting system that can be used with any bioprinting materials with higher repeatability as compared with the existing conventional visual and qualitative evaluations (e.g., [18]–[20]).

As can be observed in Table I, applying the proposed SPR approach considerably reduced the global and uniformity errors for all ground truth thicknesses with maximum errors of $E_g=0.12mm$ (for 1 mm groove thickness) and $E_u=0.04mm$ (for the 2 mm groove thickness). On the contrary, solely using pixel-level segmentation maps (i.e., “w/o SPR” cases) is erroneous with maximum errors of $E_g=0.42mm$ (for the 2 mm groove thickness) and $E_u=0.13mm$ (for 1 mm groove thickness). Overall, results demonstrate that our proposed 3D visual measurement framework is capable of robustly measuring the thickness of printed filaments with acceptable global errors of $E_g<130\mu m$ and $E_u<40\mu m$, which, as opposed to the previous literature (e.g., [18], [20]), is critical for online quantification of the quality of the bioprinting constructs.

Through performing 90 quantitative characterization experiments, shown in Fig. 8), it was confirmed that filament printing quality highly depends on the chosen values for $v_r,r_e$, and $D_{\mathit{\text{tip}}}$. As can be observed in Fig. 8(b), the minimum errors was obtained at ${\mathit{\text{Case}}}_{\mathrm{4,3.2}}^{2.8}$ with the $E_g=0.22mm$ and $E_u=0.02mm$. From Fig. 8, it can be seen that, large $V_r$ and small $r_e$ would increase the risk of printing a discontinuous filament (i.e., a failure case). From Fig. 8(a), it can be seen that, cases with various $D_{\mathit{\text{tip}}}$ shared similar trends where all of them reached minimum errors at the combinations near $v_r=3~4mm/s$ and $r_e=3.2\mu L/s$. Also, results show that an increased $v_r$ leads to printing disconnected filaments or failure if a slow extrusion rate $r_e$ (i.e., $\le 4\mu L/s$) has been chosen for the printing procedure. These results, therefore, clearly highlight the importance of the performed experiments in finding the optimal parameters for resulting in a high-quality printing constructs.

Aside from the reported global uniformity metrics in Fig. 8, analyzing element-wise local $e_l^i$ and uniformity $e_u^i$ errors, reported in Fig. 9 and Table II, would also provide us with extra insight to select the optimal parameters that lead to a uniform printed constructs. For instance, as shown in Fig. 9(d–f), the considered local analysis and error metrics clearly demonstrate the location of disconnection or non-uniformity along the printed filament. Moreover, as shown in Table II and Fig. 9(a) and Fig. 9(c), for cases with similar $E_g\text{s}$, such as ${\mathit{\text{Case}}}_{\mathrm{4,8}}^{2.8}$ and ${\mathit{\text{Case}}}_{\mathrm{6,12}}^{1.8}$ (with $E_g=1.12mm$ and $E_g=1.23mm$, respectively), a larger maximum local $e_l^{max}\left(1.99mm\right)$ and uniformity $e_u^{max}\left(0.31mm\right)$ errors can be observed in ${\mathit{\text{Case}}}_{\mathrm{4,8}}^{2.8}$, indicating the importance of analyzing local uniformity metrics.

In other words, to optimally find the autonomous printing parameters, all of the proposed local and global error metrics need to be considered and simultaneously minimized to ensure a high-quality bioprinting procedure. Notably, when cases result in similar local and global error metrics, the speed of printing procedure, determined by $v_r$, needs to be considered to determine the optimal printing parameters, as it determines the overall time of printing procedure. For instance, comparing ${\mathit{\text{Case}}}_{\mathrm{3,3.2}}^{1.8}$ with ${\mathit{\text{Case}}}_{\mathrm{5,4}}^{0.8}$, despite similar values and distributions for the global and local errors (shown in Fig. 8 and Fig. 9), ${\mathit{\text{Case}}}_{\mathrm{5,4}}^{0.8}$ presents a faster $v_r$, resulting in a shorter time for the printing procedure, which is of interest for the surgical application of volumetric muscle loss repair. Overall, thanks to the proposed systematic quantitative evaluation and parameter selection approach, the performed experiments can readily be repeated for any kind of bioinks and range of bioprinting parameters to optimally select them based on the proposed novel metrics.

V. Conclusion and Future Works

In this study, we propose design and development of a unique autonomous in situ robotic bioprinting system for treatment of VML injuries with a generic bioprinting material. We also introduced complementary and novel computer vision algorithms, processes, and novel metrics to quantitatively evaluate the performance of this robotic system in printing uniform filaments. Using these metrics, we performed 90 experiments to characterize and identify optimal bioprinting parameters and minimize the proposed uniformity error metrics to ensure the biological functionality of a bioprinted construct [24], [25]. Results demonstrated that the proposed measurement framework is able to robustly measure the thickness of the printed filaments with a global errors of $E_g<120\mu m$ and $E_u<40\mu m$. Also, with the aid of introduced error metrics, we could obtain the optimal combinations of the printing parameters associated with the ${\mathit{\text{Case}}}_{\mathrm{4,3.2}}^{2.8}$ that resulted in the global uniformity error metrics of $E_g=230\mu m$ and $E_u=20\mu m$ for a printed filament.

Despite all the progress made in this paper, the developed robotic system and quantitative characterization framework is solely a fundamental step towards developing a more advanced system for treatment of VML injuries and has various limitations. For instance, the developed characterization algorithms solely focused on the evaluation of the filament thickness printed on a flat surface. In the future, we will evaluate the proposed framework in printing filaments on non flat and 3D geometries. As mentioned, any complex 3D non-planar geometry and shape can be reconstructed through a layer by layer printing of planar constructs and slices of the original geometry (similar to the procedure performed in the current commercialized 3D printing machines). Further, to enhance the performance of the proposed algorithm and towards performing real-time characterization and evaluation of bioprinting procedure, in the future and as we are collecting more data, we plan to use advanced machine learning algorithms. Moreover, to ensure safety of procedure and consider potential patient motion and/or tissue deformation during the bioprinting procedure, we plan to utilize a navigation and optical tracking system. Using this sensing system, we will implement advanced registration and calibration algorithms together with closed-loop control techniques performed on the robotic system to ensure an accurate and safe bioprinting procedure. Moreover, based on the developed bioink and its characterization, we will perform more realistic bioprinting studies on live animal models.

Biographies

Shuojue Yang received the B.Sc. degree in mechanical engineering from the University of Electronic Science and Technology of China, Chengdu, China, in 2020, and the M.S.E. degree in robotics from the Johns Hopkins University, Baltimore, USA, in 2022. In 2023, he joined the Walker Department of Mechanical Engineering and the Advanced Robotic Technologies for Surgery (ARTS) Lab at the University of Texas at Austin. He is currently working toward the Ph.D. degree with the Department of Biomedical Engineering, National University of Singapore, Singapore. His current research interests include surgical robotics, 3D surgical scene reconstruction, and artificial intelligence in healthcare.

Hansoul Kim received his B.S. degree from the Division of Robotics at Kwangwoon University, Seoul, South Korea, in 2017. He earned his M.S. degree from the Robotics Program at the Korea Advanced Institute of Science and Technology (KAIST), Daejeon, South Korea, in 2019. Subsequently, he attained his Ph.D. degree in the Department of Mechanical Engineering from KAIST in 2022. From 2022 to 2023, he served as a Post-doctoral researcher at the Walker Department of Mechanical Engineering and Texas Robotics, the University of Texas at Austin, TX, USA. Currently, he holds a position as a Post-doctoral researcher in the Department of Electrical Engineering and Computer Sciences at the University of California, Berkeley, CA, USA. His research interests primarily focus on surgical robotics, soft robotics, automation, and learning.

Omid Rezayof received his B.Sc. in Mechanical Engineering in 2023 from Sharif University of Technology, Tehran, Iran. Currently, he is pursuing his Ph.D. in Walker Department of Mechanical Engineering, University of Texas at Austin and Texas Robotics. His research contains developing robots, systems and algorithms for an in-situ robotic bioprinting framework, with interests in robotics, dynamics and control.

Jeff Bonyun received a B.Commerce degree from the University of Alberta, Canada, in 1999 and a M.B.A. degree from the Harvard Business School in 2003. He is currently completing a M.S.E. degree in Mechanical Engineering from the University of Texas at Austin, where he studies robotics. His research focus has been motion control for robot-assisted surgery and algorithms for vision-based tactile sensors.

Johnson V. John is an Assistant Professor at the Terasaki Institute for Biomedical Innovation. He was born in Kerala, India. He received his bachelor’s degree in chemistry from Kannur University and his master’s in biochemistry from Bharathidasan University, India. Then, he finished his Ph.D. in “Stimuli-Responsive Synthetic Polymer Bioconjugate Hybrid Nanocarriers for Controlled Drug Delivery” under the guidance of Prof. Il Kim at Pusan National University, Republic of Korea. He joined Prof. Carlo. D. Montemagno’s Lab as a postdoctoral Research Associate at the University of Alberta, Canada (2016–2018). exthe joined Jingwei Xie’s lab at the University of Nebraska Medical Center as a postdoctoral Research Associate (2018–2021). Since 2021, he has been one of the core faculty members at the Terasaki Institute for Biomedical Innovations. Dr. Johnson and his team are mainly focusing on the development of fibers reinforced scaffolds and modular biomaterials for various tissue engineering applications.

Mehmet R. Dokmeci received B.S. (with distinction) and M.S. degrees from the University of Minnesota, Minneapolis and the Ph.D. degree from the University of Michigan, Ann Arbor, all in Electrical Engineering. Dr. Dokmeci is an Associate Professor at the Terasaki Institute for Biomedical Innovation (TIBI) since July 2020. He was previously an Associate Adjunct Professor in the Radiology Department at University of California-Los Angeles from November 2017 to June 2020. He was a former Instructor at Brigham and Women’s Hospital, Harvard Medical School, a position he has held for 7 years. Previously, he was on the faculty of the Electrical and Computer Engineering Department at Northeastern University. Before joining academia, he had 4 years of industrial experience at Corning-Intellisense Corporation, Wilmington, MA, developing MEMS-based products for the telecommunications and life science industries from 2000 to 2003. He has extensively published in refereed journals and conferences in the areas of BioMEMS, Micro- and Nanotechnology, and Organs-on-a-chip. He is also a long-term member in the Institute of Electrical and Electronics Engineers (IEEE), Materials Research Society (MRS), American Chemical Society (ACS) and American Association for the Advancement of Science (AAAS). He has a strong background and more than two decades of experience in designing and fabricating micro and nanoscale sensors and systems, biomedical devices, tissue engineering and implantable biosensors. He has been actively involved in research areas such as flexible electronics for monitoring and modulation of wound healing, electrical/electrochemical biosensors and microfluidic systems for organs-on-a-chip applications and 3D bioprinting. He has authored more than 200 journal papers, 5 patents/disclosures; 112 conference publications/abstracts, has an h-index=88, citations > 31,700.

Ali Khademhosseini is currently the CEO and Founding Director at the Terasaki Institute for Biomedical Innovation. Previously, he was a Professor of Bioengineering, Chemical Engineering and Radiology at the University of California-Los Angeles (UCLA). He joined UCLA as the Levi Knight Chair in November 2017 from Harvard University, where he was a Professor at Harvard Medical School and faculty at the Harvard-MIT’s Division of Health Sciences and Technology, Brigham and Women’s Hospital, and associate faculty at the Wyss Institute. At Harvard University, he directed the Biomaterials Innovation Research Center, a leading initiative in making engineered biomedical materials. Dr. Khademhosseini has served as an Associate Editor for ACS Nano from 2013 to 2021. He is a fellow of the American Institute of Medical and Biological Engineering (AIMBE), Biomedical Engineering Society (BMES), Royal Society of Chemistry (RSC), Biomaterials Science and Engineering (FBSE), Materials Research Society (MRS), NANOSMAT Society, and American Association for the Advancement of Science (AAAS). He is also the recipient of the Mustafa Prize ($500,000 prize) and is a member of the International Academy of Medical and Biological Engineering, Royal Society of Canada, Canadian Academy of Engineering, and National Academy of Inventors. He is an author of ¿700 peer-reviewed journal articles, editorials, review papers, ¿70 book chapters/edited books, and ¿50 patents/patent applications. He has been cited ¿126,700 times and has an H-index of 182. He has made seminal contributions to modifying hydrogels and developing novel biomaterial solutions for addressing pressing problems in healthcare. He has founded 3 companies, Obsidio Medical, which was recently acquired by Boston Scientific, OMEAT, and Biorae. He received his Ph.D. in bioengineering from MIT (2005), and MASc (2001) and BASc (1999) degrees from the University of Toronto, both in chemical engineering.

Farshid Alambeigi is currently an Assistant Professor at the Walker Department of Mechanical Engineering at the University of Texas at Austin since August 2019. He is also one of the core faculties of the Texas Robotics. Dr. Alambeigi received his Ph.D. in Mechanical Engineering from the Johns Hopkins University, in 2019. He also holds an M.S. degree (2017) in Robotics from the Johns Hopkins University. He received his B.Sc. and M.Sc. degrees in Mechanical Engineering from K. N. Toosi University of Technology (2009) and Sharif University of Technology (2012) in Iran, respectively. Dr. Alambeigi is the recipient of several prestigious awards including the 2019 SIEBEL Scholarship Award, the NIH NIBIB Trailblazer Career Award (2021), and the NIH Director’s New Innovator Award (2022). At The University of Texas at Austin, Dr. Alambeigi directs the Advanced Robotic Technologies for Surgery (ARTS) Lab. His research focuses on developing high dexterity and situationally aware continuum manipulators, soft robots, and intelligent instruments especially designed for less/minimally invasive treatment of various medical applications.