3D digital breast cancer models with multimodal fusion algorithms

Abstract

Breast cancer image fusion consists of registering and visualizing different sets of a patient synchronized torso and radiological images into a 3D model. Breast spatial interpretation and visualization by the treating physician can be augmented with a patient-specific digital breast model that integrates radiological images. But the absence of a ground truth for a good correlation between surface and radiological information has impaired the development of potential clinical applications.

A new image acquisition protocol was designed to acquire breast Magnetic Resonance Imaging (MRI) and 3D surface scan data with surface markers on the patient’s breasts and torso. A patient-specific digital breast model integrating the real breast torso and the tumor location was created and validated with a MRI/3D surface scan fusion algorithm in 16 breast cancer patients.

This protocol was used to quantify breast shape differences between different modalities, and to measure the target registration error of several variants of the MRI/3D scan fusion algorithm. The fusion of single breasts without the biomechanical model of pose transformation had acceptable registration errors and accurate tumor locations. The performance of the fusion algorithm was not affected by breast volume. Further research and virtual clinical interfaces could lead to fast integration of this fusion technology into clinical practice.

Highlights

• •MRI/3D surface scan fusion algorithm to create 3D breast cancer models.
• •A replicable clinical validation protocol for MRI/3D surface scan fusion algorithms.
• •Anthropometric study that quantifies breast deformations by area in MRI and 3D scans.

1. Introduction

Since mammography has emerged as a viable technology in the 1960s, the technical advances in breast imaging have been significant over the last decades, with new modalities like breast ultrasound, magnetic image resonance (MRI), digital breast tomosynthesis and elastography [1]. Innovations in 3D imaging and digital high-tech industrial science have accelerated experimental advances in medical imaging. Recently, medical image fusion has been developed by combining multiple images from multiple imaging modalities to augment image quality and to increase clinical applicability [2,3]. The implicit image fusion and spatial interpretation of the treating physician or the patient can be subjective and prone to error if performed without a patient-specific digital breast model integrating the real breast torso and preoperative tumor location. A breast digital model could be very useful in diverse clinical practice settings like the operating theatre scenario by increasing the accuracy of tumor location through augmented reality in breast cancer conservative surgery [4]. But there is no ground truth to assess the validity and reliability of breast image fusion algorithms that combine radiological and surface data of the breast. To extrapolate meaningful clinical data from breast 3D surface scan and MRI combined registration, the validity and reliability of breast image fusion algorithms need to be determined. The purpose of this paper is to validate the MRI/3D fusion algorithm from Bessa et al. [5] to create breast cancer 3D models and to approach clinical applications and technological constraints. The influence of the patient’s position on the morphology of the breast is quantitatively analyzed with an image acquisition protocol that acquires multi-modal images with breast surface markers and torso fiducial points visible on both the 3D surface scan and MRI data.

2. Patients and methods

Sixteen patients with a Tis/T1-T3 breast cancer proposed for breast conservative treatment at the Champalimaud Clinical Centre, Portugal, between April 2017 and January 2019 were assessed for inclusion in the current study. Contraindications included T4 cancers, inflammatory carcinoma and breast recurrence post-breast conservative treatment.

After written informed consent, all patients were proposed for image acquisition and anatomical measurements:

• 1.Surface data: 3D surface scans of the patient in the standing position with hands on hips, capturing the size and shape of the breasts using Go!Scan 3D by Creaform™^,2.
• 2.MRI (Philips Ingenia 3T) with gadolinium contrast was performed according to institutional protocols - patient in prone position with arms up.
• 3.A secondary protocol for validation of the MRI/3D scan fusion and quantitative analysis of breast morphology was applied to a subgroup of 7 patients which includes breast surface markers (BSM) on the patient’s breasts and torso [[6], [7], [8]] (Fig. 1). These reference points were annotated with a black permanent marker before surface data acquisition. After 3D surface scan, cod liver oil pills were fixed upon these marks for MRI acquisition (Fig. 2).
• 4.Annotation, segmentation and volume computation of the MRI tissue portions (with T1 and sd3 image sequences) were performed and validated by two radiologists using the Horos R software v2.4.0. (breast contour, breast tissue including malignant tumor(s), pectoral major muscle, latissimus dorsi muscle anterior border, sternum, clavicle). Breast densities were categorized and BSM were manually annotated on the 3D surface scan (black permanent marker) and MRI data (cod liver oil pills).

Fig. 1.

Reference points around the breast: four cardinal breast surface markers around the areola (inner circle, I1 – I4) and eight surface markers (outer circle, O1 – O4, Oa - Od) placed at a 30 mm distance from the areolar circle. SN - sternal notch, X - xiphoid, AC acromium, MP - medial point, LP - lateral point, AP - axillary point, MAP - mid-axillary point.

Fig. 2.

3D surface scan and MRI data with reference points.

2.1. Multimodal fusion algorithm

Medical data is multi-sourced: images are acquired in different instants of time and settings, inducing distinct deformations on the soft-tissue anisotropic breast. Previous research has focused primarily on the multimodal radiological registration [9,10], or 3D surface data reconstruction [11], while image registration between both modalities (breast surface and radiological information) is seldomly addressed [3].

The shape of the breast is unarguably different in the MRI — where radiological exams are commonly fused — compared to the 3D surface scan: there is a lack of shared natural landmarks apart from the nipples, pose related deformations are a major challenge (distinct patient/arms positioning) and the equipment of the MRI imposes variable breast compressions depending on the patient’s body mass index (BMI), torso width and breast size [12]. The difficulties in objectively quantifying the accuracy of MRI/3D scan fusion algorithms further impose barriers to the development and acceptance of these algorithms in clinical sets [13]: there is an absence of ground truth for validating surface/radiological fusion.

Despite extensive literature on 3D finite elements simulation or breast tissue modelling, few papers have addressed breast MRI/3D scan fusion [5,14,15]. All these works have in common limited clinical validation (Appendix A) and the use of breast biomechanical modelling: either integrating an iterative optimization of the unknown material parameters of the biomechanical model (BM) [14,15], or aggregating the BM with a step of non-rigid geometric registration such as the Free Form Deformation (FFD) [5]. Although BMs are helpful in pose estimation, they might be insufficient to recover all deformations [14]. The inclusion of non-rigid registration algorithms introduces extra degrees of freedom that allow fine adjustments, but when unconstrained they can lead to unrealistic deformations. Therefore, biomechanically informed registration algorithms have been shown to provide more accurate matches than registration using only BMs or non-rigid registration methods independently [16].

In this paper, breast MRI/3D scan fusion is accomplished with an improved version of the Bessa et al. [5] algorithm (Fig. 3) that includes the tumor: coarse alignment transformations are applied to the tumor points, which are further deformed on the FFD step - tumor is initially converted to the original grid of control points, and its shape and position is updated with the deformed grid resultant from the surface matching FFD. Four variants of the algorithm were implemented to explore the influence of:

• •integrating or not the BM (MRI_prone versus MRI_up);
• •single breast fusion (each breast independently) versus the fusion of the full torso.

Fig. 3.

MRI/3D scan fusion algorithm using the full torso: the MRI prone mesh is created from pairs of MRI T1 images and annotations, and the tumor location annotated on the MRI sd3 volume - expert annotations were used to avoid the bias of automatic segmentation errors on the fusion validation. The fusion algorithm can use the surface points from prone mesh directly, or from the upright mesh obtained after the BM [17]. [A] the orientation of MRI breast contours is corrected, [B] 3D scan and MRI point clouds are aligned - translation by breast mounds followed by rigid Iterative Closest Points algorithm, [C] FFD is applied to the translated MRI point clouds to recover remaining deformations. Steps [A] - [C] were also applied to the point cloud of the tumor. In [C] the point cloud of the tumor was converted to the initial local frame of control points and then modified with the deformed frame. For additional technical details, please refer to Ref. [5].

2.2. Quantitative analysis of breast morphology and algorithm validation

Despite generalized use of BMs in registration tasks there has never been, to the best of our knowledge, a quantitative analysis of the shape similarities between MRI breast contour (both in prone and simulated upright positions) and a 3D surface scan. Therefore, this work evaluates the influence of the patient positioning on the breast morphology by comparing the contour (geodesic) distances between pairs of BSM on the 3D scan, on the MRI_prone and on the MRI_up. BSM used for distances were manually annotated on the 3D scan and on the MRI data, establishing the correspondence between their centroids and the tags described in Fig. 1.

The reference points were marked on the MRI_prone mesh (before BM), enabling the precise location of those points on the MRI_upmesh (after BM). Fig. 4 shows the resulting reference points among modalities. The BSM were also used to evaluate the performance of the fusion algorithm: euclidean distances between pairs of corresponding BSM on the fused modalities were measured, establishing the Target Registration Error (TRE) of the algorithm. The positioning of the tumor in the 3D scan after the fusion algorithm was qualitatively evaluated by the surgeon, comparing tumor locations on front views with the pre-operative carbon tattooing localization paper.

Fig. 4.

Reference points used for validation and quantitative analysis of the breast morphology. Only reference points visible in both 3D scan and MRI are shown: a) and c) show the annotations (yellow) and centroids (red) of the BSM overlaps; b), d) and e) show the centroids and BSM tags on the 3D scan and on the MRI_prone and MRI_up, respectively. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

3. Results

3.1. Quantitative analysis of breast morphology

The breast morphometric measurements indicate distance differences (percentage of increase or decrease) between BSM on the 3D scan and on the MRI_prone and MRI_up (Fig. 5). In MRI_prone, the inframammary fold is visible, the breasts become more rounded and the upper poles are more filled and closer to the MAP (Fig. 4d)). Points on the lower poles also shifted closer to the MAP, as evidenced by the negative distance variance associated with MAP (Fig. 5a)). The lateral compression imposed by the MRI coils is visible for most patients (Fig. 6c)), with particular emphasis on patients with larger breasts. This also explains the increase in distances between inner and outer circle BSM on MRI_prone versus the 3D scan. The breast in MRI_up is more tear-shaped (Fig. 6), which is more similar to its appearance on the 3D scan. Breast lower poles have more volume than upper poles, and the BM simulates upper poles with a concave profile for all patients. Regardless of the visual similarities of the breast shape between the MRI_up and the 3D scan, the heatmaps show generally higher distance variances (stronger colours) compared to the MRI_prone. In MRI_up, the distance variance is particularly higher for the O1, O3 (vertical outer markers) and Od breast markers. The BM seems to elongate the breast shape on the axial direction, which justifies the increase on vertical distances (Fig. 4e)). Conversely, the distances to X and MAP are slightly more similar between the MRI_up/3D scan, than with the MRI_prone. There is also a decrease in the distances to the LP, counterbalanced by an increase in the distances to the MP: breasts are shifted laterally toward the outside, after the BM.

Fig. 5.

Breast morphometry: the percentage of distance variance - defined as the ratio of pairwise distances between BSM, normalized by their distances on the 3D scan [18]- is shown for pairs of BSM a) on the MRI_prone and b) on the MRI_up. Average results are shown for the 7 patients of the validation subgroup, with a total of 12 breasts with BSMs. Axis labels correspond to the BSM from Fig. 1.

Fig. 6.

MRI breast contour in prone position (gray) and after biomechanical pose transformation to upright (blue). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

It is noteworthy that the AC and the AP markers were not visible on any MRI from the validation protocol, and the SN appeared only on MRI images of two of the patients that had the largest breasts of the group. This enlightens that the MRI scans different portions of the torsos depending on the breast size. Moreover, the absence of AP even when the SN can be identified, evidences the effect of the up arms position during MRI acquisition: with arms up, APs become superior to the SN, and induce an upwards shift of the MAP and LP. The tumor location is also affected by the BM simulation (Fig. 6). Comparing to the MRI_prone: the tumor moves downwards and inwards on the axial and coronal directions, due to the elongation of the breast and upper pole loss of fullness. All BM simulations of the validation protocol were carried out with the actual breast density of the patient.

3.2. Fusion algorithm validation

Fig. 7 shows the superposition of the 3D scan and the MRI in prone and upright positions after the coarse alignment step using the full torso (Fig. 3 [C]). Although the overlap is more precise when using the MRI_up, the BM is still unable to recover all deformations between the two modalities: coil deformations are still visible (Fig. 7a)), the top profile of the breast is more concave (Fig. 7b)) and the lower pole does not cover completely its counterpart on the 3D scan (Fig. 7c)). Thus, upper pole tumors located near the skin might be aligned outside the breast on the fused MRI_up/3D scan, and lower pole tumors would appear more medially than expected.

Fig. 7.

MRI/3D scan fusion results after the coarse alignment step with full torso. Results are shown for variants using MRI in prone (white) and in upright (blue) positions. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Although the use of the BM combined with a rigid coarse alignment falls short on the MRI/3D scan fusion, it provides better adjustments when compared to fusion with the MRI_prone. Tumor positioning outside the 3D breast surface scan would be more frequent if only coarse alignment with MRI_prone was used. Fig. 7 c) also evidences the differences in the torso alignment using MRI_prone or MRI_up: the algorithm maximizes the superposition of points near the inframammary contour, causing the MRI_prone to be lower when compared with the MRI_up. The effect of BM elongating the breast on the axial direction also introduces an artifact in the fusion algorithm: knowing that APs did not appear on any MRI, one can argue that the MRI_up is wrongly aligned with the 3D scan, because its upper limit is above the armpit on the scan.

Figs. 8 and 9 show the superposition of 3D scan, MRI and tumor volumes after the FFD step (Fig. 3 [D]). As expected, the FFD compensates the shape dissimilarities remaining after the coarse alignment. FFD also adjusts the tumor position, but its location is mostly conditioned by the rigid registration step, which varies depending on the use of MRI_prone or MRI_up. The FFD also introduces artifacts on the boundaries of the MRI torsos: both MRI_prone and MRI_up are slightly compressed on the axial direction and lateral limits are curved and narrowed to favor breasts overlap. This explains why, even with full torso fusion and inclusion of the BM, medial points such as X and MP have lower TRE average (13.75 mm and 13.52 mm, respectively) than MAP and LP (> 30 mm, also caused by the upwards shift of these points induced by the arms in up position). Yet, one can argue that these artifacts are unimportant in light of the main purpose of the fusion algorithm: to provide an accurate tumor location on the 3D scan. Therefore, Table 1 lists the average TRE for all BSM positioned on top of the breast (inner and outer circles). Results are shown for variants using MRI_prone or MRI_up, and fusion with full torsos or single breast.

Fig. 8.

MRI/3D scan fusion results after the FFD step with full torso. Results are shown for variants using MRI in prone (white) and in upright (blue) positions. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 9.

Example of the four tumor locations shown to the surgeon for qualitative validation of the tumor location. Fusion results are shown for variants using a) full torso or b) single breasts, using MRI in prone (white) or MRI in upright (blue). Visible carbon marks on the 3D scan, emphasized with dashed red line). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Table 1.

Target Registration Error of the MRI/3D scan fusion algorithm, in millimeters: average and standard deviation results of all BSM on top of the breast (inner and outer circle reference points). The use of the BM to simulate the MRI in upright position results in smaller TRE, in comparison to the use of the breast contour from the original MRI in prone. Best results are obtained when breasts are registered independently (single breast) in opposition to register MRI and 3D scan torsos. The cod oil liver pills used in the study had an ellipsoide shape with a major axis of 10 mm.

	MRIprone	MRIup
Single Breast	26.26 ± 6.61	18.50 ± 3.88
Full Torso	34.21 ± 9.29	18.75 ± 5.32

Fig. 9 illustrates an example of the four resulting tumor positions shown to the surgeon to validate tumor location. Tumors were shown in random colours during the subjective analysis, for all sixteen patients (all with identified uni-lateral tumors at the moment of acquisition). The surgeon resorted to clinical reports, surgical annotations (Fig. 9c)) and carbon marks/tattoing (when visible) to decide on the best location. The fusion of single breasts resulted in better tumor locations, for all patients. Moreover, the exclusion of the BM from the algorithm resulted in better tumor locations in 80% of cases. Thus, despite the higher TRE error (26.26 ± 6.61 mm vs. 18.50 ± 3.88 mm), single breast fusion with MRI_prone provides better tumor location than the MRI_up counterpart.

Fig. 10 shows the single breast fusion TREs, distributed by BSM. The TRE by BSM varies differently according to their spatial location. In detail, single breast fusion with MRI_prone and MRI_up, share bigger TREs on the BSM: O2, O1, Ob (upper pole and medial points), while smaller TREs are found for I4 and I3 (lower pole). Single breast fusion with MRI_up also has small TREs on O4 and O3. This suggests that the fusion algorithm is more precise on the lower poles of the breast, and the BMs has a reduced impact on the distribution of the errors. Regarding the TRE by patient (Fig. 11), nonparametric Mann-Whitney.

Fig. 10.

Target Registration Error of the MRI/3D scan fusion algorithm, distributed by BSM and patients. Average lines are shown for global TRE (dashed blue), TRE by BSM (circle red) and the overall size of the cod liver oil capsules overlap areas on 3D scan and MRI. Bars represent TRE errors per patient. (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

Fig. 11.

Target Registration Error of the fusion algorithm by patient. Patients are sorted by breast volume (blue dashed line). (For interpretation of the references to colour in this figure legend, the reader is referred to the Web version of this article.)

U test at the 5% significance level suggest that there is not enough evidence of the single breast fusion algorithm dependence on the breast volume, either using MRI_prone or MRI_up. However, patients 6 and 7 have consistently higher TREs than the overall average (considerable deformations caused by the MRI coils, that persist after the BM of pose transformation, Fig. 12), which is particularly evident when the fusion algorithm uses MRI_prone. Fig. 10 a) shows that the other 5 patients have TRE closer to or lower than the average size of the cod oil pills overlap (9.88 mm). In fact, when comparing average TREs comprising only the other 5 patients, MRI_prone has slightly better results than MRI_up: 14.03 ± 5.68 mm versus 14.95 ± 7.57 mm, respectively. This provides an explanation on how the single breast fusion algorithm without BM can provide better tumor locations while having bigger overall TRE than the variant using the BM.

Fig. 12.

Large lateral breast deformation caused by MRI coil - patient 6.

4. Discussion

Breast image fusion technology applied to MRI images and 3D surface scan into a synchronized (3D) digital breast model, is an achievement towards precision medicine. Research from previous authors has focused on multimodal registration of mammography, breast ultrasound and MRI, but never with both clinical validation and 3D surface data integration in the upright position [3]. To the best of our knowledge, this is the first paper validating MRI/3D scan fusion algorithms to create patient-specific 3D breast models, from cancer patients. The fusion performance was evaluated with low cost BSM (cod liver oil pills) positioned on strategic anatomic points of breasts and torso, providing an objective ground truth to quantify breast morphology differences between 3D scan and MRI acquisitions (before and after fusion). Results are reported for sixteen patients included in the study, a number of samples that points out difficulties of obtaining clinical data with a new acquisition protocol, but is still above the average of subjects used in similar studies (Appendix A).

Breast image fusion algorithms often resort to BMs to recover pose related deformations or compressions induced by the imaging equipment. These algorithms tend to combine BMs with other strategies [16], such as iterative optimization of the BM parameters, but published papers are faulty on validation and do not objectively assert the effects of BMs on the spatial distribution of breast reference points. Additionally, these models are complex, require expert knowledge to create proper meshes for simulation and the material properties are not patient-specific [9].

Our results revealed that the high shape similarity between the MRI_up and the 3D surface scan hides BM artifacts. The BM increased the distance of BSM on the axial direction and changed the distribution of lateral distances by shifting the breast toward the outside of the torso. Moreover, the coarse alignment of MRI_up/3D scan further supported the inability of BM to recover all breast deformations and to provide proper tumor location/alignment of MRI/3D scan. As a consequence, a second stage of FFD was integrated to encompass all possible tumor locations and ensure a proper overlap of the fused modalities. The integration of the BM on the fusion pipeline and the hypothesis of fusing single breasts to increase tumor location accuracy were investigated. Results showed that both average TRE and tumor location were improved when breasts were fused independently.

Best TRE performance (18.5 ± 3.88 mm) was observed with the inclusion of the BM of pose transformation, but tumor locations were consistently worst (80% of the cases) compared to the fusion results without the BM (TRE of 26.26 ± 6.61 mm). The artifacts from the BM affected the coarse alignment stage and had a negative impact on the tumor location, as it tends to elongate the breast on the axial direction. Both variants of the fusion algorithm have TREs around 20 mm: the cod liver oil pills used on this study add an ellipsoide shape with a major axis of 10 mm, some pills detached from the surface of the patient during MRI acquisition, and the skin/pills overlap varied according to the BSM position. The use of smaller pills (low cost) or commercial breast adhesives can reduce this technical limitation.

Tumor location accuracy increased without the BM and it seems not to be affected by breast volume variances in the subgroup control (7 patients with BSM), making it suitable for automation and clinical sets. MRI breast acquisition in the supine position (after the normal MRI acquisition protocol, the patient would change position to supine) should be considered to improve algorithm performance on tumor location, despite expected artifacts from breast motion due to respiration may lead to image quality variances.

After further fusion technology development, feasibility trials will enhance clinical applications in different scenarios. The spatial computing applied to this dataset, merging digital and physical anatomic structures of the breast (with tumor) into a digital 3D model, can pave the way for the next generation breast surgeon, with the integration of augmented reality into the operation theater, allowing direct tumor visualization inside the patient’s breast. The concept of uploading a patient specific breast cancer 3D model to an augmented reality glasses in the operating theater as a non-invasive pre-operative tumor localization technique has already been addressed [19] but never attempted. The proposed validation protocol and the 3D model simulations performed are a first step to bring breast cancer computer vision into clinical use cases.

5. Conclusion

This paper provides valuable insights on breast MRI/3D surface scan fusion. The proposed validation protocol is adequate for morphometric studies and provides a ground truth to compare different fusion algorithms. Results suggest that it is worthy to explore fusion algorithms without the use of BM, depending on the main goal: tumor location had acceptable accuracy with the fusion algorithm without the BM, which is a more suitable alternative for automation and clinical sets. Further research is needed to objectively quantify tumor location accuracy. Additionally, the acquisition of a bigger sample of data with the validation protocol can boost the development of statistically and machine learning alternatives to predict pose transformation and breast compressions.

Funding

This work was supported by the ERDF–European Regional Development Fund through the Norte Portugal Regional Operational Programme (NORTE 2020), under the PORTUGAL 2020 Partnership Agreement and through the Portuguese National Innovation Agency (ANI) as a part of project BCCT.Plan–NORTE-01-0247-FEDER-01768 and also by Fundação para a Ciência e a Tecnologia (FCT) within Ph.D grant number SFRH/BD/115616/2016.

Declaration of competing interest

None.

Appendix A.

Table A.2.

Summary of relevant contributions on the creation of 3D breast cancer models, with task description, type of publication and the number of samples used in the study.

Author	Description	Type of Publication	Samples
Eiben et al. [14]	3, 1 for MRI/scan fusion	Registration of prone/supine MRI, MRI and 3D scan in upright	International Conference
Salmon et al. [15]	1	Registration MRI and 3D scan in upright	International Conference
Bessa et al. [5]	7	Registration MRI and 3D scan in upright	International Conference
Duraes et al. [19]	9	Registration MRI and 3D scan in supine	Journal
Lee et al. [16]	5	Registration MRI prone and supine	International Conference
Vavourakis et al. [17]	4	Biomechanical modeling of pose transformation - MRI	Journal
Reece et al. [18]	5	Anthropometric study of pose induced breast deformations - 3D surface data only	Journal