A boundary-guided transformer for measuring distance from rectal tumor to anal verge on magnetic resonance images

Summary

Accurate measurement of the distance from the tumor’s lowest boundary to the anal verge (DTAV) provides an important reference value for treatment of rectal cancer, but the standard measurement method (colonoscopy) causes substantial pain. Therefore, we propose a method for automatically measuring the DTAV on sagittal magnetic resonance (MR) images. We designed a boundary-guided transformer that can accurately segment the rectum and tumor. From the segmentation results, we estimated the DTAV by automatically extracting the anterior rectal wall from the tumor’s lowest point to the anal verge and then calculating its physical length. Experiments were conducted on a rectal tumor MR imaging (MRI) dataset to evaluate the efficacy of our method. The results showed that our method outperformed surgeons with 6 years of experience (p < 0.001). Furthermore, by referring to our segmentation results, attending and resident surgeons could improve their measurement precision and efficiency.

Highlights

• •An automatic method for measuring DTAV from sagittal MR images
• •A boundary-guided transformer for simultaneously segmenting rectum and tumor
• •A dataset consisting of 212 sagittal MR images with rectal tumor
• •The method shows great potential to help surgeons accurately estimate DTAV

The bigger picture

The distance from a rectal tumor’s lowest boundary to the anal verge (DTAV) is critical for determining the treatment plan for rectal cancer. However, the currently used DTAV measurement method, i.e., colonoscopy, is invasive and will cause pain to the patient, while manual measurement on the magnetic resonance (MR) image is time consuming, and its accuracy highly depends on the surgeon’s experience. In this work, we transform the DTAV measurement on an MR image as an image segmentation task and propose a boundary-guided transformer for DTAV measurement method. Our method can segment both the rectum and tumor well and estimate the DTAV accurately from the patient’s MR image. Moreover, under the guidance of our segmented rectum and tumor, both attending and resident surgeons can improve their DTAV measurement accuracy and efficiency significantly. In the future, our method can be deployed in clinical scenarios to help surgeons design optimal treatment plans for patients with rectal cancer.

Achieving accurate distance from the tumor’s lower boundary to the anal verge (DTAV) measurements is important for designing optimal treatment plans for patients with rectal cancer. Through exploiting an advanced transformer-based method, we transformed the manual measurement of DTAV to an automatic one for the first time. The proposed automatic DTAV measurement method is fast and accurate and can provide surgeons with reliable DTAV references. Moreover, our intermediate segmentation results can assist surgeons in improving their own DTAV measurement accuracy and efficiency.

Introduction

As one of the most fatal malignant tumors, rectal cancer is a major health-related burden worldwide, and the incidence of cases diagnosed annually is increasing.^1,2 Locally advanced rectal cancer (LARC) (clinical tumor stage 3–4 and/or positive clinical lymph nodal stage) is the most common form.³ Compared with colon cancer, the treatment of rectal cancer is more complex because of the intricacy of rectal anatomy and tumor-involving structures. According to the European Society for Medical Oncology’s clinical practice guidelines, rectal cancer is defined as tumors arising within 15 cm of the anal verge, which should receive preoperative radiation and chemotherapy to achieve acceptable locoregional control and increase the likelihood of anal sphincter preservation.⁴ Surgical approaches, with (low anterior resection) or without anal sparing (abdominoperineal resection), for rectal cancer are determined by the distance from the tumor’s lowest boundary to the anal verge (DTAV).⁵ Furthermore, previous studies have suggested that pathologic response to neoadjuvant chemoradiotherapy (nCRT),^6,7 perioperative complications, and oncological outcomes^8,9,10 of rectal cancer are highly correlated with tumor location. The DTAV plays a decisive role throughout the treatment plan for rectal cancer and is ultimately useful for individualizing treatments,^11,12 as shown in Figure 1C.

Figure 1.

Flowcharts of the two current DTAV measurement methods and the roles of DTAV during rectal cancer treatment

(A) DTAV measurement through colonoscopy.

(B) DTAV measurement through manual annotation on 2D sagittal MR image. In this MR image, the tumor region, lowest verge of tumor, and anal verge are pointed at by the white asterisk, white arrowhead, and white arrow, respectively. In addition, the DTAV is measured as the length of the white dotted line.

(C) Illustration of the roles of DTAV during rectal cancer treatment.

Currently, colonoscopy is the preferred method for measuring the DTAV.⁴ However, bowel preparation and colonoscopy are often painful and time consuming. Therefore, in clinical practice, the DTAV is frequently measured using digital rectal examinations, transanal ultrasound, computed tomography (CT), and magnetic resonance imaging (MRI). Among these, the rectal cancer protocol MRI is a noninvasive and nonionizing diagnostic method. This method provides excellent soft-tissue resolution and precise anatomical structure, and it is widely used for tumor localization and local tumor staging.^7,10 Nevertheless, to recognize tumors and measure the DTAV precisely, surgeons require years of training. In addition, the measurement accuracy of surgeons with varying experience levels may differ significantly, potentially leading to different treatment strategies.¹³ Overall, the commonly used measurement methods have their own disadvantages, such as the invasive nature of colonoscopy,¹⁴ time-consuming manual measurements by surgeons via CT or MRI, dependence on surgeon experience, and subjectivity to physician error.¹⁵ Automatically measuring the DTAV based on rectal and tumor image segmentation may be a potential method for rapid and accurate DTAV measurements. Thus, our goal is to develop an effective method to automatically measure the DTAV in specific patients to help surgeons obtain reference DTAV measurements, avoid human bias in measurements, and improve the efficiency of the surgeon’s manual measurements.

The key to realizing automatic DTAV measurements is to accurately locate both the rectum and tumor. Only sagittal MR images could present the rectum, tumor, and anus in the same image, which enables the direct measurement of the DTAV. Moreover, surgeons often measure the DTAV on sagittal MR images,^13,16 as shown in Figure 1B. Therefore, in this work, we aim to develop an automatic DTAV measurement method based on the sagittal MR images. However, in different MR scans of the sagittal view, the tumor and rectum vary in shape and size, and the location of the tumor on the rectum also varies. Thus, it is challenging to accurately localize the rectum and tumor both from sagittal MR images.

Target localization in an image is generally modeled as object detection or semantic segmentation. Object detection can only locate the outline of the target, whereas semantic segmentation can identify the entire target region by assigning a semantic label for each pixel of a given image. Semantic segmentation methods can be divided into two categories: traditional and deep learning-based methods. Traditional segmentation methods exhibit extremely limited performance in rectal segmentation.^17,18,19 Therefore, the current segmentation of rectal cancer uses a method based on deep learning, primarily based on convolutional neural networks (CNNs).^20,21,22,23 CNNs can extract local details well and play an important role in numerous biomedical imaging applications such as segmentation of the small intestine from MRI,²⁴ identification of lung adenocarcinoma from CT,²⁵ and localization of tumors on mammograms.²⁶ However, pure CNN-based methods generally exhibit limitations in modeling explicit long-range relations because of the intrinsic locality of convolution operations.²⁷ Transformers have emerged as alternative architectures that entirely dispense with convolution and rely solely on attention mechanisms.²⁸ Unlike previous CNN-based methods, transformers are powerful in modeling global contexts through imposing attention on all image patches.²⁹ The transformer imposes an attention operator on all patches to capture the localization information of the target regions, but it only focuses on modeling the global context at all stages, so it is challenging to use the vanilla transformer in tasks with pixel-level outputs.³⁰ Hybrid CNN-transformer architecture can leverage both detailed high-resolution spatial information from CNN features and the global context encoded by transformers so as to achieve superior results in various fields (e.g., image synthesis,³⁰ image reconstruction,³¹ and image denoising³²). In particular, hybrid CNN-transformer architecture has achieved excellent results in the image segmentation tasks.^27,33,34 For example, the Cerberus transformer utilizes both the CNN’s local features and multiple scales of the transformer’s global features so as to achieve an advanced level in the scene semantic segmentation task.³³ TransUNet is composed of transformer and UNet, and thus has their advantages both and could achieve excellent performance in the image segmentation task.²⁷ Therefore, the hybrid CNN-transformer-based method has great potential for achieving success in segmenting rectal tumor. However, to our knowledge, no method has been developed for rectal tumor segmentation. Moreover, in general, the DTAV is measured, prior to treating the patient with rectal tumor, from their sagittal MR images, in which boundaries of both the rectum and tumor are relatively clear.³⁵ In addition, the boundary information is often exploited to improve the performance of CNN-based image segmentation methods,^36,37 in particular by supervising the boundary accuracy of the segmentation results. Based on the above investigations, developing a method embedded with both hybrid CNN-transformer architecture and boundary information may be an effective way to achieve the purpose of simultaneously segmenting the rectum and the tumor from the patient’s sagittal MR image.

Therefore, in this study, we have developed a method for automatically measuring the DTAV on the sagittal MR images.³⁸ The proposed method comprises two stages. In the first stage, the rectum and tumor regions are simultaneously segmented from the sagittal MR images by using a hybrid CNN-transformer-based model. In the second stage, the DTAV is estimated through automatically measuring the length of the anterior rectal wall between the lowest verge of the tumor and the anal verge. At present, there is no public medical image dataset containing both rectum and tumor, but simultaneous localization of rectum and tumor is the basis for achieving the purpose of automatic DTAV measurements. Thus, we have built a sagittal MR dataset containing both the rectum and tumor in particular and will make this dataset publicly available to facilitate developing the automatic DTAV measurement methods.³⁹ Overall, the major contributions of this study can be concluded as follows.

• (1)We propose a boundary-guided transformer that can simultaneously identify the rectum and tumor regions from the sagittal MR images of patients with rectal cancer.
• (2)We propose an automatic method to measure the DTAV based on our segmented rectum and tumor, and the measurement accuracy of our method exceeds that of the attending surgeon.
• (3)A sagittal MR rectal image dataset was constructed for the field of DTAV measurement. The DTAV dataset will be made available at publication to encourage more advanced research on this topic.
• (4)Extensive experiments demonstrate that the surgeons can significantly improve their precision in DTAV measurements and reduce their measurement time by referring to our segmentation results. This indicates that our method has great potential to facilitate the surgeons in their clinical practices.

Results

Ablation results

The ablation results have been listed in Table 1. The experimental results for verifying the efficacy of each module or block of our boundary-guided transformer (BGTR) are elaborated as follows.

Table 1.

Quantitative results of baseline models incorporated with different components

Method	ACC (%)	PRE (%)	SEN (%)	SPE (%)	DICE (%)
Baseline	98.28 ± 0.93 (97.80 ± 0.79, 98.70 ± 61.13)	76.68 ± 13.84 (76.75 ± 13.56, 76.60 ± 23.04)	73.56 ± 10.46 (74.06 ± 20.85, 73.05 ± 10.31)	99.19 ± 0.38 (98.97 ± 0.58, 99.41 ± 0.67)	73.38 ± 9.54 (74.28 ± 16.67, 72.47 ± 8.54)
Baseline+FFM	98.60 ± 0.93 p = 0.008 (98.22 ± 0.80 p = 0.003, 98.97 ± 1.11 p < 0.001)	79.42 ± 8.72 p = 0.007 (80.22 ± 12.90 p = 0.038, 78.62 ± 18.59 p = 0.170)	79.75 ± 11.95 p < 0.001 (82.15 ± 23.59 p < 0.001, 77.34 ± 9.71 p = 0.075)	99.30 ± 0.38 p = 0.081 (99.06 ± 0.51 p = 0.169, 99.54 ± 0.70 p = 0.116)	78.25 ± 12.83 p < 0.001 (80.03 ± 20.86 p < 0.001, 76.47 ± 8.98 p = 0.016)
Baseline+FFM+BG	98.65 ± 1.08 p = 0.293 (98.34 ± 0.99 p = 0.198, 98.96 ± 1.22 p < 0.001)	81.98 ± 9.55 p = 0.293 (81.03 ± 11.27 p = 0.985, 82.92 ± 17.67 p = 0.366)	80.06 ± 12.68 p = 0.859 (83.94 ± 19.50 p = 0.256, 76.17 ± 9.81 p = 0.251)	99.33 ± 0.53 p = 0.266 (99.08 ± 0.71 p = 0.956, 99.57 ± 0.66 p < 0.001)	79.95 ± 11.10 p = 0.571 (81.81 ± 16.55 p < 0.001, 78.08 ± 8.66 p = 0.956)
Baseline+FFM+BG+FFB (BGTR)	98.75a ± 0.78 p = 0.892 (98.43a ± 0.65 p = 0.861, 99.07a ± 1.01 p = 0.900)	82.68a ± 7.97 p = 0.931 (81.48a ± 12.11 p = 0.726, 83.88a ± 15.61 p = 0.944)	81.60a ± 10.00 p = 0.685 (85.15a ± 17.68 p = 0.393, 78.05a ± 8.99 p = 0.564)	99.38a ± 0.35 p = 0.921 (99.11a ± 0.32 p = 0.863, 99.64a ± 0.64 p < 0.001)	81.04a ± 9.09 p = 0.726 (82.51a ± 14.38 p < 0.001, 79.57a ± 8.56 p = 0.645)

Metrics (e.g., ACC, PRE, SEN, SPE, DICE) are represented as “mean of rectum and tumor segmentation result ± standard deviation (rectum segmentation result ± standard deviation, cancer segmentation result ± standard deviation).” p values were obtained by comparison of the metrics (e.g., ACC, PRE, SEN, SPE, DICE) of the network before adding the module (e.g., FFM, BG, FFB) with the network after adding the module (e.g., FFM, BG, FFB) with the metrics (e.g., ACC, PRE, SEN, SPE, DICE) of BGTR. Statistics were performed by Mann-Whitney U test.

^aThe best results in the current column.

Feature fusion module (FFM)

The results in the second and third rows show that the baseline model with an FFM improved over that without an FFM by 0.0032 accuracy (ACC), 0.0274 precision (PRE), 0.0619 sensitivity (SEN), 0.0011 specificity (SPE), and 0.0487 Dice coefficient (DICE) for mean segmentation accuracy of both rectum and tumor. From the aspect of mean segmentation accuracy of rectum, the baseline model with an FFM outperformed that without an FFM by 0.0042 ACC, 0.0347 PRE, 0.0809 SEN, 0.0009 SPE, and 0.0575 DICE, respectively. As for the mean segmentation accuracy of the tumor, the FFM boosted the performance of the baseline model by 0.0027 ACC, 0.0202 PRE, 0.0429 SEN, 0.0013 SPE, and 0.0400 DICE, respectively.

Boundary-guided block (BG)

The results in the third and fourth rows show that the baseline model with a BG improved over that without a BG by 0.0005 ACC, 0.0256 PRE, 0.0031 SEN, 0.0003 SPE, and 0.0170 DICE in terms of mean segmentation accuracy of both rectum and tumor. From the aspect of mean segmentation accuracy of rectum, the baseline model with a BG improved ACC, PRE, SEN, SPE, and DICE by 0.0012, 0.0081, 0.0179, 0.0002, and 0.0178, respectively, compared with that without a BG. As for mean segmentation accuracy of tumor, the baseline model with a BG decreased ACC and SPE by 0.0001 and 0.0117, respectively, but improved PRE, SEN, and DICE by 0.0430, 0.0003, and 0.0161, respectively, compared with that without a BG.

Feature fusion block (FFB)

The results in the fourth and fifth rows indicate that our complete image segmentation method (i.e., BGTR) outperformed that excluding an FFB by 0.0010 ACC, 0.0070 PRE, 0.0154 SEN, 0.0005 SPE, and 0.0109 DICE, respectively, in terms of the mean segmentation accuracy of both rectum and tumor. From the aspect of mean segmentation accuracy of rectum, our BGTR improved ACC, PRE, SEN, SPE, and DICE by 0.0009, 0.0045 0.0121, 0.0003, and 0.0007, respectively, compared with that without an FFB. Considering the mean segmentation accuracy of tumor only, our BGTR improved over that without an FFB by 0.0011 ACC, 0.0096 PRE, 0.0188 SEN, 0.0070 SPE, and 0.0149 DICE, respectively.

Visual comparison of segmentation results

Figure 2 illustrates the three sets of segmentation results of the fully convolution network (FCN),⁴⁰ UNet,²⁰ GatedSCNN,³⁶ TransUNet,²⁷ Swin transformer (Swin-T),³⁴ Cerberus transformer,³³ and BGTR. The first row of Figure 2 shows the segmentation results of an image of a rectal tumor with a curved tail and a tumor with a hole. The segmentation results of the FCN, UNet, and GatedSCNN are very poor, and they can only roughly locate the tumor and rectum. TransUNet can roughly segment the rectum and tumor but is insufficient to identify the tail of the rectum and the hole in the tumor. Cerberus and Swin-T have the problem of oversegmentation of tumors, but they can segment the rectum well. The rectum and tumors segmented by BGTR are consistent with the ground truth; they can segment curved rectal tails and holes in tumors.

Figure 2.

Visual comparison of segmentation results of different methods

In the two leftmost columns, three pairs of rectal MR image and corresponding ground-truth segmentation results are shown, respectively. The following seven columns show the results of the seven compared image segmentation methods, respectively. In the three MR images from top to bottom in the leftmost column, there exist a tumor with a hole, a large tumor, and a discontinuous tumor, respectively. In the three ground-truth images in the second column, the rectum region, tumor region, and background region are annotated in white, blue, and black, respectively.

The second row of Figure 2 shows the segmentation results from the MR image of a large tumor in the rectum. The segmentation results of the FCN, UNet, GatedSCNN, TransUNet, and Swin-T are very poor. Cerberus can segment the rectum and tumor well; however, there are discrete points in the results. Our BGTR method can accurately identify tumors and segment the rectum.

The third row of Figure 2 shows the segmentation results of an MR image with a discontinuous tumor and curved rectum. The rectum segmented by the GatedSCNN has holes, and Swin-T cannot segment the curved part of the rectum. The rectum segmented using other methods had a good fit. The FCN only segments part of the tumor on the left, and the UNet segments only part of the tumor on the right. Compared with FCN and UNet, GatedSCNN can detect more tumor regions, but the segmented tumor is still insufficiently accurate. TransUNet, Swin-T, and Cerberus roughly segment the tumor, but the segmentation details are insufficient. Only BGTR accurately identified discontinuous tumors.

Quantitative comparison of segmentation results

The quantitative evaluation results of the seven image segmentation methods have been listed in Table 2.

Table 2.

The quantitative results of different image segmentation methods

Method	ACC (%)	PRE (%)	SEN (%)	SPE (%)	DICE (%)
FCN38	97.72 ± 1.38 p < 0.001 (97.08 ± 1.17 p < 0.001, 98.35 ± 16.77 p < 0.001)	68.71 ± 16.10 p < 0.001 (68.69 ± 15.44 p < 0.001, 68.72 ± 26.89 p < 0.001)	71.75 ± 15.66 p < 0.001 (73.28 ± 25.78 p < 0.001, 70.22 ± 14.42 p = 0.373)	98.74 ± 0.86 p < 0.001 (98.39 ± 1.14 p < 0.001, 99.09 ± 1.09 p = 0.013)	66.88 ± 15.45 p < 0.001 (68.26 ± 22.64 p < 0.001, 65.50 ± 11.87 p < 0.001)
UNet20	97.74 ± 1.37 p < 0.001 (97.02 ± 1.17 p < 0.001, 98.46 ± 1.82 p = 0.001)	72.16 ± 13.23 p < 0.001 (66.17 ± 16.60 p < 0.001, 78.15 ± 20.46 p = 0.347)	70.95 ± 16.31 p < 0.001 (73.80 ± 25.78 p < 0.001, 68.09 ± 16.99 p = 0.044)	98.80 ± 0.78 p < 0.001 (98.31 ± 1.14 p < 0.001, 9.29 ± 1.01 p = 0.654)	67.22 ± 16.27 p < 0.001 (67.99 ± 22.64 p < 0.001, 66.45 ± 15.17 p < 0.001)
GatedSCNN36	98.05 ± 1.16 p < 0.001 (97.58 ± 0.89 p < 0.001, 98.52 ± 1.61 p = 0.0012)	74.33 ± 9.10 p < 0.001 (76.70 ± 13.93 p < 0.001, 71.96 ± 16.52 p < 0.001)	72.38 ± 12.82 p < 0.001 (70.72 ± 21.68 p < 0.001, 74.04 ± 14.33 p = 0.658)	99.09 ± 0.49 p < 0.001 (99.01 ± 0.75 p = 0.480, 99.17 ± 0.71 p < 0.001)	70.01 ± 10.90 p < 0.001 (72.12 ± 15.14 p < 0.001, 67.90 ± 11.40 p < 0.001)
TransUNet27	98.02 ± 1.23 p < 0.001 (97.44 ± 1.05 p < 0.001, 98.60 ± 1.51 p = 0.010)	73.76 ± 13.69 p < 0.001 (71.02 ± 16.00 p < 0.001, 76.50 ± 23.91 p = 0.347)	72.03 ± 14.46 p < 0.001 (78.06 ± 27.12 p < 0.001, 65.99 ± 10.29 p = 0.013)	98.96 ± 0.71 p < 0.001 (98.45 ± 0.70 p < 0.001, 99.74 ± 1.12 p = 0.636)	70.09 ± 14.56 p < 0.001 (72.93 ± 23.60 p < 0.001, 67.25 ± 11.33 p = 0.001)
Swin-T34	98.00 ± 1.22 p < 0.001 (97.46 ± 1.06 p < 0.001, 98.54 ± 1.46 p = 0.002)	69.98 ± 11.80 p < 0.001 (70.80 ± 15.29 p < 0.001, 69.15 ± 21.00 p < 0.001)	79.43 ± 12.41 p = 0.472 (78.72 ± 21.52 p < 0.001, 80.14 ± 11.76 p = 0.010)	98.77 ± 0.65 p < 0.001 (98.44 ± 0.71 p < 0.001, 99.09 ± 1.12 p < 0.001)	72.44 ± 12.43 p < 0.001 (72.89 ± 18.3 p < 0.001, 71.99 ± 11.68 p = 0.005)
Cerberus33	98.61 ± 1.07 p = 0.645 (98.26 ± 0.90 p = 0.623, 98.96 ± 1.32 p = 0.888)	82.94a ± 9.35 p = 0.507 (81.73a ± 14.73 p = 0.417, 84.14a ± 16.93 p = 0.759)	78.26 ± 15.32 p = 0.654 (81.32 ± 25.76 p = 0.024, 75.20 ± 10.82 p = 0.606)	99.39a ± 0.44 p = 0.253 (99.16a ± 0.50 p = 0.320, 99.61 ± 0.72 p = 0.610)	78.22 ± 14.40 p = 0.806 (80.32 ± 20.94 p = 0.428, 76.12 ± 11.05 p = 0.985)
BGTR (ours)	98.75a ± 0.78 (98.43a ± 0.65, 99.07a ± 1.01)	82.68 ± 7.97 (81.48 ± 12.11, 83.88 ± 15.61)	81.60a ± 10.00 (85.15a ± 17.68, 78.05a ± 8.99)	99.38 ± 0.35 (99.11 ± 0.32, 99.64a ± 0.64)	81.04a ± 9.09 (82.51a ± 14.38, 79.57a ± 8.56)

Quantitative results (i.e., ACC, PRE, SEN, SPE, DICE) are represented as “mean segmentation accuracy of both rectum and tumor ± standard deviation (mean segmentation accuracy of rectum ± standard deviation, mean segmentation accuracy of tumor ± standard deviation).” In each row, the p value was obtained by comparing the metric value of our method and that of the current method with the Mann-Whitney U test.

^aThe best results in the current column.

Among the CNN-based methods, UNet and GatedSCNN achieved better segmentation results than FCN in most cases. From the aspect of the mean segmentation accuracy of rectum and tumor, our BGTR outperformed UNet by 0.0101 ACC, 0.1052 PRE, 0.1065 SEN, 0.0058 SPE, and 0.1382 DICE, respectively, and outperformed GatedSCNN by 0.0070 ACC, 0.0835 PRE, 0.0922 SEN, 0.0029 SPE, and 0.1103 DICE, respectively. Considering the mean segmentation accuracy of rectum only, BGTR improved over UNet by 0.0141 ACC, 0.1531 PRE, 0.1135 SEN, 0.0080 SPE, and 0.1452 DICE, respectively, and over GatedSCNN by 0.0085 ACC, 0.0478 PRE, 0.1143 SEN, 0.0001 SPE, and 0.1039 DICE, respectively. Moreover, our method outperformed UNet by 0.0061 ACC, 0.0573 PRE, 0.0996 SEN, 0.0035 SPE, and 0.1312 DICE, respectively, and obtained improvements by 0.0055 ACC, 0.1192 PRE, 0.0401 SEN, 0.0047 SPE, and 0.1167 DICE compared with GatedSCNN when only considering the mean segmentation accuracy of tumor.

Among the transformer-based methods, Cerberus performed the best, achieving an ACC of 0.9861, PRE of 0.8294, SEN of 0.7826, SPE of 0.9939, and DICE of 0.7822. Compared with Cerberus, our method decreased by 0.0026 PRE and 0.0001 SPE, but improved by 0.0014 ACC, 0.0334 SEN, and 0.0282 DICE, in terms of the mean segmentation accuracy of rectum and tumor. As for the mean segmentation accuracy of rectum, our method underperformed Cerberus by 0.0025 PRE and 0.0005 SPE but outperformed Cerberus by 0.0014 ACC, 0.0334 SEN, and 0.0282 DICE. Finally, in terms of the mean segmentation accuracy of tumor, our proposed method achieved improvements in ACC, SEN, SPE, and DICE by 0.0011, 0.0285, 0.0003, and 0.0345, respectively, and decreased in PRE by 0.0026 compared with Cerberus.

Comparison of the proposed DTAV measurement method and surgeons’ measurement methods

Colonoscopies revealed that the median tumor height from the anal verge was 5.85 cm (range: 2–15.0 cm). The results of our study and those of different surgeons are presented in Table 3. From Figure 3, the median of absolute differences between colonoscopy measurements and other methods’ measurements (i.e., |Col-Ours|, |Col-Chief1|, |Col-Chief2|, |Col-Attending1|, |Col-Attending2|, |Col-Resident1|, and |Col-Resident2|) were 0.93, 0.83, 0.59, 1.32, 0.94, 1.45, and 1.67, respectively. Compared with |Col-Attending1|, |Col-Attending2|, |Col-Resident1|, and |Col-Resident2|, the median of absolute differences between our method and colonoscopy is closer to 0, indicating a better performance of our method. In addition, the interquartile ranges of |Col-Ours|, |Col-Chief1|, |Col-Chief2|, |Col-Attending1|, |Col-Attending2|, |Col-Resident1|, and |Col-Resident2| were 1.26, 1.15, 1.17, 1.58, 1.56, 2.01, and 1.99, respectively. Compared with |Col-Attending1|, |Col-Attending2|, |Col-Resident1|, and |Col-Resident2|, the interquartile range of |Col-Ours| was smaller.

Table 3.

DTAV measurement achieved by colonoscopy and those manually achieved on sagittal MR images by two chief surgeons, two attending surgeons, two resident surgeons, and our method

Method	Median DTAV (range)
Colonoscopy	5.85 (2.00–15.00)
Chief1	5.91 (2.13–11.89)
Chief2	5.47 (2.17–11.00)
Attending1	5.50 (2.36–11.35)
Attending2	5.48 (2.64–10.05)
Resident1	4.64 (2.03–10.06)
Resident2	5.56 (1.82–9.88)
Ours	6.55 (0.74–16.03)

Unit: cm.

Figure 3.

Scatterplot of differences between colonoscopy’s measurement and other seven methods’ measurements

The intraclass correlation coefficient was calculated to investigate the clinical utility of the proposed method. Our method, Chief1, and Chief2 achieved good reliability of 0.87, 0.81, and 0.7874, respectively, while our method showed the highest performance compared with the chief surgeons, although the differences were not statistically significant. The intraclass correlation coefficient (ICC) was moderate for the attending surgeons (Attending1: 0.6177; Attending2: 0.5173), and poor for the resident surgeons (Resident1: 0.2448; Resident2: 0.3535); both performances were significantly worse than our method; detailed results are shown in Table 4. Furthermore, with the exception of the chief surgeons (r = 0.9542), the interobserver consistency of the attending and resident surgeons did not perform well (r = 0.2695 and r = 0.6818, respectively) (Table 5).

Table 4.

ICCs between the measurement result of the colonoscopy method and those of other methods

Assessment	ICC (95% CI)	p value
Chief1, colonoscopy	0.8100 (0.6680 to 0.8911)	0.2981
Chief2, colonoscopy	0.7874 (0.6310 to 0.8777)	0.1613
Attending1, colonoscopy	0.6177 (0.3304 to 0.7812)	0.002
Attending2, colonoscopy	0.5173 (0.1595 to 0.7229)	0.0001
Resident1, colonoscopy	0.2448 (−0.2833 to 0.5602)	<0.0001
Resident2, colonoscopy	0.3535 (−0.1257 to 0.6288)	<0.0001
Ours, colonoscopy	0.8700 (0.7669 to 0.9266)	–
Ours + Attending1, colonoscopy	0.7760 (0.6092 to 0.8715)	0.1129
Ours + Attending2, colonoscopy	0.7532 (0.5696 to 0.8584)	0.0395
Ours + Resident1, colonoscopy	0.7759 (0.6117 to 0.8709)	0.0001
Ours + Resident2, colonoscopy	0.7822 (0.6226 to 0.8746)	0.0006

p values of the first six rows were obtained by comparing the ICC of (ours, colonoscopy) and those of other six combos with Z tests, respectively. p values of the last four rows were obtained by comparing the ICC of surgeon and colonoscopy with the ICC of ours + corresponding surgeon and colonoscopy with Z tests, respectively. ICC, intraclass correlation coefficient; CI, confidence interval.

Table 5.

Correlation coefficients of the measurement results of two surgeons with same seniority

DTAV measurement methods	r (95% CI)	p value
Chief1, Chief2	0.9542 (0.9211 to 0.9736)	<0.0001
Attending1, Attending2	0.2695 (−0.004 to 0.5053)	0.0533
Resident1, Resident2	0.6818 (0.5024 to 0.8049)	<0.0001
Ours + Attending1, ours + Attending2	0.9373 (0.8927 to 0.9637)	<0.0001
Ours + Resident1, ours + Resident2	0.9553 (0.9231 to 0.9742)	<0.0001

p values were obtained by Spearman correlation analysis to test the interobserver consistency between two surgeons with same seniority. r, correlation coefficient; CI, confidence interval.

Assistance of our DTAV measurements to surgeons

To identify our method’s ability to assist surgeons (denoted as ours + surgeons), we compared the surgeons’ performance regarding our assisted measurement and others’ measurement contexts. Overall, according to Table 4, our method’s influence in assisting the attending surgeons and resident surgeons was substantially consistent. The ICC between attending surgeons and resident surgeons for colonoscopy measurements showed a significant improvement (0.776 vs. 0.6177; 0.7532 vs. 0.5173; 0.7759 vs. 0.2448; 0.7822 vs. 0.3535) (Table 4). The interobserver consistency of attending surgeons and resident surgeons also performed better when assisted by our method (0.9373 vs. 0.2695 and 0.9553 vs. 0.6818, respectively), values of which were all statistically significant (Table 5). In addition, our method is fast, measuring in approximately 9.7 s per patient, and with the assistance of our method, resident and attending surgeons can significantly reduce the time taken in measuring the DTAV (Table 6).

Table 6.

Time cost comparison between surgeons and ours + surgeons

Method	Time cost (s)	p value
Attending1	81	<0.0001
Ours + Attending1	45	–
Attending2	90	<0.0001
Ours + Attending2	48	–
Resident1	138	<0.0001
Ours + Resident1	56	–
Resident2	127	<0.0001
Ours + Resident2	51	–

p values were obtained by comparing the time cost of surgeon’s measurement (without ours+) process and that of the surgeon’s measurement (with ours+) process referring to our segmentation results. Note that the time cost of each method for processing all testing images was not distributed normally, thus the statistical comparison was conducted by the Mann-Whitney U test.

Discussion

In this study, we pioneered a method for DTAV measurement and transformed it from manual measurement to automatic measurement. The contribution of our method can be summarized 2-fold: (1) we propose a hybrid CNN-transformer-based image segmentation network that involves a boundary-guided module to accurately identify the rectum and tumor regions both from the sagittal MR images, and (2) we develop an automatic DTAV measurement method based on the segmented rectum and tumor regions by automatically extracting the anterior rectal wall (between anal verge and the lowest verge of the tumor) and then calculating the physical length of the extracted anterior rectal wall. Specifically, because the rough location of segmentation targets is relatively stable in the sagittal MR images, a network with the capability of localizing this rough location is preferred. Compared with translation-invariant convolutional networks, the global patch attention used in the vision transformer can well serve this goal. Moreover, we introduce a boundary-guided module that promotes boundary awareness of the transformer, and this module can make good use of the boundary information of MR images. These two design principles collaborate to achieve accurate segmentation. Then, we automatically extracted the anterior rectal wall between the lowest verge of the tumor and the anal verge to achieve automatic DTAV measurements based on the segmentation result. The proposed method can help doctors obtain reference DTAV measurements, reduce measurement error rates, and increase measurement speed. Especially in situations where qualified surgeons are not readily available, the measurement results obtained by our method are comparable to those obtained by an attending surgeon. Surgeons can refer to our intermediate results to improve measurement accuracy, thus demonstrating great potential for application in clinical scenarios.

Performance of the semantic segmentation network

BGTR is a boundary-guided transformer for rectal and tumor segmentation. Based on the characteristics of the MR images, we added a boundary-guided module to better segment the rectum and tumors. As shown in Figure 2, it can be concluded that our method can effectively deal with various problems (such as irregular rectum, discontinuous tumor, and tumor with holes) and accurately segment both the rectum and tumor regions. In addition, the segmentation results from the second to the eighth column imply that the guidance of the boundary information and the hybrid CNN-transformer structure are both effective for the task of rectum and tumor segmentation. Because of these two merits, our method is able to segment both rectum and tumor with greater accuracy and outperform other state-of-the-art image segmentation methods. The ablation study results in Table 1 demonstrate the effectiveness of the main building blocks of our network. The usage of a hybrid CNN-transformer structure and the introduction of a boundary-guided module improve the performance for segmenting both rectum and the tumor. As shown in Table 2, compared with CNN-based methods (i.e., FCN, UNet, and GatedSCNN), we use a transformer structure with the advantage of extracting global information to better localize the rectum and tumor. Compared with the pure transformer method (i.e., Swin-T), our method is a hybrid CNN-transformer structure because the segmentation results are not fine enough only using the transformer, and the hybrid CNN-transformer structure combines the advantages of CNN and transformer to obtain better segmentation results. Compared with the existing hybrid CNN-transformer-based methods (i.e., TransUNet and Cerberus), which do not exploit boundary information, we introduce the boundary-guided module, and the boundary-guidance module only deals with boundary information, which can further improve the segmentation performance of the network by making better utilization of the boundary information in MR images. Note that there is no significant difference between our method and Cerberus in the statistical sense. The mean metric values of our method are better than those of Cerberus in most cases; meanwhile, the standard deviations of the metric values of our method are smaller than those of Cerberus. These facts can imply that the proposed method performs slightly better than Cerberus. Overall, this success is credited to the ability of the transformer to globally localize tumor regions by self-attention on all patches and the boundary-guided module, which only focuses on processing relevant boundary-related information. This design enables our network to exploit the transformer’s natural localization advantages; it takes full advantage of the border information of the rectum and the tumor and generates more accurate predictions, significantly improving the segmentation performance on discontinuous tumor and tumors with holes and an irregular rectum.

Performance of our DTAV measurement method

Currently, there are still controversies regarding the manual measurement of DTAV on MRI. Attenberger et al.,⁴¹ in a retrospective study of 108 patients with rectal cancer, reported that MRI measurement can be an alternative to colonoscopy measurement. However, Keller et al.⁴² have shown significant variations between MRI and colonoscopy measurements. Some possible causes may have contributed to this discrepancy. First, different landmarks were chosen to report the distal margin of rectal cancer. Second, there was a mixed level of awareness regarding the DTAV among the investigators. In this study, we propose a highly efficient anterior rectal wall extraction method based on automatic segmentation by a radiologist experienced in pelvic MRI. As shown in Table 4 and Figure 3, our measurement accuracy was much higher than that of resident and attending surgeons. In addition, the results indicated poor internal consistency in measurements between the attending and resident groups. However, with the proposed method, surgeons can improve the measurement accuracy, interobserver agreement, and efficiency, as shown in Tables 5 and 6. Taken together, the proposed automated measurement method offers reliable assistance to surgeons and is a good teacher for inexperienced surgeons; it also standardizes the measurement of the DTAV.

Limitations of the study and future work

Although the proposed method is able to achieve the level of manual measurements performed by highly experienced surgeons, it measures the DTAV only from 2D MR images rather than on complete 3D MR images, leading to relatively low measurement precision compared with the colonoscopy measurement method. To further approach the precision of the colonoscopy method, an important future work is to build a 3D MR or CT rectal image dataset and further develop automated DTAV measurement methods based on the 3D image dataset. Besides, the proposed BGTR was trained and tested only on rectal MR images with both the rectum and tumor. Thus, our method might produce false segmentation results when the input MR image does not contain a rectum or tumor. This limitation could be solved by adding negative samples during BGTR training or by equipping a classification branch with BGTR in particular to recognize whether the input MR image contains both the rectum and tumor. Finally, the current training dataset is still not that sufficient. Thus, we will further build a larger-scale dataset and utilize it to improve the applicability of our algorithm in the clinical scenarios.

Method

In this section, we describe the proposed DTAV measurement method in detail. The proposed method comprises two main stages. In the first stage, we segmented the rectum and tumor regions simultaneously using a BGTR. Inspired by Takikawa et al.,³⁶ we construct BGTR by adding a boundary-guidance module to a conventional transformer. In the second stage, we extracted the anterior rectal wall from the lowest point of the tumor to the anal verge according to the segmented rectum and tumor. The DTAV can then be calculated by accumulating the Euclidean distances of every two adjacent pixels located on the extracted anterior rectal wall. In the following subsections, the dataset and evaluation metrics are first introduced, and then two stages of the proposed method are introduced.

DTAV data

We retrospectively collected 212 2D sagittal MR images from the MR image scans of 212 patients with LARC from the Peking University Third Hospital.³⁹ In particular, we only collected one image from the MR image scans of each patient. More specifically, the study included 152 males and 60 females with an average age of 59.5 years who underwent radical surgery. The cohort of patients used to test the image segmentation methods and DTAV measurement methods included 42 men and 12 women (ages: 26–81 years, mean age: 59.5 years). The cohort of patients used to train the image segmentation methods included 78 men and 40 women (ages: 22–82 years, mean age: 59.5 years). The cohort of patients used to validate the image segmentation methods included 32 males and 8 females (ages: 22–80 years, mean age: 59 years). The Ethics Review Board of Peking University Third Hospital approved this study (IRB00006761-M2019387) as required by the Helsinki Declaration. The institutional review board of the Third Hospital of Peking University waived the requirement for informed consent.

As part of the inclusion criteria, the following items were considered: (1) the diagnosis of rectal cancer was histopathologically confirmed; (2) all patients were diagnosed with LARC (cT3-4/N0-2/M0) by a multidisciplinary team; (3) the patients were 18–85 years of age; and (4) a standard report was provided for all patients following a colonoscopy. The exclusion criteria were (1) insufficient MR image quality to obtain measurements (due to motion artifacts), (2) incomplete patient clinical information, and (3) additional cancers besides the primary rectal cancer.

Rectal MR images were scanned from each patient (before receiving neoadjuvant chemoradiation) by a 3.0 T MRI machine (Discovery MR750; GE Medical Systems, Waukesha, WI, USA) according to departmental protocol with a 32-channel phased array coil. Multiplanar T2-weighted scans were acquired. All sequences acquired in the clinical routine were available to assist tumor localization. For the sagittal T2-weighted fast spin-echo images on which measurements were made, the following technical parameters were set: field of view, 18 cm; slice thickness, 5 mm; gap, 1 mm; flip angle, 90°; bandwidth, 32 kHz; and number of excitations (NEX), 4. MR images were extracted from picture archiving and communication systems (PACS).

Each patient’s DTAV was independently measured by six surgeons with varying levels of experience: two chief surgeons with 15 years of experience, two attending surgeons with 6 years of experience, and two resident surgeons with 3 years of experience. They are represented by chiefs 1 and 2; attending surgeons 1 and 2; and resident surgeons 1 and 2. Additionally, all MR images were taken by a radiologist with over 20 years of experience. The regions of interest (ROIs) were delineated along the contours of the rectum and tumor by the ITK-SNAP software (www.itksnap.org), and they were reviewed and corrected by a radiation oncologist with over 20 years of experience.

Rectum and tumor segmentation

The major architecture of the proposed BGTR is presented in Figure 4.

Figure 4.

Overall workflow of our proposed DTAV measurement method

(A) Overview of our DTAV measurement method.

(B) Flowchart of our boundary-guided transformer.

(C) Flowchart of the transformer block.

(D) Flowchart of the boundary-guided block.

A feature of the rectum and tumors in MR images is that they generally appear at a specific location on the MRI.⁴³ However, the convolutional networks are intrinsically translation invariant, and CNN-based methods do not have the capability to localize the target region. Because of its self-attention structure, a transformer network has the inherent advantage of extracting global information, so it can be exploited to localize the tumor and rectum well in MR images. However, the transformer only focuses on modeling the global context at all stages, which results in low-resolution features. In addition, the high-resolution localization features cannot be efficiently recovered by directly upsampling, which leads to coarse segmentation results. Although CNNs cannot efficiently use global information, they provide an avenue for extracting low-level visual cues, which can well refine the coarse segmentation results. Thus, a hybrid CNN-transformer is of great potential to significantly improve its segmentation performance and achieve accurate localization of the rectum and tumor. Another feature of the rectum and tumor in MR images is their easily recognizable boundaries when compared with other medical imaging modalities like CT. Thus, in our segmentation task, utilizing boundary information is meaningful and potentially fruitful for MRI image segmentation. Therefore, we process the boundary information separately, instead of predicting boundary information within the same branch, to effectively utilize the easily recognizable boundary of the MR image.

Overall, we solved the rectum and tumor segmentation problem from the sagittal MR images as a BGTR. Besides the standard semantic segmentation head, our network uses a parallel boundary detection head to predict the boundaries of an MR image. As mentioned above, the transformer block is designed in a hybrid CNN-transformer architecture so that it can leverage both the local convolutional features and the transformer’s global features to better localize the rectum and tumor regions. In addition, it introduces a BG in particular to process the image’s boundaries and output a boundary feature. Afterward, this boundary feature and the transformer block’s semantic features are fused together by a FFB to obtain a set of boundary-aware features. Finally, the boundary-aware features and the hybrid loss function enable the final prediction block to generate segmentation results. These two techniques can further improve the segmentation performance of our method by making better utilization of the boundary information lying in the MR images. The proposed transformer consists of a transformer block, a BG, an FFB, and a prediction block, as shown in Figure 4B. Next, we describe the transformer block, BG, FFB, and prediction block in detail.

Transformer block

This block first divided the image into several patches of size $\mathrm{p}\times \mathrm{p}$. Therefore, there were ${\mathrm{N}}_{\mathrm{p}}=\mathrm{H}\times \mathrm{W}/{\mathrm{p}}^2$ patches in total. The ${\mathrm{N}}_{\mathrm{p}}$ patches were flattened into a 1D sequence, respectively. ResNet-50⁴⁴ was then used to form embeddings, that is, a set of tokens denoted by ${\mathrm{t}}_{\mathrm{n}},\mathrm{n}=1,...,{\mathrm{N}}_{\mathrm{p}}$. Further, we took the outputs of the first and second ResNet50 blocks, layers 9 and 12 of the transformer encoder, as four representations (denoted by ${\mathrm{f}}_1$, ${\mathrm{f}}_2$, ${\mathrm{f}}_3$, and ${\mathrm{f}}_4$ respectively) of the MR image in different resolutions. Afterward, an FFM is gradually exploited to decode these four representations. In this study, the FFM is designed in the RefineNet style⁴⁵ for enhancing dense prediction with multiscale features. As shown in Figure 4C, the FFM was used to fuse ${\mathrm{f}}_1$, ${\mathrm{f}}_2$, ${\mathrm{f}}_3$, and ${\mathrm{f}}_4$ as ${\mathrm{f}}_{\mathrm{t}}$. ${\mathrm{f}}_{\mathrm{t}}$ is the output semantic features of the transformer block.

BG

At first, the ground-truth boundary map was generated by performing the Canny operation⁴⁶ on the ground-truth semantic map and then taken as the input of this module. The core of this module is the gated convolutional layer (gated CNN), which primarily processes the boundary-related information.³⁶ As shown in Figure 4D, the gated CNN has two inputs: one is the intermediate representations (${\mathrm{T}}_{\mathrm{n}}$) of the transformer block and the other is the intermediate representations (${\mathrm{B}}_{\mathrm{n}}$) of the BG. The gated CNN first generated the attention map from ${\mathrm{T}}_{\mathrm{n}}$ and ${\mathrm{B}}_{\mathrm{n}}$, in which larger weights were given to those regions with salient boundary information. Then, we dot multiplied ${\mathrm{B}}_{\mathrm{n}}$ with the attention map pixel by pixel and obtained the boundary features by a residual connection. In the BG, the gated CNN can be regarded as the collaborator of the transformer block and BG. The transformer block has formed a high level of semantic features of the rectum and tumor. The gated CNN helps the BG focus on the boundary-related information at the very beginning so that the BG can be constructed in a shallow structure to process the boundary features at a high resolution.

FFB

We take the well-known atrous spatial pyramid pooling (ASPP) module⁴⁷ as the base of our FFB, which fuses the semantic features from the transformer block and the boundary features from the BG while preserving multiscale context information. Specifically, the FFB was exploited to fuse the output (${\mathrm{f}}_{\mathrm{t}}$) of the transformer block and that (denoted by ${\mathrm{f}}_{\mathrm{b}}$) of the BG together as ${\mathrm{f}}_{\mathrm{f}}$.

Prediction block

$\mathrm{T}\mathrm{h}\mathrm{e}{\mathrm{f}}_{\mathrm{f}}$ obtained from FFB was fed into the prediction block to infer the semantic segmentation results of rectum and tumor, as shown in Figure 4B.

Finally, we constructed a hybrid loss function to effectively train our proposed network. We used binary cross entropy loss for boundary recognition and standard cross entropy loss for semantic segmentation. The hybrid loss function is constructed as

$$\mathrm{L}\mathrm{o}\mathrm{s}\mathrm{s}={\lambda }_1{\mathrm{L}}_{\mathrm{B}\mathrm{C}\mathrm{E}}\left(\mathrm{b},\hat{\mathrm{b}}\right)+{\lambda }_2{\mathrm{L}}_{\mathrm{C}\mathrm{E}}\left(\hat{\mathrm{t}},f\right)\text{,}$$ (Equation 1)

where $\hat{\mathrm{b}}$ denotes the ground-truth boundary map, $\mathrm{b}$ denotes the predicted boundary map, $\mathrm{f}$ denotes the predicted segmentation result, and $\hat{\mathrm{t}}$ denotes the ground-truth semantic map. ${\lambda }_1$ and ${\lambda }_2$ are two hyperparameters that balance the weights of two different losses.

DTAV measurement based on segmented rectum and tumor

The DTAV refers to the distance from the lowest boundary of the rectal tumor to the anal boundary. Ideally, the rectum is symmetric with respect to its central line, thus the length of the rectum’s anterior wall and that of the rectum’s posterior wall are the same. Therefore, in our work, we used the anterior wall of the rectum between the lowest point of the rectal tumor and the anal verge (i.e., the left arc between the two yellow lines in Figure 5A) to estimate the DTAV. Owing to the advantages of our segmentation method, the segmented rectum and tumor generally do not contain noise pixels and do thus not require any post-processing procedures to clean up the segmentation results. After obtaining the segmentation results for the rectum and tumor, we have directly utilized the Canny operator to extract the boundaries of the segmented rectum and those of the segmented tumor. Furthermore, the rectal anterior wall (denoted by $\mathrm{L}$) from the lowest point of the rectal tumor to the anal verge was extracted from the boundary image, as shown in Figure 5A. Finally, the DTAV can be conveniently estimated by summing the Euclidean distance of every two adjacent pixels in $\mathrm{L}$ as

$$\mathrm{D}\mathrm{T}\mathrm{A}\mathrm{V}={\mathrm{n}}_{\mathrm{x}}·{\mathrm{d}}_{\mathrm{x}}+{\mathrm{n}}_{\mathrm{y}}·{\mathrm{d}}_{\mathrm{y}}+{\mathrm{n}}_{\mathrm{x}\mathrm{y}}·{\mathrm{d}}_{\mathrm{x}\mathrm{y}}$$ , (Equation 2)

where ${\mathrm{d}}_{\mathrm{x}}$, ${\mathrm{d}}_{\mathrm{y}}$, and ${\mathrm{d}}_{\mathrm{x}\mathrm{y}}$ denote the distances between two adjacent pixels along the horizontal axis, vertical axis, and diagonal direction, respectively. ${\mathrm{n}}_{\mathrm{x}}$, ${\mathrm{n}}_{\mathrm{y}}$, and ${\mathrm{n}}_{\mathrm{x}\mathrm{y}}$ denote the number of pairs of adjacent pixels along the horizontal axis, vertical axis, and diagonal direction, respectively, in the anterior wall of the rectum. Figure 5B illustrates three types of two adjacent pixels. We define the current point/pixel as ${\mathrm{P}}_1$, and then the distance from ${\mathrm{P}}_1$ to ${\mathrm{P}}_4$ is ${\mathrm{d}}_{\mathrm{y}}$, that from ${\mathrm{P}}_1$ to ${\mathrm{P}}_2$ is ${\mathrm{d}}_{\mathrm{x}}$, and that from ${\mathrm{P}}_1$ to ${\mathrm{P}}_5$ is ${\mathrm{d}}_{\mathrm{x}\mathrm{y}}$, which equals $\sqrt{d_x^2+d_y^2}$. By this means, the DTAV can be calculated with more accuracy compared with the surgeon’s current measurement method on an MR image, thus potentially leading to better treatment outcomes.

Figure 5.

Demonstration example of the proposed rectal anterior wall extraction method and that of the DTAV calculation method

(A) Demonstration example of the proposed rectal anterior wall extraction method. In this subfigure, L1 denotes a horizontal line (i.e., the top yellow line) passing the lowest point of the rectal tumor, and L2 denotes a horizontal line (i.e., the bottom yellow line) passing the anus. The anterior rectal wall (denoted by L) is extracted as the line segment of the rectum’s boundary between L₁ and L₂.

(B) Demonstration example of the DTAV calculation method. Assuming that the blue line (i.e., P₁-P₄-P₅-P₉) is the extracted rectal anterior wall, P is the tumor’s lowest point, and P9 is the point located at the anus, then the DTAV (i.e., the physical length of P₁-P₄-P₅-P₉) can be calculated by summing the distance from P₁ to P₄ (i.e., dx), that from P₄ to P₅ (i.e., dy), and that from P₅ to P₉ (i.e., dxy = sqrt(dx² + dy²)). In this example, DTAV equals (dx + dy + dxy).

Ablation study

In this subsection, we conducted sufficient ablation experiments to demonstrate the value of each building block of BGTR in improving the segmentation performance of the baseline model (i.e., vit_base_resnet50_384).²⁸ The ablation experiment of the FFM in transformer block was first conducted, that is, the baseline models with and without an FFM were quantitatively compared. Then, we conducted another ablation experiment to demonstrate the effectiveness of the BG. Note that when we exclude a BG from the baseline model, we use a simple concatenation in the baseline model to fuse the features from the transformer block and the BG. Finally, we verified the effectiveness of the FFB. Note that this FFB is different from the previously mentioned FFM; this FFM is used to fuse the multiscale features extracted from the transformer block, and that FFB was used to fuse the features extracted from the transformer block and those extracted from the boundary guide block.

Experimental settings

Performance metrics

For the segmentation step, we compared our proposed BGTR with six state-of-the-art methods. A qualitative evaluation was first performed by visually comparing the segmentation results. Then, to evaluate the performance of the rectal tumor segmentation network quantitatively, five metrics, namely ACC, SEN, SPE, PRE, and DICE, are defined as follows:

$$ACC=\frac{TP+TN}{TP+TN+FP+FN}\text{,}$$ (Equation 3)

$$SEN=\frac{TP}{TP+FN}\text{,}$$ (Equation 4)

$$SPE=\frac{TN}{TN+FP}\text{,}$$ (Equation 5)

$$PRE=\frac{TP}{TP+FP},\mathrm{a}\mathrm{n}\mathrm{d}$$ (Equation 6)

$$DICE=\frac{2TP}{2TP+FP+FN}$$ (Equation 7)

where TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative, respectively.

For the measurement step, we used a scatterplot to show the difference in distance between the colonoscopy and the seven measurement methods. The Kolmogorov-Smirnov test was used to assess normal distribution of the data. Normally distributed continuous variables were compared using the t test, and abnormal distributed continuous variables were compared using the Mann-Whitney U test. When interobserver consistency between surgeons were analyzed, the Spearman correlation coefficient (r) was applied. The ICC was used to quantify the correlation between the measurement results of the proposed method, those of the surgeons’ methods, and those of the gold standard (colonoscopy). ICC values of <0.4, 0.4–0.75, and >0.75 were regarded as poor, moderate, and good reliability, respectively. p values for comparison between ICCs were calculated using Z tests. The statistical significance was set to p <0.05.

Comparison methods

At first, selection of the comparison methods for evaluating our image segmentation methods is discussed. To the best of our knowledge, there is only literature for rectal tumor segmentation at present, and no work has been investigated to segment rectum and tumors simultaneously. As is known, FCN is a pioneering deep learning-based method for image segmentation. UNet and its transformer version (i.e., TransUNet) have been widely used in segmenting medical images. Swin-T is a variant of the visual transformer that reduces the computational overhead while enhancing the hierarchical representation. The Cerberus transformer is a recently proposed hybrid CNN-transformer network and has achieved superior results in the task of semantic image segmentation. As for GatedSCNN, it proposed the gated block to ultimately utilize the boundary information of the natural images and thus is one of our closely related works. Therefore, in order to verify the efficacy of our proposed image segmentation method, we have taken the above six state-of-the-art image segmentation methods (i.e., FCN, UNet, GatedSCNN, TransUNet, Swin-T, and Cerberus) as our comparison methods.

Afterward, the comparison methods for evaluating our DTAV measurement method are selected as follows. Currently, we only measure the DTAV automatically, so there are no other automatic DTAV measurement methods for comparison. In clinical practices, the current gold standard method for measuring the DTAV is the measurement method through colonoscopy examination. Besides the colonoscopy method, the surgeon often manually measures the DTAV from the sagittal MR image around the rectal tumor. Thus, we have taken the colonoscopy method and the surgeon’s manual measurement method as the current state-of-the-art (SOTA) methods for DTAV measurement and thus selected these two DTAV measurement methods as our comparison methods.

Implementation details

The 212 2D sagittal MR images in our dataset are randomly divided into a training set (118 images), a validation set (40 images), and a test set (54 images). Note that the test dataset used to compare the performance of the image segmentation methods is the same as that used to evaluate the performance of the DTAV measurement methods. Each source image is a sagittal T2-weighted MR image (containing both the rectum and tumor) selected from the MR image set of a specific patient. The size of each source image is 512 × 512 pixels. Before inputting each source image into our network, it was first augmented by random rotation and random flipping in both horizontal and vertical directions, and then it was normalized for stable learning. The batch size of the input images is set to 4. Further, our baseline model (i.e., vit_base_resnet50_384) was first pretrained on the large-scale ImageNet dataset. Afterward, our proposed model was trained for 300 epochs with the SDG optimizer. The parameter settings for replicating our results are as follows. The weight decay and momentum were set to 1e−4 and 0.9, respectively. The learning rate was set to 1e−2 initially and decreased according to a polynomial decay policy. In our hybrid loss function, there are two hyperparameters (i.e., λ1 and λ2) to balance the weights of the binary cross entropy loss and the cross entropy loss. λ1 and λ2 were empirically set to 20 and 1, respectively. Finally, the proposed model was implemented based on the Pytorch framework and was trained and tested on a computer platform with one NVIDIA GeForce RTX 3090.

Conclusions

In this study, we present a method for automatic measurement of DTAV from the rectal MR images of the sagittal view. Specifically, we proposed a BGTR, which incorporates a BG to improve the transformer’s ability of boundary awareness, to accurately segment the rectum and tumor from the sagittal MR images. Then, the anterior rectal wall from the lowest verge of the tumor to the anal verge can be automatically extracted based on the segmentation results. Finally, the DTAV can be simply estimated by calculating the physical length of the extracted anterior rectal wall according to the spatial resolutions of the sagittal MR image. The DTAV measurement accuracy of our method significantly exceeds that of the attending surgeons, and by referring to our segmentation results, attending surgeons and resident surgeons can significantly increase their measurement accuracy while decreasing their measurement time. Therefore, our method is of great potential to be applied in the clinical scenarios.

Experimental procedures

Resource availability

Lead contact

For any questions regarding the paper and resources, please contact Dr. Zhipeng Zhang (zhangzhipeng06@126.com).

Materials availability

This study did not generate new unique reagents.

Published: March 27, 2023

Data and code availability

Our source code is available at https://github.com/jucyLL/BGTR and has been archived at Zenodo under https://doi.org/10.5281/zenodo.7624962.³⁸ Original data have been deposited to the Zenodo data repository: https://doi.org/10.5281/zenodo.7652416.³⁹ Any additional information required to reanalyze the data reported in this paper is available from the lead contact on request.