The Application and Development of Deep Learning in Radiotherapy: A Systematic Review

Abstract

With the massive use of computers, the growth and explosion of data has greatly promoted the development of artificial intelligence (AI). The rise of deep learning (DL) algorithms, such as convolutional neural networks (CNN), has provided radiation oncologists with many promising tools that can simplify the complex radiotherapy process in the clinical work of radiation oncology, improve the accuracy and objectivity of diagnosis, and reduce the workload, thus enabling clinicians to spend more time on advanced decision-making tasks. As the development of DL gets closer to clinical practice, radiation oncologists will need to be more familiar with its principles to properly evaluate and use this powerful tool. In this paper, we explain the development and basic concepts of AI and discuss its application in radiation oncology based on different task categories of DL algorithms. This work clarifies the possibility of further development of DL in radiation oncology.

Introduction

With the massive use of computers, the growth and explosion of data has greatly promoted the development of artificial intelligence (AI). In recent years, AI has been used widely in the medical field to analyze data in pathology, radiology, cardiology, oncology, genomics, and pharmacology to better provide information for diseases prediction, ^{1 -4} screening, ⁵ diagnosis, ^6,7 treatment, ⁸ prognosis, health management, and drug development. ⁹ The various applied research currently underway may lead to the increased use of AI by clinicians, in particular, radiation oncologists. ¹⁰ The current clinical practice is both time-consuming and extremely subjective, and the rise of deep learning (DL) algorithms, such as convolutional neural networks (CNN), can simplify the complex radiotherapy work process in the clinical work of radiation oncology, including image fusion, delineation of clinical target volume (CTV), and organ-at-risk (OAR), automatic planning (AP), dose distribution prediction, and outcome prediction. ¹¹ The application of DL not only improves the accuracy and objectivity of diagnosis but also reduces the workload, thus enabling clinicians to spend more time on advanced decision-making tasks. As the development of DL gets closer to clinical practice, radiation oncologists will need to be familiar with its principles to properly evaluate and use this powerful tool. We explain the development and basic concepts of AI and discuss its application in radiation oncology based on different task categories of DL algorithms. This work clarifies the possibility of further development of DL in radiation oncology.

Basic Concepts

Artificial intelligence (AI): AI is a broad general term used to encompass various subdomains that are used specifically to create algorithms to perform tasks that mimic human intelligence. ¹²

Machine learning (ML): ML belongs to a subfield of AI and is the primary method used to realize AI. ML can provide algorithms, which can build mathematical models based on the collected data. These mathematical models map the input data to the desired output. These input elements can be any sequence of images, numbers, and classification data. ¹³ ML algorithms are divided into supervised learning, unsupervised learning, reinforcement learning, integrated learning, and DL. Among them, the input data of supervised learning contains tutor signals, which use probability function, algebraic function or artificial neural network as basis function model, adopt iterative calculation method, and learn result as function. Unsupervised learning is that there is no tutor signal in the input data, and the clustering method is adopted, and the learning result is the category. Typical unsupervised learning includes discovery learning, clustering, and competitive learning. Reinforcement learning is a learning method that takes environmental inertia (reward/punishment signals) as input and is guided by statistics and dynamic programming techniques. Ensemble learning is to combine multiple weakly-supervised models here in order to obtain a better and more comprehensive strong-supervised model. The underlying idea of ensemble learning is that even if a certain weak classifier gets a wrong prediction, other weak classifiers may correct its mistakes. In practice, to represent input to output mapping, different functional representations can be used, such as decision trees, ¹⁴ support vector machines (SVM), ¹⁵ naive Bayes classification, ¹⁶ and deep neural network (DNN). ¹⁷ In most cases, ML methods can achieve at least as good results as traditional statistical methods. When the basic input–output relationship is not linear and the data set is large enough and contains predictors that capture the nonlinear relationship, the performance of the ML method will be better than the linear statistical model. ¹⁸ The SVMs, decision trees, and naive Bayes classification all use supervised learning algorithms to build models, and DNNs use DL algorithms to build models. Ideally, ML can use computers to predict clinical outcomes, identify disease patterns, detect disease characteristics, and optimize treatment strategies, thereby transforming the acquired knowledge into clinical evidence. ¹³

Deep learning (DL): DL (also known as DNN) was introduced at an event at the end of 2012, when the DL method based on CNN won the world’s most famous computer vision competition ImageNet classification. ¹⁹ DL is a computational model that allows for multiple processing layers (input layer, hidden layer, and output layer) to discover complex structures in large data sets using a back propagation algorithm to instruct the machine how to change its internal parameters. ²⁰ In the past 10 years, as a result of the massive use of computers and the growth and explosion of data, DNN has surpassed others in computer vision applications, such as processing and understanding text, ²¹ voice, ²² and images. ²³ DNN includes CNN, recurrent neural network (RNN), and fully convolutional neural network (FCN). CNN is a type of feedforward neural network that contains convolutional calculations and has a deep structure that can classify input information according to its hierarchical structure and that usually is used for images and other data with grid-like structures. To overcome the difficulty of a network whose width increases linearly caused by convolution, some scholars have introduced downsampling to reduce the width. They have proved that downsampling a CNN can approximate the ridge function well, which illustrates the advantages of these structured networks in approximation or modeling. ²⁴ In recent years, CNN has introduced breakthroughs in image, video, voice, and audio processing.

DL task categories in radiation oncology can be divided according to the main purpose of the algorithm, as follows: image fusion, image segmentation, AP, plan evaluation, and prognosis and outcome prediction. The evaluation criteria include receiver operating characteristic curve, area under the receiver operating characteristic curve (AUC), dose volume histogram, dose difference graph, F1 score, accuracy, specificity, sensitivity, precision, dice similarity coefficient, average accuracy, and Jaccard index. ²⁵

Material and Methods

Here we provide a systematic review of the publications using CNN technology for medical image analysis, available in the National Library of Medicine database (PubMed). The search equation was the following: (convolutional OR deep learning) AND (radiotherapy) AND (image fusion OR image segmentation OR auto-planning and dose distribution prediction OR prediction of efficacy and side effects), filtered for “Human studies” and “Title/Abstract” as search fields.

The selected articles were screened according to a standard grid containing the following items: aim of the study; methods: network architecture, dataset, training, validation, test, comparison method; results: accuracy, sensibility and specificity and conclusion.

Implementation Area

Image fusion: Medical image fusion technology can fuse medical images from multiple forms, thereby making the medical diagnosis and treatment process more reliable and accurate. ²⁶ Image registration is an important part of image fusion, for which the process is to find the spatial mapping relationship between one image pixel and another image pixel. This process is not absolute. Its core purpose is to identify the conversion relationship between different images. These images can be taken at different times (multitime registration), different sensors in different places (multimode registration). The relationship between these images can be rigid (translation and rotation), affine, homography, or complex large deformation models. For image-guided radiotherapy, radiosurgery, and interventional radiotherapy, image registration is one of the key technologies of auxiliary medical care. In recent years, DL, especially CNN, has achieved good results in medical image processing, and medical registration research has developed rapidly. ²⁷ The 2 main types of existing medical image registration methods are gray-scale-based methods and feature-based methods. The primary steps of image registration include geometric size change, combined image change, image similarity measurement, iterative optimization, and interpolation process. ²⁸ In the traditional registration method, the cost function is iteratively optimized from scratch so that the images can be registered, which severely limits the registration speed. Compared with traditional medical image registration methods, the greatest contribution of DL in medical image registration is to resolve the problem of slow medical image processing. ²⁷ Eppenhof and Pluim ²⁹ studied a CNN-based deformable registration algorithm and compared it with traditional algorithms. Their results showed that the registration speed of DL networks is hundreds of times that of traditional registration methods, with an average of 0.58 ± 0.07s. Among the existing research results, the main DL methods used are CNN and FCN frameworks. A study by Cao et al ³⁰ used CNN for brain magnetic resonance imaging (MRI) image registration. The results showed that the Dice similarity coefficient (DSC) was improved in the registration of gray matter, white matter, and cerebrospinal fluid. The rate of improvement was 2.6%. Fan et al ³¹ studied 7different brain MRI deformable registration algorithms. The results showed that the DL network that did not require any iterative optimization also required the least time for registration, and the registration accuracy was based on DL. There are also improvements. Research by Jiang et al ³² showed that 4-dimensional computed tomography (CT) deformed image registration of the lung based on CNN had the smallest error compared with various traditional methods, and the registration time was 1.4s. Hasenstab et al ³³ evaluated the performance of a CNN algorithm for liver registration in 314 patients, and compared it with manual image registration. The results showed that compared with manual registration, the liver overlap and image correlation for automatic registration were higher. In conclusion, DL has made medical image registration more rapid and accurate, which is keeping with the needs of clinical practice.

Image segmentation: The delineation of CTV and OAR of tumor patients is a critical and time-consuming part of the radiotherapy process. Usually, it is manually delineated by the radiotherapist. The delineation results often are inconsistent because the experience of the delineator may vary. The emergence of DL has made the automatic segmentation of the tumor and OAR possible. The DSC is usually used as an indicator to evaluate the reliability of the test software output in this field. The closer the DSC result to 1, the higher the degree of overlap between the 2 delineations. If DSC is equal to 1, the 2 delineations completely overlap.

At present, several scholars have applied AI to OAR and CTV for head and neck tumors, lung cancer, breast cancer, prostate cancer, rectal cancer, and cervical cancer. ^{34 -51} Zeineldin et al evaluated the performance of different CNN models in 125 cases of glioma. Compared with manual rendering, the DSC of several CNN models was 81% to 84%. Zeineldin et al believed that different CNN models could be applied to magnetic resonance images and that segmentation of brain tumors was feasible. ³⁴ Deng et al developed a novel brain tumor segmentation method, which integrated the full convolutional neural network (FCNN) and dense micro-block difference feature (DMDF) into a unified framework to segment brain tumors in the MRIs of 100 patients. The average DSC index was as high as 90.98%, and the segmentation time was less than 1s. Compared with the traditional MRI brain tumor segmentation method, the experimental results showed that the segmentation accuracy and stability were greatly improved. ³⁵ Ye et al used an automated method based on CNN for segmentation of nasopharyngeal carcinoma on dual-sequence magnetic resonance imaging. Through automatic contour training of 44 patients with nasopharyngeal carcinoma, the test results obtained in 7patients had an average DSC of 0.87. ³⁶ Another prospective study used DCNN to train and automatically segment the gross tumor volume (GTV) of 22 patients with head and neck cancer on co-registered positron emission tomography (PET-CT) images. Oncologists and radiologists have manually determined the gold standard of GTV by consensus. The automatic segmentation time is less than 1 min, and the average DSC is 0.785. ³⁷ Tong et al used DNN to segment 9 OARs (brain stem, optic chiasm, mandible, optic nerve, parotid gland, and submandibular gland) in 22 head and neck cases. The average DSC ranged from 0.58 to 0.93. The median time of all OARs was 9.5s. ³⁸ Zhu et al tested the results of automatic segmentation of 9 OARs (brain stem, cross, mandible, left optic nerve, right optic nerve, left parotid gland, right parotid gland, left submandibular gland, and right submandibular gland) in CT images of 261 patients with nasopharyngeal carcinoma based on a DL framework. Tong’s model used a single network to segment the OAR and to conduct end-to-end training, called AnatomyNet. Zhu et al found that compared with the traditional U-Net model, AnatomyNet improved the DSC by 2% to 3%, and 6 out of 9 anatomical structures were better than those under U-Net. ³⁹ Subsequently, Dai et al proposed a DCNN that used a 3-dimensional U-Net DCNN combined with 2 loss functions of dice loss and generalized dice loss to automatically segment 19 OARs (left and right eyeballs, left and right optic nerves, left and right lenses, left and right inner ears, left and right temporomandibular joints, left and right parotid glands, left and right submandibular glands, brainstem, spinal cord, thyroid, laryngo-esophagus-tracheal (LET), and oral cavity) in patients with nasopharyngeal carcinoma. A total of 496 patients were enrolled in the group, and 376 cases were randomly selected for use in training set, 60 cases were included in the validation set, and 60 cases were included in the test set. Overall, the average DSC of the 19 high-risk organs was 0.91, and the Jaccard distance was 0.15. Compared with Zhu’s method, the 3-dimensional (3D) U-Net DCNN combined with Dice Loss function could be better applied to the automatic segmentation of head and neck OARs. The 3D U-Net DCNN with the segmentation time within 20S also achieved ideal automatic segmentation results for small-volume OAR. ⁴⁰ Shapey et al studied the performance of two-and-half-dimensional CNN to automatically segment schwannomas after training in the MRIs of 243 patients with schwannomas. Compared with manual segmentation, the DSC based on T1-weighted segmentation was 93.43%. The DSC for segmentation based on T2 weighting was 93.68%. ⁴¹ A prospective study included 126 patients with intracranial meningioma. The target volume contour manually drawn on MRI T1/T2 weights by 2 experienced doctors was compared with the results of a trained DNN. In these patients, a comparison between the DL model and manual segmentation showed that the average DSC of the tumor volume of the enhanced contrast agent was 0.91 ± 0.08, and the average DSC of the total lesion volume was 0.82 ± 0.12. ⁴² Another study used 2-dimensional CNN to train on 300 patients’ head CT images and automatically segment the ventricles. The results showed that compared with manual rendering, the DSCs of the left, right, and third ventricles were 0.92, 0.92, and 0.79, respectively. ⁴³ Currently, many reports are available on the automatic segmentation of DL for head and neck cancer. Peng used CNN for automatic segmentation of OARs in the chest and abdomen. The research developed and trained a CNN based on U-Net, which included 60 chest CT scan patients and 43 abdominal CT scan patients. Peng et al performed 5 organ segmentations on chest CTs and 8 organ segmentations on abdominal CTs. Compared with manual drawing, the median DSC was 0.97 (right lung), 0.96 (left lung), 0.92 (heart), 0.86 (spinal cord), 0.76 (esophagus), and 0.96 (spleen), 0.96 (liver), 0.95 (Left kidney), 0.90 (stomach), 0.87 (gallbladder), 0.80 (pancreas), 0.75 (esophagus), and 0.61 (duodenum). The automatic segmentation time for each patient did not exceed 5S. The researcher believed that this work shows that the patient’s multiorgan CT image segmentation could be performed with clinically acceptable accuracy and efficiency. ⁴⁴ Wang et al developed a patient-specific adaptive convolutional neural network (A-NET) to segment lung tumors in 9 patients’ chest MRIs. The patients in the group had a chest MRI every week during radiotherapy. Wang et al took the previously scanned images as the training set and used the latest images for verification. Compared with manual segmentation, the DSC obtained was 0.81 ± 0.10. ⁴⁵ Another prospective study proposed a new multimodal segmentation method based on 3D FCN that simultaneously considered PET and CT information for lung tumor segmentation. This method was validated on a dataset of 84 lung cancer patients. Compared with the profile drawn by abundant radiation oncologists, the average DSC was 0.85, which achieved significant performance gains compared with CNN-based methods and traditional methods that used only PET or CT. ⁴⁶ Zabihollahy et al studied a similar ensemble learning model based on U-Net to identify and describe the 3D U-Net of kidney tumors, using contrast-enhanced CT images of 315 patients as training and test sets. Compared with the gold standard, using 3D U-Net to describe the average DSC of kidney tumors was 85.95% ± 1.46%. ⁴⁷ The research of Chen et al developed a new cervical cancer segmentation method (called PIC-S-CNN). Chen et al compared this method with 6 different segmentation methods and obtained the best segmentation effect, with an average DSC of 0.84. Chen et al believed that the combination of DL and anatomical prior information could improve the accuracy of cervical tumor segmentation. Scholars who have studied models based on CNN to automatically segment pancreatic tumors, ⁴⁸ liver tumors, ⁴⁹ colorectal tumors, ⁵⁰ and prostate tumors ⁵¹ have achieved good segmentation results. These findings have shown that DL can save a significant amount of clinician time to delineate CTV and OARS. Most of these delineation results have met the requirements of clinical treatment and achieved better results than those manually delineated by physicians. Not only does this method have high repeatability, but it also can reduce the interobserver variability (IOV) among physicians.

AP and dose distribution prediction: The ability to automatically generate plans and predict a priori acceptable dose distribution is one of the most important aspects of AI-related radiotherapy plan implementation. Liu et al designed a DNN (called deep MTP) to generate pseudo-CT for AP based on the MRIs of brain tumor patients. Comparing the generated automatic plan with the clinical treatment plan dose parameters, the results provided by the automatic plan–generated dose distribution were not significantly different. ⁵² Fan et al trained the DNN framework in 195 patients with head and neck cancer to predict the dose distribution of head and neck patients receiving radiotherapy. They used 25 cases for verification and 50 cases as tests. The results showed that, except for the brainstem and lens, all clinically relevant dosimetry parameters were not detected to be statistically different from the actual clinical plan. ⁵³ The combination of CNN and the Monte Carlo (MC) method can be used to predict the dose of brachytherapy. In another study, 47 prostate cancer patients were used as a training set, 14 prostate cancer patients and 10 cervical cancer patients were used as a test set, and the results could be used for clinical needs. The accuracy of the algorithm was close to that of using only the MC method, and the calculation time was significantly reduced. ⁵⁴ Kajikawa et al compared CNN with traditional DL methods to predict the dose distribution of intensity-modulated conformal radiotherapy plans for prostate cancer patients. Kajikawa et al used an adaptive moment estimation algorithm to optimize the 3D U-net, and the results showed that the CNN model could predict a better or comparable dose distribution than that produced by DL. ⁵⁵ Similar experience in predicting dose distribution for prostate cancer was validated in a plan for 80 prostate cancer patients. ⁵⁶ Although the current clinically available ML-based automatic plan effectively saved time, the generated plan still had to be corrected manually. In the future, DL-based AP commercial software is expected to generate plans that can directly meet the needs of clinical treatment.

Other applications (for the prediction of efficacy and side effects): The vast majority of DL predictions of the outcome of radiotherapy use DL to predict the toxicity after radiotherapy. Xerostomia usually occurs in patients who are receiving radiation therapy to the head and neck. Men et al developed a model to predict xerostomia after radiotherapy based on a 3D residual convolutional neural network and included 784 patients with head and neck squamous cell carcinoma of RTOG 0522 test. Using CT planning images, 3D dose distribution, and contours of the parotid and submandibular glands as inputs, good prediction results were obtained. ⁵⁷ In patients with non-small-cell lung cancer (NSCLC), CNN has been used to predict tumor recurrence after stereotactic radiotherapy. This model was established based on the CT images reviewed after radiotherapy. In the analysis of 1605 characteristics of 46 patients, 5 predicted local recurrence, 3 of which were lobar recurrence, and 7 predicted overall survival. ⁵⁸ Liang et al constructed a 3D CNN model to predict the occurrence of radiation pneumonia after thoracic radiotherapy, and compared it with 3 prediction models based on multiple logistic regression. The 4 prediction models were all validated in 70 patients with NSCLC who received volume-modulated radiotherapy. The results showed that CNN performed better than the traditional model, with an AUC of 0.842. ⁵⁹ Lee et al applied ML and bioinformatics tools to genome-wide data to predict and explain the late genitourinary system toxicity in prostate cancer patients after radiotherapy. ⁶⁰ DL’s prediction of the efficacy and side effects of radiotherapy was able to screen out the possible beneficiaries of radiotherapy in clinical work and to prepare for possible side effects.

Conclusion

Radiation oncology is a medical specialty that closely integrates technology and computers. It should integrate computer science, statistics, and clinical knowledge. In the process of clinical radiotherapy, AI algorithms can work continuously and efficiently. In particular, the emergence of DL algorithms can automatically perform tedious tasks, reduce the deviation of dose distribution, and predict adverse effects after radiotherapy. CNN training is a key step, for which specific technical skills are required to avoid overfitting limited data, which can lead to problems when using the network to analyze wider data sets. Therefore, training needs to be evaluated and monitored. Training thus requires evaluation and monitoring. This method is expected to become the third hand of radiation oncologists. The open-source nature and public availability of the AI library enable clinical researchers from various fields to research and use AI algorithms, which can improve objectivity, reduce the need for manual intervention, and reduce the amount of staff work. At the same time, the repeatability of the process can be greatly improved. Because DL algorithms are an opaque “black box” of internal operations, applying them to clinical practice remains challenging. ¹¹ Some systems provide partial visualization techniques (heat maps, probability maps) to provide certain views of CNN internal functions. Understanding how these networks “work” is a relevant and significant challenge in medical AI.

Currently available clinically automatic registration and automatic segmentation software based on ML algorithms require manual correction before clinical use, and the segmentation results for small organs are not ideal. ^61,62 The current technology and framework have limitations, which include model interpretability, data heterogeneity, and lack of common benchmarks. ⁶³ Even if these AI systems show high accuracy in a laboratory environment, it is difficult to practically verify medical AI systems in clinical work. This difficulty is called the last mile of implementation. ⁶⁴ Therefore, before clinical implementation, in-depth research is needed to evaluate the performance of DL algorithms. ⁶⁵ One way to make DL results more acceptable in clinical practice is to enable doctors to understand the internal workings of the equipment they use, and the software must provide data protection, algorithm transparency, and accountability to earn clinician and patient trust. ⁶⁶ Artificial intelligence has clearly demonstrated its efficiency in radiotherapy tasks, but for most applications, there is still a lack of comparative clinical studies showing that the technology has been integrated into the clinical workflow. Nevertheless, the robustness of the current results and the possible simple interface that can be designed using trained CNNs lay the foundation for direct, time-saving, reliable and practical applications. Then, you can treat CNN as a colleague to provide expert second opinions on difficult clinical issues. In addition, CNN is inherently not affected by chaotic factors such as fatigue, personal beliefs, or hierarchical issues, so inter- and intra-individual variability will be minimized when completing specific tasks.

CNN will completely change all processes in the field of radiotherapy, and the role of practitioners is crucial to the development and implementation of such equipment. By understanding deep learning, participating in the concept and evaluation of new equipment, and by contributing one’s own power to conceive the regulatory framework for this new type of medical activity, the MD now has the opportunity to participate in the scientific revolution.