A Systematic Review of Automated Segmentation Methods and Public Datasets for the Lung and its Lobes and Findings on Computed Tomography Images

Summary

Objectives : Automated computational segmentation of the lung and its lobes and findings in X-Ray based computed tomography (CT) images is a challenging problem with important applications, including medical research, surgical planning, and diagnostic decision support. With the increase in large imaging cohorts and the need for fast and robust evaluation of normal and abnormal lungs and their lobes, several authors have proposed automated methods for lung assessment on CT images. In this paper we intend to provide a comprehensive summarization of these methods.

Methods : We used a systematic approach to perform an extensive review of automated lung segmentation methods. We chose Scopus, PubMed, and Scopus to conduct our review and included methods that perform segmentation of the lung parenchyma, lobes or internal disease related findings. The review was not limited by date, but rather by only including methods providing quantitative evaluation.

Results : We organized and classified all 234 included articles into various categories according to methodological similarities among them. We provide summarizations of quantitative evaluations, public datasets, evaluation metrics, and overall statistics indicating recent research directions of the field.

Conclusions : We noted the rise of data-driven models in the last decade, especially due to the deep learning trend, increasing the demand for high-quality data annotation. This has instigated an increase of semi-supervised and uncertainty guided works that try to be less dependent on human annotation. In addition, the question of how to evaluate the robustness of data-driven methods remains open, given that evaluations derived from specific datasets are not general.

1 Introduction

Computed tomography (CT) is one of the most important diagnostic modalities used in different clinical conditions for diagnosis, follow-up, and image-guided procedures [ 1 ]. Analysis of digital CT images allows the segmentation of the lungs and their lobes, by identifying the anatomic boundaries, followed by segmentation of abnormal lung tissue according to the underlying pathological process and disease [ 2 ]. The ongoing COVID-19 pandemic has highlighted the importance of lung CT in clinical settings and its association not only with medical research and diagnosis but also with prognosis [ 3 ]. This generated renewed interest in automated lung assessment using digital CT images since manual segmentation is very time-consuming and poorly reproducible.

For decades, researchers have been attempting to propose robust and accurate computational algorithms to automatically segment the lung parenchyma, its lobes, and internal findings. Recent work has achieved performance comparable to a human radiologist in automated lung parenchyma segmentation [ 4 ]. However, some challenges arise due to the variations and complexity of anatomy that appear in diseased lungs ( Figure 1 ), or when focusing the segmentation target on disease-related findings. To solve this problem, different segmentation techniques have been proposed, and are the target of this systematic review.

Fig. 1.

Chest CT scan reconstructions displaying findings (from left to right) from COVID-19, cancer, and pneumonia. Yellow arrow: extensive right lung consolidation with permeative ground-glass opacities. Green arrow: spiculated pulmonary nodule (suspected for malignancy). Blue arrow: subpleural bilateral ground glass opacities.

This review is motivated by the need to assess the literature and the forthcoming directions of segmentation of the lung parenchyma and its radiological findings. In terms of disease related findings, we employ a large search scope including not only COVID-19 related research but also many other diseases, including cancer nodules, chronic obstructive pulmonary disease (COPD), interstitial lung disease (ILD), pleural effusion (PE) and others. Methods for lung fissure and pulmonary lobe segmentation were also included since the identification of lobes also has important applications in medical research, disease assessment, and treatment planning [ 5 , 6 ].

Such a large scope requires limiting the search keys for the review to be feasible. Therefore, methods are required to be computational, completely automated, and be quantitatively evaluated for the segmentations against well-defined ground truth. Considering COVID-19 as an example, there are several recently publicly available datasets and accompanying classification methods, from a global effort from researchers to aid in the ongoing pandemic. However, these methods are not included in this review, if they do not provide evaluated segmentations.

This review is organized as follows. In section 2, we describe the methodology of this systematic review in detail, to allow for reproducibility. In section 3, we grouped the methods included in this review into methodology categories, briefly introducing most of the methods within each category. Section 4 presents a summary of our results, including public datasets, highlighted methods per target, and statistics. In sections 5 and 6, we use these results to draw conclusions about the state-of-the-art and discuss what are the current gaps and future directions for the field. The complete tables for the systematic review extraction separated by target and method category are included as Supplementary Material , with useful information and results on all included methods.

2 Systematic Review Methodology

A systematic review differs from a classic review or survey, following a more deterministic approach. All steps are recorded for ease of reproducibility, to minimize subjective decisions and to reduce author bias [ 9 , 10 ]. The first step consists of defining a research question, being in our case: “What are the quantitatively evaluated, computed, and automated segmentation methods of the lung and its lobes, and the findings, using computed tomography images”. Secondly, it is necessary to define which deterministic online databases to search on. We used Scopus 1 , Embase 2 , and PubMed 3 . The third step is to define search queries and year limitations to be used on these databases. We did not limit the search by year, however, the requirement for quantitative results effectively removed older research. The most recent results were limited to April 30, 2021, the date when we locked our search results to start the systematic methodology.

The search query, composed of a Boolean logic sentence, specified which words or combination of words to look for in the abstract and title of articles [ 10 ]. Our final query was the following: (CT OR HRCT OR computed tomography) AND automat* AND (lung OR pulmonary) AND (segment* OR contour*) AND (accuracy OR dice OR hausdorff OR iou OR intersection). The inclusion of the last term of this logic expression was necessary to filter out papers that do not discuss the accuracy of their results. Including more metrics as keywords did not result in increasing the number of included articles. The exact way queries were used as input to each database is included in the Supplementary Material .

The systematic review was carried out using the Covidence platform [ 11 ], with two blind reviewers and one senior researcher (tiebreaker). All exclusion and inclusion decisions were made blindly by the two reviewers, with conflicts being resolved by the tiebreaker. The Prisma ( Figure 2 ) details the progression of inclusion and exclusion of the review methodology. After importing all 2,028 resulting references from the searches on Scopus, Embase, and PubMed, 708 duplicate studies were automatically removed from our database by Covidence. Following this, the titles and abstracts of the 1,320 remaining studies were screened for relevance, with 876 being excluded. Exclusion criteria when analyzing titles and abstracts were: methods that did not focus on automated segmentation involving the lung, animal studies, or methods using other imaging modalities. Note that the search key is not enough to avoid the need for these exclusions as it possible that papers may use lung segmentation as a preprocessing step, or even cite it as background while focusing on other aspects such as classification. Those that do not violate these criteria go to the full-text assessment phase.

Fig. 2.

Prisma figure extracted from the Covidence tool, summarizing the systematic review process and the number of articles at each step.

Even after excluding papers by title and abstract assessment, the inclusion and exclusion criteria needed to be checked again, for the remaining 444 full texts. Only studies with full texts in English were retained. Studies that did not have quantitative results or that did not provide numerical results for the segmentations were excluded, as well as studies that used other imaging modalities other than X-Ray based CT or synthetic data. Publications that only validated but did not propose a methodology were also excluded.

Finally, studies for which we could not access the full text even after contacting the authors and duplicates not detected by Covidence were also excluded.

Finally, the last phase consisted of parallel data extraction and quality assessment [ 10 ] of the 257 remaining studies. Details of the extraction form and more information on Covidence are in the Supplementary Material . Here, the authors came to a consensus of which data should be extracted, and forms were customized in the Covidence tool to be filled by both reviewers. This data was used to compute statistics and tables and to guide the discussions accompanying this review. The only criteria we employed for exclusion of papers due to quality reasons was insufficient data variability. We excluded studies that either did not present the number of CT slices or subjects involved in segmentation or had too few slices (<50) or patients (<5). This resulted in the exclusion of research that used many subjects for classification but did not provide segmentation metrics on a significant portion of the data. Finally, no judgments about merit or writing quality were involved in this quality assessment. The final number of papers removed for quality reasons was 34, resulting in a final set of 223 methods.

3 Categories and Targets of the Methods

From our choice to include methods dealing with multiple different target structures, we proposed four target groups to facilitate analysis. Most methods somehow segment the lung, but the target refers to the main structure that is quantitatively evaluated. These groups are lung parenchyma, pulmonary nodules, fissures or lobes, and other findings. Pulmonary nodules get a separate category due to the amount of work focusing on these specific findings. Note, however, that methods that only performed nodule detection without fine segmentation were not included in this review. Examples of diseases affecting subjects on all works included: COPD, PE, ILD, idiopathic pulmonary fibrosis (IPF), cystic fibrosis, COVID-19 pneumonia, tuberculosis, cancer, emphysema, and asthma.

During the full-text screening phase of the review, we investigated if the method categorization proposed by Mansoor et al ., [ 2 ] in their 2015 review was still applicable. They proposed the following classification: neighboring anatomy-guided [ 13 ]; thresholding [ 12 , 14 ]; region-based [ 2 , 15 ]; shape or model-based [ 16 ]; machine learning [ 17 ] and hybrids [ 18 ]. Hybrids include methods where it is not clear to which category of method it fits, usually because they use a combination of multiple types of approaches [ 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 ]. We concluded that while this classification still applies, we noticed the necessity to separate machine learning into two categories: traditional machine learning and deep learning [ 33 ], due to the recent explosion of deep learning methods.

We used seven categories to classify methods during the extraction phase. Although the best effort was given by the two reviewers and the tiebreaker to correctly classify all types of methods, the authors do not claim this classification to be 100% correct for all methods, due to some approaches blurring the line between the proposed categories. The distribution of categories and target structures included in the extraction process highlights the explosion of deep learning methods after the famous UNet paper [ 34 ], and a large amount of COVID-19 segmentation research following the start of the pandemic. In the following subsections, all papers are separated into methodology categories, with an overview of the techniques involved in each category. Information on quantitative evaluation and data are present in section 4 and in the Supplementary Material .

3.1 Thresholding-based Methods

Thresholding-based methods aim to exploit the known relation between Hounsfield unit (HU) values in CT images and organs [ 1 ]. For lung segmentation, the thresholding application is generally adaptive through iterations [ 35 ]. Often, vessels, airways, and other internal findings generate noise, which can be solved in some cases by morphological post-processing [ 36 ]. In terms of computational efficiency, thresholding-based methods are fast methods, usually taking just a few seconds. Nowadays, however, thresholding is mostly used as a pre-processing or initialization stage, where additional processing from other types of methods is performed later for corrections, as thresholding does not behave well in the presence of abnormalities. Usually, the threshold value is adaptable by some algorithms [ 12 , 14 , 37 ]. Note that many articles perform lung segmentation by thresholding but do not provide quantitative evaluation of the generated segmentations, and thus were not included in this review.

3.2 Region-based Methods

Region-based methods focus on the information contained in the neighboring regions of pixels or voxels, such as intensity, color, or texture, being one of the most popular techniques in the literature before the rise of deep learning due to its low computational cost and unsupervised operation. In general, a disadvantage of the region-based methods is that they can be sensitive to noise and pathology, causing segmented regions to have holes [ 38 ]. For this reason, most methods include some form of post-processing and refinement. Furthermore, these algorithms tend to fail when the initial step of seed position or initial marker is not close to the real target, which can happen in very abnormal lungs.

Region growing is one of the most popular region-based techniques, where an initial automatically selected seed point is grown into the desired form following some criteria [ 39 40 41 42 ]. Watershed transform is also a common technique, either as a pre-processing step or as the core of the method, due to the lung gray-level topology being susceptible to “flooding” approaches, using image features as guidance [ 43 44 45 , 47 , 48 ]. The fuzzy C-means method attributes clusters to regions based on a pre-defined regional feature calculation [ 49 50 51 52 53 54 55 ]. Some methods are based on the wavelet transformation, commonly used to highlight desired frequencies in the image [ 46 , 56 57 58 59 60 61 62 ]. Several methods are mainly based on the region growing principle but include techniques from other types of methods such as Markov random fields, gaussian mixture model, graph cut, unsupervised k-means, Kapur’s entropy, convex hull, random walker and others [ 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 ]

3.3 Shape or Model-based Methods

Model or shape-based methods use a priori knowledge about the target shape and appearance. They can fit statistical models of lung shape or appearance to the image using an optimization procedure. In general, the expected shape and local gray-level structure of a target object in the image are used to derive the segmentation process in such methods. Model-based approaches follow a top-down strategy modeling on global and local variation in shape and texture. Due to their probabilistic nature during the training phase, model-based methods perform better in treating mild to moderate abnormalities. Obtaining a representative model that represents the region of the organs to be delineated is often difficult, and these approaches can be computationally expensive [ 2 ].

Shape-based methods can be specialized to the noticeable line shape of fissures [ 95 96 97 ], or the circular nodule shape [ 98 99 100 ]. Lung parenchyma segmentation can be modeled using cost functions and probabilistic models, exploring known anatomical landmarks and patient specific shape knowledge [ 16 , 100 101 102 103 104 105 106 107 108 , 122 ]. Prior contours and shapes can be adapted to the intended target using the active contour approach, where their form is iteratively guided by an energy function [ 15 , 109 110 111 112 113 ]. Also included in this classification are atlas-based methods, where the input is registered in one or multiple atlases representative of the problem [ 16 ]. An atlas is made up of a model with CT images and corresponding labels of the thoracic regions. In the case of multiple atlases, label-fusion is also employed [ 114 115 116 117 118 119 120 ].

3.4 Neighboring Anatomy-guided Methods

Neighboring anatomy-guided methods use the spatial context of neighboring anatomic objects of the lung, such as the rib cage, heart or spine [ 13 , 123 124 125 ], for delineating lung regions. For lobes and fissures, internal bronchi from the airway tree and vessels are used as guidance [ 126 127 128 ]. The main purpose of this approach is to restrict the search space of the ideal boundary and remove false-positive findings. These methods are well suited in the presence of an extreme abnormality or an imaging artifact 2 . In contrast, the effectiveness of this approach depends on the assumption that there is no abnormality in the neighboring structures of the lung, which can be difficult to guarantee. Finally, the larger the search space, the slower the algorithm execution time [ 129 ].

Some authors used thresholds together with methods based on neighboring anatomy guidance [ 129 , 130 ]. The segmentation of damaged lungs with thresholding and region growing can be improved by prior separation of neighboring air-like voxels [ 131 ]. Prior segmentation of neighboring organs such as the spleen, kidneys, trachea, bronchi and liver also helps in lung delineation [ 125 , 132 ]. Distance transforms are commonly used for the separation of lung lobes due to their anatomical relationship to vessels and bronchi within the lung [ 126 127 128 , 133 ]. In lung segmentation, prior segmentation of the human airway tree and ribs can help smooth and constrain lung boundaries and also help with lungs affected by parenchymal diseases [ 123 , 134 , 135 ]. Statistical models can also consider contextual constraints from neighboring anatomy and internal blood vessels for better segmentation of abnormalities [ 13 , 124 , 136 ].

3.5 Machine Learning-based Methods

Machine learning methods attempt to automate the construction of analytical models by learning from data and identifying patterns. New developments since 2015 lead us to diverge from the proposal of Mansoor et al ., [ 124 ] regarding the categories of methods for lung assessment in CT and we propose splitting the machine learning category into traditional machine learning and deep learning.

Traditional Machine Learning

Traditional machine learning uses what has been coined in the literature as learning from “feature engineering”, where expert knowledge is used to propose functions that extract relevant features from regions of the input image. Application of these types of methods for lung imaging started with the idea of texture classification, such as improving a rough k-means initialization with voxel classification of the uncertain border using 3D [ 138 ] or 2D [ 139 ] gray level co-occurrence matrix (GLCM). Local binary patterns, wavelets and gray level statistics are also features used for voxel-level classification with the objective of segmenting ground-glass nodules [ 140 ]. GLCM of texture features together with post-processing based on airway removal as anatomical constraints has been used for lung segmentation of ILD patients [ 141 ]. Random forest models have also been proposed for lung tissue classification [ 142 ], over a variety of 2D and 3D features [ 17 , 143 ].

Deep Learning

In deep learning [ 145 ], input data goes through deep layers, which learn hierarchical features, starting from low-level to more abstract representations. Convolutional neural networks (CNNs), a type of deep neural network, have been widely applied in problems dealing with medical imaging with great success, with the major advantage of not requiring expertly engineered features and providing fast prediction times [ 146 ]. The most common application of a CNN for any type of lung assessment is to train encoder-decoder segmentation architectures inspired by UNet [ 147 ] with supervision from a large, annotated dataset cohort in an end-to-end fashion, to segment a target structure. This supervision happens in the form of loss function, in most cases related to minimizing overlap between the network output and a ground truth segmentation [ 148 ], although voxel classification losses such as cross entropy are also used [ 149 ]. A major disadvantage of supervised learning is the necessity of large amounts of varied and annotated data, which results in data augmentation strategies being commonly used to increase the available amount of data [ 150 ]. Another disadvantage is the need for expensive computing power and training time on the scale of days. Even considering these disadvantages, deep learning-based methods are the de facto state of the art in the lung segmentation field, including internal findings. For lung parenchyma segmentation, clinical studies have found no significant difference between expert lung annotation and deep CNN results [ 151 ].

Many variations and small modifications of the original UNet architecture have been proposed [ 33 , 152 153 154 155 156 157 158 159 160 161 162 163 164 165 ], including 3D variations coined VNet or 3D UNet [ 166 167 168 169 170 171 172 173 174 175 176 177 178 179 ] that are able to process cube patches. Some research fuses features or results from multiple views (2.5D) or multiple 2D and 3D networks attempting to capture information from different angles and dimensionalities [ 180 181 182 183 184 185 186 187 188 189 190 ]. Although UNet is prevalent in the literature, different architectures originated from the field of natural imaging segmentation, such as SegNet, DeepLab and Region CNNs, are also employed. Some traditional techniques, such as SVM, K-Means and GMMs, are also sometimes involved [ 191 192 193 194 195 196 197 ]. For the input to these methods, most research uses patches of a pre-processed acquisition, normally consisting of HU intensity normalization. However, it is possible to go further by using coarse initial segmentations from other methods and unconventional inputs such as frequency decompositions, multiple HU clipping, coordinates as input and even by allowing for user corrections [ 144 , 198 199 200 201 202 203 ]. Post-processing of the output is also commonly employed, such as with conditional random fields or mathematical morphology [ 204 205 206 ].

Most research modifies the original architecture with novel propositions attempting to improve performance. One common modification is to change the type of convolution [ 186 , 207 208 209 210 211 212 ]. Additional residual connections and the use of attention gates [ 213 214 215 216 217 ] are also a common modification, and even temporal features in the form of convolutional long short-term memories have been employed to exploit spatial-temporal information [ 218 ]. Note however that recent research indicates that all those variations have not been shown to improve the performance of an UNet-like network in all cases, and likely only result in improvements for specific scenarios, with “no new UNet” (nnU-Net) having recently won many medical imaging challenges by only using a traditional well-trained 3D UNet [ 219 ]. Recent research has also shown that for lung parenchyma segmentation, the problem is solved more by data diversity than by network architecture [ 4 ].

Recent research has also been focused on how to better use annotation data and multiple representations of the same input, with more efficient networks and consideration of low confidence predictions [ 220 221 222 223 224 225 226 ]. Generative adversarial networks (GANs) are also gaining space in the literature, where a discriminator performs an auxiliary role in optimization by trying to discern predicted and ground-truth segmentations [ 227 228 229 ]. The main disadvantage of the fully supervised approach is the necessity of high quality and quantity of annotated data. This drawback has spurred a recent trend in the field of exploring semi-supervised and self-supervised approaches that try to also learn from uncertainty measures and sources other than manual annotations, attempting to reduce the impact of low-quality annotations [ 230 231 232 233 234 235 236 237 ]. Finally, some deep learning medical research and decision support methods have been tested and implemented recently in a real-world context [ 238 239 240 241 ], with promising results for the future clinical use of deep learning methods.

4 Results

This section presents some results from highlighted papers, a list of methods with some level of reproducibility and overall statistics. Additionally, we also took note of all used public datasets, to provide a public data reference for future work. Due to the sheer number of articles included in this review and space constraints, full tables containing details of evaluation metrics, all methods, number of involved patients or slices, and extracted quantitative evaluation are provided in the Supplementary Material .

4.1 Public Data

During the extraction process, we set forth with the goal of not only extracting quantitative evaluation metrics but also finding which public datasets have been used in the literature ( Table 1 ). Note that some datasets provide 2D slices, while others provide the whole scan. Most methods that used supervised training prepared training and test splits of 80/20% or used cross-validation with k-folds [ 248 ]. More recent methods are more likely to involve multiple data sources for more robustness of trained methods [ 4 , 235 ].

Table 1.

Public datasets of thorax CTs, with description, links, and provided annotation. Includes only publicly available datasets mentioned in the reviewed methods, that we could find a working link for.

4.2 Highlighted Methods

It is noticeable that better quantitative evaluation metrics [ 247 248 249 250 ] are not correlated with robust and well-validated methods and many articles with superior reported metrics used less representative datasets. Therefore, we chose to highlight five papers to represent the state of the art by target structure ( Table 2 ). These are not necessarily the best quantitative values reported, but those that caught our attention in a more subjective manner, with innovative design, extensive validation through large data annotation efforts, and generalization capabilities. The reason for being included in this table is described in the “Highlight” column. Note that metrics are defined, and all other papers have their results and data information in the Supplementary Material .

Table 2.

Highlighted methods selected to represent each target structure group, with the reasoning summarized in the “Highlight” column, data, and evaluation information. ASSD: Average Symmetric Surface Distance, DSC: Dice Similarity Coefficient, HSD: Hausdorff Surface Distance, JSC: Jaccard Similarity Coefficient, LIDC: Lung Image Database Consortium, MSD: Maximum Surface Distance, PR-AUC: Precision-Recall Area under Curve.

For lung parenchyma segmentation, highlights include Konar et al ., [ 235 ], who proposed a new self-supervised quantum activation for shallow learning, Gerard et al ., [ 220 ] with an interesting polymorphic training strategy that allows for learning from different annotation complexities of the same target, Hofmanning et al ., [ 4 ] with an extensive exploration of the real contribution of architectures, suggesting that data variability is more important. Chen et al ., [ 91 ] and Sousa et al ., [ 84 ] achieved competitive performance with the state of the art with region-based methods. From the conclusion of many recent methods, automated lung parenchyma segmentation is a solved problem.

For pulmonary nodules, Tavakoli et al ., [ 73 ] and Chung et al ., [ 105 ] showcased outperforming deep learning methods with region and shape-based methods, respectively. Liu et al ., [ 30 ] transferred knowledge from classification learning to segmentation learning. Aresta et al ., [ 203 ] provided a way for physicians to correct the result with manual interaction after the automated results, with Cui et al ., [ 178 ] being an interesting recent application of VNets with very competitive results in volumetric evaluation. Recent methods achieve between 0.7 and 0.9 DSC.

For fissures or lobes, Konietzke et al ., [ 133 ] performed an evaluation with expiration imaging and pediatric imaging using neighboring anatomy guidance. Gerard et al ., [ 221 ] used a large amount of annotated data and semi-automatic ways to very high fissure AUC. Ram et al ., [ 232 ] had an interesting use of uncertainty for automatic quality assurance of resulting segmentations. Lessmann et al ., [ 240 ] besides segmenting the lobes also quantified COVID-19 findings. Zheng et al . [ 178 ] improved the now common VNet architecture with deep supervision and attention. Recent methods achieve upwards of 0.9 DSC.

Mansoor et al . [ 16 ] is still a representative work for direct pleural effusion segmentation using a more traditional spatial context learning shape-based method. A lot of recent works on internal abnormalities of the lung focus on COVID-19 segmentation. Yan et al ., [ 226 ] is one of the works that used the largest amount of annotated data. Zheng et al ., [ 136 ] used a small amount of data but were able to perform the segmentation in a completely unsupervised manner.

Wang et al ., [ 241 ] presented challenges encountered in deploying the segmentation method to a clinical setting. Chatzitofis et al ., [ 217 ] besides achieving good segmentation metrics also provided annotations. Recent methods are achieving upwards of 0.7 and 0.8 DSC for abnormal lung findings segmentation, on average.

Finally, we listed methods that provided some form of reproducibility and open-source code ( Table 3 ).

Table 3.

Methods with their respective links to code repositories, evidencing their efforts towards reproducibility.

Global statistics are able to provide an overview of the literature, such as the fraction of methods that included pathological lungs, COVID-19 patients, publication methods (conference, journal), and data availability ( Fig.3 ). From the 224 articles included in the extraction, in 34 cases the authors worked with datasets including COVID-19 information, not necessarily in all patients in the dataset. In addition, 200 articles used datasets with pathologies in their data. Only two articles did not have pathologies in their datasets and in 22 articles, the authors did not specify whether the data included pathology. Of all the articles, 77 were published in a conference, 146 in a journal. Finally, 86 articles used only public datasets, 101 articles used only private datasets, and in 37 articles, authors used both public and private datasets. The increase in deep learning after 2016 and findings after the start of the COVID-19 pandemic is noticeable, when observing the timelines of the distribution of method categories and target structures over the years ( Fig.4 ).

Fig. 3.

Statistics by category of method for methods involving COVID-19 patients, publication method, the inclusion of pathological lungs and data availability.

Fig. 4.

Timelines of the number of publications included in the extraction phase, for method categories (left) and targets (right).

5 Discussion

Our systematic methodology has the main advantage of providing a deterministic and reproducible review. With the provided databases, dates, search keys and selection criteria, anyone can reproduce our results. However, this strategy does come with disadvantages. Research that has not been indexed on the chosen databases are not included, including arXiv publications. To keep reproducibility, we cannot use dynamic search databases such as Google Scholar and Semantic Scholar. Also, the extensive manual reviewing, consensus and extraction work required to perform all phases of the methodology also limits the number of papers that can be feasibly included, which led to our decision to limit the search only to automated, computed, and quantitatively evaluated methods. This indirectly eliminated older methods without segmentation ground-truth and favored the inclusion of recent deep learning methods which, by the nature of requiring ground-truth targets for training, tend to provide evaluation with segmentation metrics.

From the extraction process, we noticed many points of discussion, now with an overview of past and current segmentation methods. An interesting change in the research workflow can be noted with the rise of data-driven models. While data used to be a secondary focus of the research, only useful for algorithm validation, it is now the central point of many papers. Unfortunately, sharing patient data is complicated in the medical imaging field, which results in studies using in-house acquisitions and annotations from local radiologists not made available to the public. Regarding publicly available data ( Table 1 ), although there are a large number of CT scans available, it is common to annotate a subset of a publicly available dataset without making the annotations public, thus limiting the number of public annotations. The most commonly found public annotations are for lung masks, pulmonary nodules, and now COVID-19 findings. The remainder of this section will discuss methodologies, the state of the art regarding each target, and some gaps and opportunities for future research.

5.1 Methodology Trends

Threshold and region-based approaches are relatively easy to run in terms of computation but susceptible to noise and abnormalities. Shape/model, neighboring anatomy and machine learning approaches might be computing heavy and require more pre-definition of models and atlases but are more robust to noise and abnormalities. All these approaches require prior knowledge of the problem and hand-crafted tuning of parameters. Deep learning-based segmentation networks, on the other hand, can achieve better baseline performance and processing speed than more traditional methods, without the need for hand-crafted features, but requiring more data annotation. Note, however, that among the state-of-the-art deep learning-based methods, proper data collection and processing have been shown to be as important as the deep architecture [ 4 ].

In general, modifications to encoder-decoder segmentation networks provide the best overall performance. These modifications are often inspired by architecture changes proposed in the natural image segmentation literature. Even though these architectural modifications can bring improvements to the medical imaging segmentation performance, it has been shown that pre-training in natural images does not necessarily translate to better training in medical images. Regarding other types of methods, they are still being proposed and providing competitive performance in all included target structures [ 73 , 84 , 246 ]. Due to the deep learning requirements for high quality and varied annotated data, traditional techniques are starting to be used again in deep learning pipelines for regularization and semi-supervised learning [ 234 ]. More future improvements might come from exploring the benefits of traditional techniques in conjunction with deep learning. Finally, we noticed recent developments in leveraging border uncertainty information in learning [ 232 , 233 ], trying to reap benefits from annotation variability and make learning suffer less from poor quality annotations.

5.2 Targets

For lung parenchyma segmentation, the literature is at a point where deep learning-based methods are reliably good in many different domains, even when involving diseases that completely change the lung appearance [ 226 ], with publicly available command-line interface tools validated by both the authors [ 4 ] and the public ( Table 3 ). This stability facilitates the development of future unbiased methods for internal findings that can focus on the lung area, leveraging lung extraction using these publicly available tools and codes.

Pulmonary nodule segmentation is currently following the same trend as deep segmentation networks [ 154 , 178 ], in some cases with pipelines for simultaneous detection and/or classification [ 149 ]. Manual volumetric nodule segmentation is still challenging, considering time and reproducibility issues, which resulted in the RECIST nodule evaluation method used in medicine where only one slice is used [ 230 ]. For automated segmentation, the achieved overlap metrics have been comparable to human interobserver variability ( Table 2 ) but are not likely to reach upwards of 0.9 DSC in large data cohorts with the current difficulty in manual annotation reproducibility.

Fissure and lobe segmentation has always been a challenge when fissures are not visible due to pathology [ 6 ] since most region-based methods do not allow fissures to be seen as edges. However, recent deep learning methods were able to teach networks to extrapolate the location of fissures in diseased lungs by training on large cohorts of data, reaching high lobe overlap metrics [ 221 ]. With more robust lobe segmentation methodologies, automated characterization of disease by outcome localization becomes more feasible [ 240 ].

In our search, only six publications classified as findings dealt with targets from diseases other than COVID-19 [ 16 , 42 , 74 , 107 , 111 , 129 ]. Many articles that dealt with diseases such as ILD and pleural effusion have instead focused on lung parenchyma segmentation. This suggests that research in automated segmentation of findings from diseases other than cancer and COVID-19 pneumonia needs further investigation. We have the hypothesis that this is due to the lack of publicly available annotated data ( Table 1 ). For the segmentation of COVID-19 related findings, many studies surfaced with the COVID-19 pandemic ( Figure 4 ), alongside private annotation efforts and some public datasets ( Table 1 ). For now, most methods operating on significant amounts of data achieve upwards of 0.7 to 0.8 findings DSC ( Table 2 ). Studies are required in the interobserver variability between humans providing manual annotations for COVID-19, given the fact that visual inspection of public datasets reveals noticeable differences in annotation protocol. We believe this is one reason why current methods do not achieve higher DSC.

5.3 Gaps

Some gaps noticed in the literature are as follows, in the authors’ opinions. Firstly, in many cases data is not described correctly, with missing relevant information such as if the data consists of 2D slices or 3D volumes. Secondly, when reporting evaluation metrics, most authors do not provide information whether they were performed per scan (3D metric) or slice. Additionally, current quantitative evaluation does not allow for quantitative comparisons between methods, unless performed on the same dataset and data split, which is rare. Especially now, when many methods are achieving upwards of 0.9 DSC in some targets, it is important to be able to differentiate if a new methodology is performing well because of overfitting to learned data or if it is robust and general. Perhaps the use of unsupervised, data-independent techniques could provide the robustness that is lacking when most research is based on human-annotated data, especially in cases where annotation variability is high. Annotation efforts need also to be supervised for quality and to follow proper protocols, avoiding poor quality annotations and, as a result, avoiding poor quality models.

Another point being discussed in many other areas of academic research is that of reproducibility [ 249 ]. The great majority of the studies involved in this review did not provide an easy way to reproduce their results, not providing open-source code, private data, nor any details about how public data was used. Recent publications, mostly on deep learning, have improved in this regard ( Table 3 ). However, for medical imaging, there are also subject privacy and ethics to consider, which makes the reproducibility effort even more challenging. We suggest that future contributions aspire to provide open-source implementations and at least validations on known public datasets, with instructions for reproducibility.

Finally, we noted that no method was demonstrated to automatically segment all types of findings at the same time. The articles included in this review all focused on specific tasks generally related to specific datasets or challenges. Therefore, there is no general automated method for lung assessment and segmentation prepared to deal with all common types of lung findings, for example for use in a clinical setting. We are happy to see a recent effort to deploy deep learning-based methods for COVID-19 classification and segmentation in real hospitals and its use in medical research, with promising results [ 238 239 240 241 ], but surveys state that deep learning-based methods are not ready for clinical use [ 7 , 8 ].

6 Conclusion

We presented an extensive systematic review of automated lung segmentation in CT images, answering the research question: “What are the quantitatively evaluated, computed, and automated segmentation methods for the lung and its lobes and findings, using computed tomography images?”. We also grouped all included work into method categories and provided a list of public datasets used by these methods.

The state of the art is undoubtedly data-driven deep learning methods, although there are still recent high-quality propositions using traditional methods. During the review process, we noticed that methods should be evaluated based on their robustness and generalizability, rather than on good quantitative metrics. Good metrics can be achieved with methods that are overfitted, due to low variability or quality of annotations.

One of the consequences of the rise of data-driven methods is the increased dependence on high-quality data annotation. This has instigated newer methods to strive for semi-supervised approaches such as the use of regularization by unsupervised algorithms and the exploitation of border uncertainty to their advantage. As we approach performance close to annotation by radiologists, how to prove that a method is robust enough for clinical use while considering the variability of manual annotations used as the basis for these methods is an interesting research question for future studies.