Automated neural network femur segmentation performance in computed tomography images of older adults with obesity

Abstract

Femur segmentation is a precursor to image analysis pipelines that evaluate hip bone measures with subject-specific finite element models, but historically required time-intensive efforts of operators. Implementing deep learning techniques offers a promising pathway for rapid, automated, and accurate segmentation. This study evaluates the feasibility of using a convolutional neural network (CNN) developed and validated for femur segmentation of Icelandic older adults to segment femurs in a sample of 166 CT scans of older adults with obesity in the United States. The performance of the segmentation model was quantitatively evaluated against manually segmented ground truth data using the Dice similarity coefficient (DSC) and the 95% Hausdorff distance (HD95), and qualitatively evaluated using manual image review. The mean DSC was 0.974 ± 0.009 and mean HD95 for the foreground was 1.078 ± 0.41 mm, indicating excellent segmentation quality. The CNN segmentation model averaged 32 ± 3 s for each mask prediction. Visual inspection revealed segmentations with minor errors in delineating boundaries at the femoral head and detecting osteophytes, requiring refinement during post-processing. The present CNN, when evaluated on this dataset of older adults with obesity, produced femur segmentations with improved quality and speed compared to manual segmentation. Additional model training with a more diverse training dataset could help further minimize manual intervention that is currently required during pre- and postprocessing. This study demonstrates the potential for femur segmentation CNN models to be applicable to diverse clinical datasets.

Introduction

The global population of older adults is predicted to rise in the years to come, with the number of individuals over 65 yrs old expected to double to 1.5 billion by 2050.¹ In parallel, hip fractures are predicted to increase in prevalence among these individuals, which is concerning, because fractures in this region have devastating impacts on quality of life and elevate mortality risk.²

Advanced techniques involving biomedical image analysis pipelines that use quantitative CT and subject-specific finite element models are gaining traction as a method to monitor the bone changes, including fracture risk, with aging and treatment-based interventions for bone health.^3–5 Yet, image segmentation, one of the integral precursor steps to modeling subject-specific geometry and extracting bone quality measures in these pipelines, has limited the amount of data that can be processed and analyzed because it traditionally required a time intensive amount of manual annotation by expert annotators.

Recently, several implementations of deep learning methods, notably convolutional neural networks (CNN), have emerged as a promising way to rapidly segment femurs from CT images.^6–9 These methods leverage a neural network architecture of stacked computational layers that detect and extract local features from the imaging data at the pixel level and use training data in learning to recognize global patterns in the spatial data.¹⁰ Collectively, this allows the model to efficiently label imaging data and thereby distinguish the image background from the foreground (femur bone). However, many of the available models still struggle with bone segmentation among femur images of older adults in particular because of the prevalence of aging-related bone characteristics and disease-related pathologies, which may be associated with narrow joint spaces,¹¹ increased presence of bone tissue overgrowth (osteophytes),¹² and inhomogeneities in bone tissue density (enostoses)¹³—all of which render it difficult for deep learning models to accurately delineate the boundaries between bone structures.

Notably, Björnsson et al.¹⁴ recently developed a robust and efficient CNN architecture using the open-source MIScnn software (Medical Image Segmentation with Convolutional Neural Networks)¹⁵ and trained the model on CT data from a large cohort of Icelandic older adults. Among a large evaluation set of 1147 images, this model demonstrated excellent quantitative segmentation performance metrics and appears to be successful at correctly identifying femur boundaries among the thin bone interfaces common in older adults.⁸

However, there is an ongoing need to determine if these models are applicable to other imaging datasets from individuals of different races/ethnicities and body sizes, including images collected from CT scanners with acquisition parameters that differ from what was available in the original training and validation datasets. The objective of this study was to evaluate the feasibility and accuracy of using an existing CNN that was trained on hip CT images from older adults in Iceland in order to efficiently segment proximal femur CT images from a study of older adults with obesity in the United States.

Material and methods

The CT dataset

The UPLIFT (Utilizing Protein During Weight Loss to Impact Physical Function; R01 AG050656; NCT03074643; K25AG058804; NCT03819478) clinical trial at Wake Forest University School of Medicine provided baseline CT scans of 187 enrolled older adults (65-85 yr) with obesity (BMI 30-45 kg/m² or 27-30 kg/m² with an obesity-related risk factor) and limited physical functioning. Individuals who screened positive for osteoporosis (T-score < −2.5 from hip or spine DXA screening) were excluded from the trial due to safety concerns that the weight loss intervention could exacerbate osteoporosis and fracture risk. This study was approved by the institutional review board at Wake Forest University School of Medicine and all participants read and signed an informed consent document.

Helical CT scans of the participant’s dominant hip from the superior acetabulum to mid-thigh were acquired on a Siemens SOMATOM Definition Flash dual source CT scanner (Siemens Healthineers) at Atrium Health-Wake Forest Baptist Medical Center in Winston-Salem, NC, USA. Images were collected with a standard reconstruction filter with the following acquisition parameters: 140 kV, 350 mA, 33 cm field of view, and 0.8 s gantry rotation speed. The matrix size of each scan was 512 × 512, with a voxel size of 0.97 × 0.97 × 1 mm³. A quality assurance phantom and CT calibration phantom (Mindways Software, Inc.) were scanned monthly according to the manufacturer guidelines to monitor operational characteristics of the Siemens CT scanner and ensure standardized collection of images.

CNN model developed by Björnsson et al.

The model utilized in this study was previously developed by Björnsson et al.⁸ using CT imaging data from the Age, Gene/Environment Susceptibility-Reykjavik Study (AGES-RS), a cohort of older men and women (67 yr or older) in Iceland, who were born between 1907 and 1935 and have been followed by researchers to prospectively study cardiovascular disease.¹⁶ For the AGES-RS dataset, images of the hip region were acquired with a Siemens Sensation CT scanner (Siemens Medical Systems) with a standardized protocol with the following parameters as previously described¹⁶: 120 kV, 140 mA, 50 cm field of view, voxel size of 0.977 × 0.977 × 1 mm³, and a 512 × 512 matrix size. A total of 30 CT scans of the bilateral femur region were manually segmented and separated into left and right femurs. This resulted in a total of 60 available femur images with associated ground truth segmentations, each with 512 × 256 matrix size. Björnsson et al. used these 60 images to develop the CNN model, with 54 images used for training and 6 used for validation. Their design uses a 4-layer patch-based 3D U-net with a patch size of 128 × 128 × 128 voxels (determined to be optimal size to capture the entire femoral head), on-the-fly data augmentation, and a batch size of 2. Their CNN model architecture was implemented in Python using the open-source MIScnn software¹⁵ and trained for 300 epochs on a single Nvidia GeForce GTX 1080Ti graphics processing unit (GPU).

Image preprocessing

In order to match the image size expected by the CNN and limit the generated segmentation to the proximal femur region, the original CT images from the UPLIFT participants were manually cropped to a 512 × 256 image size with the medial edge positioned at the center of the pubis bone, the superior edge positioned slightly above the top of the femoral head, and the inferior edge positioned below the lesser trochanter (approximately 135 slices below the femoral head; Figure 1). Any left femurs were then mirrored to the right side to standardize the inputs into the model. These preprocessing steps requiring manual inputs from the user took less than a minute per scan.

Figure 1.

Visual overview of the imaging inputs and outputs for the Björnsson et al. segmentation CNN. The original CT scan (A) is cropped in the sagittal and axial planes during preprocessing (B) and inputted into the Björnsson et al. 2023 CNN segmentation model, producing a mask prediction of the femur in the foreground voxels (C), which can subsequently be rendered into a 3D representation of the femur (D) for future use in applications such as patient-specific finite element models.

Preparation of ground truth dataset

To create a ground truth segmentation dataset, the available baseline CT images were segmented in Materialise Mimics using semi-automated methods and then manually corrected and reviewed by a single expert operator (K.A.G.). Each segmentation was exported as a binary mask, producing images where the segmented femur is white, and the surrounding background is black. Any left femur masks were mirrored to the right side and the binary ground truth masks were cropped using the same parameters as was used for the corresponding preprocessed CT images (unsegmented) for the CNN, resulting in a 512 × 256 image size for each femur segmentation.

CNN application to UPLIFT images

All preprocessed CT images were input into the CNN model by Björnsson et al.⁸ The model ran on our local workstation equipped with one NVIDIA GeForce GTX 1080 GPU and 32 GB of RAM, outputting a binary mask for each segmentation prediction. A visual overview of the end-to-end image processing pipeline is depicted in Figure 1.

Evaluation of segmentation predictions

Accuracy of the foreground segmentation predictions produced by the CNN was quantified using 2 evaluation metrics which are standard to the biomedical image segmentation field¹⁷: the Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95). The DSC is a similarity metric based on pixel overlap, which determines the pixel-wise agreement between a predicted segmentation and the associated ground truth according to eqn 1, where X and Y are the ground truth and predicted mask, respectively.¹⁸ A DSC score of 1 corresponds to a perfect overlap and a score of 0 to a complete absence of overlap.

1

Meanwhile, the HD95 is a surface distance metric that quantifies the largest segmentation error by measuring the contour distances between the ground truth and prediction mask.¹⁹ This metric was measured in mm and calculated according to the implementation in the MedPy package.²⁰ Lower HD95 values indicate better segmentation accuracy. The 95th percentile is used because the original HD measure is sensitive to the inherent outliers present in medical imaging.

After generating the femur mask predictions from the model, a single expert annotator (K.A.G.) visually evaluated the quality of every segmented volume with respect to the presence of internal holes, incomplete sections of the segmentation, excess capture of bone, and the handling of osteophytes surrounding the bone. Descriptive statistics for the performance measures were calculated in R and data visualizations were produced using the ggplot2 package.²¹

Results

Study population and CT dataset

Participants in the study were on average (mean ± SD) 72 ± 4 yr of age, had a baseline BMI of 34.0 ± 4.2 kg/m², 67% were female, and 26% were non-white (24% Black/African American; 2% Other). Of the enrolled participants, 186 had a hip CT scan at baseline (one individual refused the scan), of which 166 were able to be processed for segmentation. Baseline scans in 20 participants were excluded because of image acquisition errors or insufficient image quality (n = 8) or due to the presence of hardware in one or both hips (n = 12).

Quantitative performance on the UPLIFT dataset

In evaluating the CNN segmentation performance on the UPLIFT CT hip dataset, we determined that the overall accuracy of the model was very good with a mean DSC of 0.974 ± 0.009 and a mean HD95 of 1.078 ± 0.41 mm, with the distributions for each measure shown in the boxplots in Figure 2 and an example of the predicted segmentation mask output is shown in Figure 3. The CNN segmentation model averaged 32 ± 3 s for each femur mask prediction. Examining the distribution of DSC values and HD95 values in the boxplots and using the threshold of 1.5 × IQR, 9 segmentations were flagged as potential DSC outliers with DSC < 0.96, and 19 segmentations were flagged as suspected HD95 outliers with HD95 > 1.38 mm. Nine segmentations presented as outliers in both metrics. These outlier values indicate the predicted segmentations that appear to be of suboptimal accuracy compared to the ground truth.

Figure 2.

Visual representation of the segmentation performance metrics comparing the ground truth mask and the prediction mask from the CNN. Metrics include (A) the Dice correlation coefficient (DSC) and (B) the 95% Hausdorff value (HD95).

Figure 3.

Example of femur segmentation output from the convolutional neural network (CNN) with performance metrics representative of the means from the study. The femur mask prediction from the CNN (red pixels) overlaid onto the ground truth (blue pixels, visible at the outer edge of the contours) is shown anteriorly in the coronal plane (A) and superiorly in the axial plane (B). The 3D representation of the mask prediction is also shown (C).

Qualitative segmentation accuracy results

Upon visual inspection of the predicted segmentations, we report 8 (5%) of the segmentations had sections of the femoral head that were missing (Figure 4A), 33 (22%) contained internal holes (Figure 4B), 4 (2%) segmentations showed areas, where excess bone beyond the contours of the femur surface was captured (Figure 4C), and 78 (47%) omitted osteophytes attached to the femur (Figure 4D).

Figure 4.

Examples of segmentation predictions that did not fully capture the bone morphology. Includes (A) missing sections in the femoral head, (B) internal holes, (C) capturing excess bone beyond the outer boundary of the structure, and (D) not capturing osteophytes.

Discussion

The central objective of this work was to evaluate the performance of the Björnsson et al. automated femur segmentation CNN on CT scans of a racially diverse older adult cohort with obesity in the United States, thereby providing a valuable source of external validation. We demonstrate that the originally trained model rapidly produced femur segmentations with excellent overall accuracy, which provides preliminary evidence that suggests this model may be applicable to imaging data collected from diverse patient populations and acquired by different scanners beyond the training dataset. However, additional future work is still needed to improve the model architecture, challenge model validation with more patient cohorts, and automate the image preprocessing and segmentation postprocessing steps—all of which are integral to the ultimate goal of developing and leveraging a fully autonomous image processing pipeline.

When considering the quantitative performance metrics, the metrics we report here (mean DSC of 0.974 ± 0.009 and a mean HD95 of 1.078 ± 0.41 mm) are comparable to the results from prior studies of other automated femur segmentation techniques, including those based on the graph cut algorithm (average HD of approximately 5.4 mm),²² a combination of thresholding and a watershed algorithm (DSC = 0.972 ± 0.004),²³ and a combination of thresholding and morphing to a template shape (DSC = 0.93 ± 0.02).²⁴ The present results are also similar to those of other CNN implementations for femur segmentation, including that of Balagopal et al. (DSC = 0.95 ± 0.01),²⁵ Deng et al. (mean DSC = 0.978),⁹ and Zhao et al. (DSC = 0.982 ± 0.001).⁶

However, in examining the model’s performance in our study against the previously published metrics reported from validating the CNN model on a subset of the AGES-RS cohort in Björnsson et al., we note that our metrics were slightly inferior to the originally reported DSC score of 0.990 ± 0.008 and the mean HD95 of 0.999 ± 0.331 mm.⁸ Examining the subset of the flagged outliers in our segmentation performance metrics, we noted several similarities between the images—nearly all of them had cases of calcification that was not fully captured by the model (Figure 4D), incomplete sections in the femoral head (Figure 4A), or leakage of the prediction contour into adjacent bones (Figure 4C). Below we highlight one particular segmentation from the model to demonstrate these observations, which had simultaneously the lowest DSC value of 0.91 and the highest Hausdorff distance of 5.28 mm. As shown in Figure 5, we note the adjacent calcification surrounding the bone, the particularly narrow joint space between the femoral head and acetabulum, and what appears to be a clinical presentation similar to a geode in the femoral head. Anecdotally, this participant had a hip replacement less than 6 mo after this scan occurred, likely indicating that the region was impacted by pathology of degenerative or inflammatory joint disease, which is common among older adults with obesity.^26,27 These physical characteristics visible on the CT offer reasonable explanations for the subpar accuracy of the segmentation model in this scan. While the original CNN model was trained on older adults in the AGES-RS cohort, these participants were of a lower BMI (approximately 25 kg/m²),²⁸ which might indicate that during training, the model was not exposed to heavy calcification and narrower joint spaces that often present in images of older individuals with obesity,²⁹ and make differentiating the boundaries of the bone contours particularly challenging. This omission of osteophytes represents a limitation of the present model architecture when considering clinical applications that might require detailed analysis of these localized growths on the bone. In the future, model performance could be improved by re-training on a mix of scans from the AGES-RS cohort and other more diverse patient cohorts, including ours which is comprised of patients with obesity, to be able to better handle a wider variety of bone morphologies associated with different body compositions and optimize noise reduction routines to better capture osteophytes on the outer bone surface. Additionally, the processing pipeline could be improved by including additional automated post-processing methods that incorporate contour filling morphological operations to close small internal holes while mitigating the creation of false connections in the segmentation produced by the model.

Figure 5.

Visualization of the CT scan corresponding to one of the predicted segmentations with the lowest Dice similarity coefficient (DSC) and highest 95th percentile Hausdorff distance (HD95) metrics in the present study dataset. Depicts (A) presentation of excessive osteophytes around the proximal femur that the CNN model prediction (red mask overlay) did not capture, and (B) an abnormal pathology in the femoral neck adjacent to diffuse joint space narrowing, both of which likely explain the associated suboptimal accuracy of the CNN compared to the ground truth in this case.

The segmentation times in our study were also noticeably higher than the reported times in the Björnsson paper (32 ± 3 s vs 11 s), which we attribute to hardware differences of the workstations used to generate the CNN prediction masks in these studies. Inference time reflects our computing setup and may vary depending on hardware and implementation differences. Experimentation on a newer machine with improved GPU and processing capabilities improved our average prediction times to 15-20 s. Nevertheless, we underscore that the speed of the segmentation predictions in this analysis are still far superior to the methods requiring more manual user interaction on which our research group has historically depended. On average, the ground truth segmentations in this dataset required expert operators approximately 5-10 min to complete, which results in many hours consumed by data annotation and introduces inter-operator variability. By contrast, the CNN was able to consistently segment the entire dataset in just over an hour of computational time with no required user interaction during the prediction process.

The primary limitation of applying the current CNN to the UPLIFT data in this study was the non-automated image preprocessing steps that necessitated user input to manually crop the images to specific required dimensions and ensure only a single femur was visible in the images for the model to create accurate predictions. The narrow field of view imposed by our imaging protocol and the requirements for the parent clinical trial to capture all the soft tissue surrounding the dominant leg rendered it challenging to achieve images with a consistent positioning of the femur in the image. In the future, we would be interested in implementing image normalization techniques to automatically locate structural landmarks in the images and register the spatial positioning of the femur, which could then be used to automate the image cropping and mirroring process, thereby minimizing manual user interaction required in the preprocessing steps of the segmentation pipeline. This process optimization would be integral to developing a fully automated end-to-end segmentation tool.

We also note that the UPLIFT study excluded individuals with osteoporosis, which could have influenced the reported performance of the CNN model. Our dataset did not fully challenge the CNN model with a variety of bone phenotypes defined by much thinner cortical regions and low-density trabecular bone. These cases could have introduced compounding partial volume effects, making boundary detection inherently more difficult when defining the contours of the bone mask. In addition, we recognize that the ground truth segmentations used to assess model performance were produced by a single operator, which is another limitation of this study. While the operator was experienced and followed established protocols, this approach may have introduced some bias. Future studies would benefit from including multiple annotators to establish consensus ground truth segmentations and quantify inter-rater variability.

Ultimately, the current study is relevant because it contributes to highlighting the importance of continuing to develop and apply CNN-based femur segmentation at the intersection of clinical research, healthcare, and biomedical engineering. In terms of clinical research, this tool is useful for research groups, including our own, who are working together to develop pipelines for generating fracture prediction tools based on subject-specific finite element models derived from CT images.^4,30,31 Image segmentation is the integral precursor step for these fracture assessment tools, as this process provides us with 3D models of the femur (as shown in Figure 3C) that are converted into a finite element model of uniformly-distributed nodes and elements via a meshing process. We can then further leverage patient CT images to extract bone quality measures, such as volumetric BMD and cortical thickness, and subsequently integrate those measures into the femur models through adjustment of the material properties in the finite element mesh so that they represent the mechanical properties of a patient’s bone. These femur models can then be used to conduct computational simulations of fall scenarios, from which we can extract clinically relevant outcomes relating to bone strength and fracture.

Moreover, the scalability of this segmentation tool for large imaging datasets could be particularly valuable for longitudinal clinical trials involving interventions for musculoskeletal health. Since this CNN performed well on the UPLIFT baseline dataset, it would be of interest to conduct further analysis that characterizes its performance in consistently capturing femur morphology for the purpose of accurately assessing intervention-related changes in cortical thickness and femur strength from imaging data collected at multiple timepoints. There is a wealth of imaging data available through clinical radiology records that could be input into these neural networks to collect quantitative research measures for large population-based studies. This could also be integrated into patient care workflows to opportunistically screen for and treat musculoskeletal disease pathologies³⁰ and thereby attenuate the risk of future hip fractures.

Conclusions

In sum, this study demonstrates quantitative and qualitative performance of the Björnsson model in automatic segmentation of proximal femurs from a population of older adults with obesity in the United States, providing promising evidence in the form of external validation that the model is robust and may be applicable to multi-racial data and different CT scanners. Additional validation across more patient cohorts is needed, but the results of this study serve as an important step towards developing an automated image segmentation tool which can be integrated into clinical research and patient care for fracture risk detection and to help older adults age in place longer with better overall quality of life.

Author contributions

Katelyn A. Greene (Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Visualization, Writing—original draft), Vee San Cheong (Methodology, Resources, Software, Writing—review & editing), Páll A. Björnsson (Conceptualization, Methodology, Resources, Software, Writing—review & editing), Lotta M. Ellingsen (Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Software, Supervision, Writing—review & editing), Alexander M. Baker (Methodology, Project administration, Resources, Supervision, Writing—review & editing), Benedikt Helgason (Conceptualization, Funding acquisition, Methodology, Project administration, Resources, Software, Supervision, Writing—review & editing), Denise K. Houston (Conceptualization, Funding acquisition, Investigation, Project administration, Supervision, Writing—review & editing), and Ashley A. Weaver (Conceptualization, Funding acquisition, Investigation, Project administration, Resources, Supervision, Writing—review & editing)

Funding

This study was supported by the National Institute of Health through funding to Dr. D.K.H. (R01 AG050656), Dr. A.A.W. (K25 AG058804), and K.A.G. (F31 AG069414), the Wake Forest Claude D. Pepper Older Americans Independence Center (P30 AG021332), the Wake Forest Clinical and Translational Science Institute (UL1 TR001420), the Icelandic Centre for Research, and the Strategic Focus Area “Personalized Health and Related Technologies” of the ETH Domain, grant #2018-325, and the National Research Foundation Singapore (NRF) under its Campus for Research Excellence and Technological Enterprise (CREATE) Programme. Future Health Technologies at the Singapore-ETH Centre was established collaboratively between ETH Zurich and the National Research Foundation Singapore. The funding sponsors had no role in the study design, data collection/analysis, or the manuscript writing.

Conflicts of interest

A.A.W., K.A.G., V.S.C., P.A.B., L.M.E., A.M.B., B.H., and D.K.H. declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Data availability

The medical imaging dataset that supports the findings of this article are not publicly available to protect the privacy of the participants in the UPLIFT clinical trial, but a de-identified dataset can be shared upon request. The code can be made available from the authors upon request, with permission from Dr. L.M.E. who maintains the CNN model source code.