Performing a Research Study Using Open-Source Deep Learning Models

Publications on deep learning (DL)-based models for medical images are rapidly increasing. Some scientific journals now mandate that authors upload computer codes and/or models (e.g., weights) to publicly accessible repositories, such as GitHub, Bitbucket, or SourceForge. According to a blog post from the Public Library of Science [1], publicly available codes enhance understanding, support reproducibility and reuse, and increase efficiency across the entire scientific ecosystem. The development of prediction models, particularly DL models using medical images, requires considerable resources (e.g., data, labor, and time). Specifically, data collection involves image retrieval from a picture archiving and communication system, cleansing erroneous images and noisy labels, and sometimes manual or automatic lesion annotations, which are often labor-intensive procedures. Therefore, reusing the developed models for validation or clinical deployment could be an efficient research strategy compared to the repetitive development of multiple similar models. In this editorial, I share research examples that validate open-source DL models.

External Validation of an Open-Source DL Model for CRs

Chest radiographs (CRs) are among the most widely used imaging examinations globally, and their extensive availability facilitates the early application of DL algorithms in this domain. Common DL models include segmentation and detection algorithms for lung nodules, masses, consolidations, pneumothorax etc. [2,3,4]. However, CRs may contain prognostic information beyond the traditional diagnostic findings, and DL models can effectively quantify this prognostic signature. For instance, Lu et al. [5] recently developed a convolutional neural network capable of predicting the long-term incidence of lung cancer for up to 12 years using publicly available CRs from a large randomized controlled trial. Their objective was to identify high-risk smokers for lung cancer CT screening. The model exhibited superior discrimination performance compared with that of the Centers for Medicare and Medicaid eligibility criteria in independent test sets.

Along with my colleagues, I conducted an external validation study of the model developed by Lu et al. [5] considering that the selection criteria for lung cancer CT screening are important in optimizing nationwide CT screening programs in Korea. The model was downloaded from the Github repository (https://github.com/vineet1992/CXR-LC), and image preprocessing was performed for CRs in accordance with the authors’ instructions. In a retrospective analysis of 19488 individuals undergoing health checkups in Korea, the model showed good discrimination performance, and we demonstrated its added value to the 2021 United States Preventive Services Task Force recommendations (i.e., an update of the Centers for Medicare and Medicaid eligibility criteria) [6]. The model proved to be useful in reducing the number of screening candidates while maintaining the inclusion rate and positive predictive value for incident lung cancer [6]. Although the model was originally developed as a potential replacement for the Centers for Medicare and Medicaid eligibility criteria, we intentionally extended its application to test its added value against the updated criteria. It is worth noting that a prediction model can not only be validated precisely according to the target population and scheme of the original model development study, but can also be tested in an intentionally different target population or clinical workflow [7].

A User-Friendly Example Using Google Colab

A user-friendly example of an open-source DL model is available. Weiss et al. [8] published a CR-based DL model to predict lung-related mortality (https://github.com/AIM-Harvard/CXR-Lung-Risk). As in the previous example, this model was developed and validated using publicly available datasets from randomized controlled trials. The authors offered three ways to run their model: 1) a cloud-based approach, 2) a Dockerized version, 3) and a local setup. The cloud-based approach provided the code for the environment setup, data preparation (including preprocessing with sample images), and model inference using Google Colab (https://github.com/AIM-Harvard/CXR-Lung-Risk/blob/main/notebooks/cxr_lung_risk_mwe.ipynb), a cloud-based Jupyter notebook environment. Users with minimal coding proficiency can follow the codes provided to obtain model outputs using their CRs. External validation studies can be performed on various populations, including individuals undergoing cancer screening, patients with chronic pulmonary diseases, and those who have recovered from COVID-19 pneumonia. As mentioned previously, prognostic signatures extracted using DL models can be tested in intentionally different target populations.

Segmentation Models

Numerous open-source DL models are available for use in various imaging modalities and tasks. For example, DL algorithms have greatly advanced body segmentation [9,10,11,12], enabling the accurate quantification of organ dimensions. This allows the analysis of the area or volume of organs, fat, and muscle as imaging markers for diagnosing diseases and predicting patient outcomes. Segmented organ images can be used as inputs for separate DL models. The open-source models are https://github.com/QIMP-Team/MOOSE and https://github.com/wasserth/TotalSegmentator.

Technical Tips for External Validation of Open-Source DL Models

A caveat is the need to check whether the DL model inputs or outputs require transformation. Failure to apply the necessary transformations can lead to incorrect model outputs and reduced performance. Similarly, it is essential to verify and adhere to the image preprocessing steps, including windowing, cropping, and resizing.

External validation studies employing open-source DL models usually do not require cutting-edge graphics processing unit-equipped workstations. The DL models for CRs introduced in this editorial can be executed on Google Colab. However, if the number of images exceeds a few thousand, it is advisable to opt for a local setup after downloading the model weights.

Key Steps for Utilizing Open-Source DL Models from Public Repositories

1) When reviewing research articles related to prediction models, assess whether the models and/or their weights have been shared on public repositories.

2) Clone open-source models to your local repository or directory.

3) Create your dataset, which includes images for external validation.

4) For external validation, one may choose to strictly adhere to the target population and indications specified in the original study or intentionally explore broader applications for the model.

5) Follow the data preprocessing steps, which may involve variable transformations, employed in the original model development study.

6) Assess whether model calibration or recalibration is required.

CONCLUSION

External validation and/or an extended application study of open-source DL models is prudent and highly efficient. It not only reduces time and cost investment but also leverages the collective knowledge and expertise of a broader community.