Removing Radiographic Markers Using Deep Learning to Enable Image Sharing

See also the article by Khosravi and Mickley et al in this issue.

Ken Chang, MD, PhD, is a diagnostic radiology resident at Stanford University. He completed his PhD at the Martinos Center for Biomedical Imaging at Massachusetts General Hospital/Harvard Medical School, developing artificial intelligence techniques to enhance medical imaging workflows. His research interests include distributed learning, response assessment, and radiogenomics.

Matthew D. Li, MD, is an assistant professor in the department of radiology and diagnostic imaging at the University of Alberta and a radiologist at MIC Medical Imaging in Edmonton, Alberta, Canada, with a clinical specialization in musculoskeletal radiology. His research focuses on the use of artificial intelligence for assessing disease severity and change over time in medical imaging.

For medical imaging artificial intelligence (AI) algorithms to be clinically useful, they must be generalizable across various patient populations, radiology device vendors, image acquisition parameters, and clinical settings. The performance of AI algorithms can improve with increased dataset sizes (1,2), data from multiple institutions, and greater diversity (3), which can be achieved by sharing datasets across institutions. However, before datasets can be shared, they must be curated to remove protected health information (PHI) that could compromise patient privacy. Dataset anonymization is a labor-intensive task, which hinders the ability to share or publicly release experimental data. Other than studies using public domain medical imaging datasets, the radiology AI literature has few instances of data sharing (4). Increased sharing of datasets would allow improved collaboration and reproduction of research findings and facilitate public release of more high-quality datasets, which in turn could catalyze further AI algorithm development and validation.

Khosravi and Mickley et al (5), in this issue of Radiology: Artificial Intelligence, offer a proof of concept to anonymize medical images to lower the barriers for image data sharing. The authors trained a YOLOv5-x model, a widely used object detection deep learning model, to detect patient-identifying information in radiographs of the hip or pelvis. Importantly, the authors used a two-step approach to retain information—in their specific use case, the burned-in laterality markers on radiographs—that may be useful to train AI algorithms on the dataset.

The authors demonstrated 100% accuracy of their AI model on an internal test set and 96% accuracy on an out-of-domain set of chest radiographs from the publicly available CheXpert dataset (6). The authors highlighted some notable failure modes on the external test set, including the false-positive detection of electrocardiography leads. Using a tuning set of only 20 chest radiographs, the authors were able to increase external test set performance to 99.6%.

This study used a two-pass approach to remove PHI and selectively retain laterality markers. A natural next step for this approach could include removing PHI while retaining other diagnostically important radiographic markers that indicate radiographic positioning or technique (eg, supine, decubitus, posteroanterior, weight bearing). Such information could aid AI applications, such as the detection of pneumoperitoneum on abdominal radiographs, joint space narrowing on knee radiographs, or cardiomegaly on chest radiographs. One could extend this anonymization approach to other imaging modalities where burned-in markers may be present, such as US, CT, MRI, or nuclear radiology.

AI algorithms similar to this work could form part of a larger, multicomponent anonymization pipeline to enable dataset sharing or public release. AI-based large language models have been used to anonymize radiology reports (7). Automated anonymization of both images and reports would be a powerful tool. The anonymized radiology report could provide information to train an AI model on the image data with decreased manual labeling. Such datasets could facilitate unsupervised or weakly supervised learning.

Although the potential of automated radiology data anonymization is tantalizing, the performance of such models, including in this work, on external test data may be near perfect, but not perfect. How would AI anonymization algorithms be implemented in real-world use cases? The release of even a single individual's information would be a problem for both the individual and the institution from which the data originate. Given this concern, such AI tools could be used in tandem with a manual reviewer or as a first reviewer in a two-reviewer schema, with calibration of the AI model for high sensitivity to ensure that no PHI is missed, at the cost of specificity. Although not completely automatic, such an approach would greatly decrease the time and expense of manual review of imaging data.

There are important caveats to this study. First, the authors showed improved performance with in-domain fine-tuning on a small dataset. Requiring fine-tuning for every use case, even if the tuning dataset is small, may prevent many researchers from choosing to take on the task of anonymization for data sharing. Second, many possible modes of dataset shift could affect the performance of the anonymization AI algorithm, including previously unseen medical devices (such as the electrocardiography leads in this study) and changes in image contrast, patient positioning, image orientation, or the font or shape of the markers. Checking of the algorithm's performance will remain a necessary safeguard to prevent inadvertent sharing of incompletely anonymized image data. Third, the authors evaluated only a single deep learning object detection algorithm in a landscape of many similar or superior detection algorithms, as the YOLOv5-x model was released originally in 2020. Improvements in AI models have proceeded rapidly since then, and algorithm performance may be improved with newer model architectures and training strategies. Last, only a single modality and two anatomic regions were considered in this study, while coverage of many more modalities of all bodily regions are needed.

Nonetheless, the advances reported here are a quality proof of concept of the anonymization process to enable medical image data sharing and public release.