Uncertainty-Aware Deep Learning Characterization of Knee Radiographs for Large-Scale Registry Creation

Abstract

Background:

We present an automated image ingestion pipeline for a knee radiography registry, integrating a multilabel image-semantic classifier with conformal prediction-based uncertainty quantification and an object detection model for knee hardware.

Methods:

Annotators retrospectively classified 26,000 knee images detailing presence, laterality, prostheses, and radiographic views. They further annotated surgical construct locations in 11,841 knee radiographs. An uncertainty-aware multilabel EfficientNet-based classifier was trained to identify the knee laterality, implants, and radiographic view. A classifier trained with embeddings from the EfficientNet model detected out-of-domain images. An object detection model was trained to identify 20 different knee implants. Model performance was assessed against a held-out internal and an external dataset using per-class F1 score, accuracy, sensitivity, and specificity. Conformal prediction was evaluated with marginal coverage and efficiency.

Results:

Classification Model with Conformal Prediction:

F1 scores for each label output > 0.98. Coverage of each label output was >0.99 and the average efficiency was 0.97.

Domain Detection Model:

The F1 score was 0.99, with precision and recall for knee radiographs of 0.99.

Object Detection Model:

Mean average precision across all classes was 0.945 and ranged from 0.695 to 1.000. Average precision and recall across all classes were 0.950 and 0.886.

Conclusions:

We present a multilabel classifier with domain detection and an object detection model to characterize knee radiographs. Conformal prediction enhances transparency in cases when the model is uncertain.

INTRODUCTION

Clinical registries are the cornerstone of orthopaedic evidence, yet most lack linked radiographic data. Manual sorting and cataloging of images take considerable time and effort and would be prohibitive for most institutional registries. While image metadata like the study description and the radiographic equipment used is stored in the DICOM header, the available information varies in completeness and accuracy between sites and almost never contains data on image content.

Deep learning (DL) methods of labeling imaging content are an ideal solution for registry creation and have been recently developed for hip radiographs [1]. The DL algorithms are orders of magnitude faster than comparable human annotators and reduce much of the variability associated with manual labeling. A potential concern with DL models is that they are considered “black boxes” where little information is given about how certain the model is about its predictions. Uncertainty quantification, including conformal prediction methods, has been employed across various medical specialties to offer important insights into the reliability of predictions [2-5]. Because radiographic images in a knee registry can vary in quality and presentation due to factors such as patient positioning and image acquisition techniques, conformal prediction can account for such variability by providing a measure of uncertainty that provides transparent confidence scores [6].

In this study, we established an automated image ingestion pipeline for a knee radiography registry of total knee arthroplasty (TKA) patients by linking a multilabel image semantic classifier and domain detector with an object detection model for knee instrumentation. We validated our ingestion pipeline with a held-out internal test set and by using images from the Osteoarthritis Initiative (OAI) for external validation. We also implemented uncertainty quantification with conformal prediction for the image semantic classifier to enhance model trustworthiness and clinical utility.

METHODS

This retrospective study was approved by the institutional review board (IRB: 23010083). All patients included in this study consented to have their data used in research and anonymized images published. The developed knee registry was intended for monitoring and surveillance purposes to complement a robust clinical total joint arthroplasty registry.

Data Source:

Data for training and testing the two models came from three sources. Figure 1 demonstrates the flow of data through the study.

Figure 1:

Data flow through the model training process. A detailed explanation of the inclusion and exclusion steps is given in the method. Overall, some 31,000 images total were annotated. Picture Archiving and Communication System (PACS), Open Reduction and Internal Fixation (ORIF), Anterior Cruciate Ligament (ACL), Osteoarthritis Initiative (OAI)

First, knee radiographs from roughly 26,000 TKA patients whose clinical records were pulled from an existing clinical registry between January 1^st, 2000 and December 31^st, 2020 were extracted from the institutional picture archiving and communication system (PACS) by study description. These images were filtered using the DICOM header modality tags to exclude non-radiography images, yielding 1,295,902 studies. There were 25,000 of these images randomly sampled for manual annotation of the classification task. There were 6,846 of these images that contained implants and were used for annotating the implant detection task at the patient level.

Second, 1,000 knee images were pulled from the OAI database, primarily comprising antero-posterior (AP) and postero-anterior (PA) views, with a small portion of lateral, hip-knee-ankle (HKA), and notch views.

Third, we queried the text of radiology reports in our PACS system for knee images with non-arthroplasty implants that were relatively rare in comparison to native knee radiographs and TKA radiographs. There were 4,995 of these images, and they primarily comprised anterior cruciate ligament (ACL) hardware and nails or plates used for open reduction internal fixation (ORIF) of fractures (resulting in 11,841 total implant identification images).

Manual Annotation

Manual annotation of images is required to establish a ground truth that deep learning algorithms can learn from. We had two categories of tasks: 1) Classification of image content; and 2) Object detection of knee implants. The specific annotation methods for each task are described in the following two paragraphs:

Manual Annotation for the Classification Task:

There were two trained medical student research assistants (EK and AG) who classified 26,000 (25,000 internal + 1,000 OAI) knee radiographs under the supervision of a fellowship-trained knee arthroplasty surgeon. The annotators labeled the images based on the following characteristics: the presence of the left and/or right joints in the imaging field, the presence of prosthesis in the left and/or right knee, and the view of the radiograph: AP, PA, lateral, sunrise, HKA, or notch. In ambiguous cases, such as distinguishing between an AP versus PA radiograph, the supervising orthopaedic surgeon acted as the final authority on the content of the image. As DICOM tags can be unreliable, many radiographs in the set were not of the knee. There were 6,853 of these images, which were marked for exclusion from the classification task and subsequently used in the anomaly detection portion of the algorithm.

Manual Annotation for the Object Detection Task:

The same trained student researchers were instructed by a data scientist and an orthopaedic surgeon to draw bounding boxes around any surgical constructs for 11,841 knee radiographs using the open source LabelMe software [7]. The following constructs were labeled with bounding boxes cropped with a minimal margin to the visible implant: TKA, unicompartmental knee arthroplasty (UKA), distal femur replacement, open reduction and internal fixation (ORIF) plate, ORIF nail, anterior cruciate ligament (ACL) metal screw (i.e., interference screw), ACL metal button, ACL other fixation (i.e., screw and washer fixation devices and other fixation devices), patello-femoral joint (PFA) arthroplasty, static spacer, and fusion. All categories of constructs were further specified by femoral side and/or tibial side, resulting in a total of 20 different labels. Bounding boxes were annotated on both non-occluded and occluded constructs. Occluded cases occurred when multiple constructs within the knee (e.g., ORIF plate overlapping with a TKA) or only partially visible constructs were present within the image. The surgeon provided oversight for all discrepancies about surgical construct type. Images were excluded if the majority of the knee joint was not visible or not present at all.

Model Development

We trained three separate algorithms:

Image Content Classification Model

A total of 26,000 images were used for training, validation, and testing of the classification and out-of-domain detection tasks. 1,000 were from OAI and reserved for external testing. Of the remaining 25,000, 3,000 were set aside for testing and 3,000 for validation, leaving 19,000 images for training. After removing the non-knee images, there were 14,610, 1,623, and 1,914 images in the training, validation, and internal test sets, respectively, for the classification task (Figure 1). A multilabel classifier based on the EfficientNet model architecture was trained to identify the presence of both knees, implants, and the view of the radiograph. The model produces an array of size 9x1, where the first four digits represent the binary output of the presence of the left and right knees and the implants. The last five digits identify the view. We used cross-entropy or binary cross-entropy loss to train the model depending on the dimensionality of the target class. The overall loss for the model was the sum of the individual losses. The model was trained over 60 epochs, where in each epoch the model sees each training image once. We calculated performance metrics after each epoch on the validation set and used that information to tune model hyperparameters. Tuned hyperparameters included: image size, initial learning rate, learning rate schedule and associated parameters, batch size, EfficientNet model, and sampling technique (unweighted versus weighted). Various image augmentations were tested, including random small rotations (up to 15 degrees), random shifts in intensity, and randomly added Gaussian noise. Model performance was monitored by the training and validation loss, weighted F1 score, and per-class accuracy, sensitivity, and specificity. The model weights with the highest F1 score were saved as the top-performing model and subsequently used in testing.

Conformal Prediction:

The validation set (N = 1,623) from training the classifier was used to scale the model outputs and calibrate the conformal predictor. Temperature scaling was implemented before conformal calibration. We implemented a Mondrian Conformal Predictor, a type of conformal predictor that performs better with class imbalances [8]. Several hyperparameters for the conformal predictor were tuned using grid search to a domain-specific level of performance. We calibrated a conformal predictor for each output label of the classifier.

Domain Detection Model

We hypothesized that the EfficientNet classifier trained only on knee images would produce embeddings–abstract features produced by the deep learning algorithm–that an additional classifier could use to distinguish between in-domain (knee) and out-of-domain (not-knee) images. To test this, we added a forward hook to the trained EfficientNet classifier to extract the embeddings produced just before the fully connected layer at the end of the model. We added the non-knee images back into the dataset to get 19,000 training, 3,000 validation, and 3,000 test images. A gradient-boosted classifier was trained on those frozen embeddings to distinguish between knee and non-knee images. Hyperparameters of the gradient-boosted classifier were found by iterating over the options and assessing performance on the validation set. The top-performing model in terms of F1-score and sensitivity in identifying knee images was chosen for testing.

Object Detection Model

An object detection model with the You-Only-Look-Once (YOLOv8) deep-learning object detection architecture was trained to detect different kinds of knee implants. The different categories are listed above in the Manual Annotation for Object Detection Task section. The YOLO family of models treats object detection as a regression task and is designed for fast inference [9,10]. We used a YOLOv8 implementation from the Ultralytics Python package to train and validate the object detection model [11]. A five-fold cross-validation was used during training. In each fold, the model was trained for a maximum of 150 epochs, with early stopping of training triggered if performance on the validation set had not improved after 20 successive epochs. Figure 2 demonstrates an image being processed by the end-to-end algorithm.

Figure 2:

Detail of an image being processed by the end-to-end algorithm. The features extracted by the multilabel semantic classifier are used to determine whether the image is a knee radiograph. The output of the semantic classifier is calibrated and passed through a conformal predictor, which produces prediction sets. Images with identified hardware are passed through an object detection model which localizes implants.

Model Testing

Classification and Domain Detection Models:

There were two forms of testing used. First, the 3,000 labeled radiographs from our institution set aside during training and validation were used as an internal validation set. Second, 1,000 knee radiographs from all available views were extracted from the OAI public dataset to function as an external validation set. These images were labeled with the same categories as the internal dataset. The OAI dataset contains only the PA, lateral, and HKA views. None of the images in these two sets were seen by the model or the data during training or validation.

Conformal Predictor:

Coverage and efficiency are the primary metrics of conformal prediction. Coverage is analogous to accuracy and is the proportion of prediction sets produced by the model that contain the correct label. Class-conditional coverage is coverage calculated separately for each class label. Efficiency is the proportion of prediction sets of length one. In all metrics, higher values indicate better performance, with 1.0 being perfect, although in some cases efficiency of 1.0 is not ideal if the model makes additional errors instead of producing prediction sets with length greater than one. A market basket analysis–which examines associations between items that appear together in sets–using the apriori algorithm probed for associations between prediction set components.

Object Detection Model:

The final model was tested on the internal held-out test set. The most important benchmark for this model was the detection of each relevant object in the image. Therefore, we report mean average precision (MAP-50) at a 0.50 intersection over the union threshold that was used for precision and recall. This metric prioritizes correct identification of the object’s class over perfect placement of the bounding box.

Code and Environment

All code was written in Python (Version 3.11) using PyTorch (Version 2.0), Monai (Version 1.1), or the SciKit-Learn package (Version 1.3). The code is publicly available: LINK. All model training and testing were performed on a single Nvidia A100 GPU.

RESULTS

Classification Model Performance:

Model performance metrics on the classification task without the conformal predictor are displayed in Table 1. The internal test set contained 1,914 images. The “Left Knee Present” label output had an F1 score of 0.99, corresponding to 18 total errors. The “Right Knee Present” label output had an F1 score of 0.99, corresponding to 23 total errors. The “Left Knee Instrumented” label output had an F1 score of 0.98, corresponding to a total of 39 errors. The “Right Knee Instrumented” label output had an F1 score of 0.98, corresponding to 49 total errors. The “View” label output had an F1 score of 0.98, corresponding to 24 total errors; 13 of those errors were between the AP and PA views, 10 AP images were incorrectly labeled as lateral (N = 1) and HKA (N = 2), and one notch image was incorrectly labeled as AP. Confusion matrices are displayed in Figure 3.

Table 1:

Classification Results

	F1	Precision	Recall	Coverage	Efficiency
Left Knee Present	0.99	0.99	0.99	1.0	0.99
Right Knee Present	0.99	0.99	0.99	1.0	0.99
Left Knee Instrumented	0.98	0.98	0.98	1.0	0.98
Right Knee Instrumented	0.98	0.98	0.98	1.0	0.98
View	0.98	0.98	0.98	0.99	0.99
Domain Detection	0.99	0.99	0.99	-	-

Figure 3:

Confusion matrices for the multilabel classifier for each output label. The model performed slightly better on the presence of each knee label versus the presence of instrumentation label. The majority of errors were due to inconclusive laterality in an image with a single knee.

Classification Model with Conformal Prediction:

Coverage of the four binary label outputs was 1.0, and efficiency was 0.99. The conformal predictor will yield a “size” of two or more predictions when the conformal score for two or more categories exceeds a user-defined threshold that is determined on the calibration set. There were 20, 25, 32, and 35 prediction sets of size two for the Left Knee Present, Right Knee Present, Left Knee Instrumented, and Right Knee Instrumented label outputs, respectively. Coverage of the view label output was 0.99, and conditional coverage was 0.99, 0.98, 1.0, 1.0, 1.0, 0.99 for AP, PA, Lateral, Sunrise, HKA, and Notch views, respectively. Coverage on the external OAI set was 0.99 across all categories.

Prediction Set Analysis:

There were 64 images that produced prediction sets for the view label output that contained two or greater predictions; 83% of those sets contained both AP and PA predictions, and an additional 10% added Notch to those two. Across all categories, 82% of errors made by the base model were corrected by the conformal predictor, including the correct label in the prediction set.

Domain Detection Model:

There were 4,000 images in the validation set for the domain detection model. The best hyperparameter set corresponded to an F1 score of 0.99, with 48 total errors. In the 3,000 image test set, the F1 score was also 0.99, with precision and recall for knee radiographs of 0.99, and 35 total errors. There was no discernible pattern to the errors.

Object Detection Model:

There were 1,316 radiographs in the test set for the object detection model. MAP-50 across all classes was 0.945 and ranged from 0.695 to 1.000 (Table 2). One class had a MAP below 0.80, ACL other fixation devices, not screws or buttons (MAP-50 = 0.695, N = 6). Object detection metrics are displayed in Table 2. Common mistakes were between femoral ORIF nails and ORIF plates and between fusions and static spacers. Performance was more variable in classes with fewer than 200 instances.

Table 2:

Object Detection Results

	MAP50	Precision	Recall	N (Training + Validation)	N (Testing)
Instrumented Region	0.987	0.976	0.986	10057	1517
TKA Femoral Component	0.992	0.987	0.984	6334	1371
TKA Tibial Component	0.992	0.985	0.981	4907	1073
UKA Femoral Component	0.984	0.978	1.000	526	69
UKA Tibial Component	0.982	0.986	0.976	423	55
PFJ	0.992	0.991	0.991	1223	233
ORIF Femoral Plate	0.813	0.836	0.789	211	42
ORIF Femoral Nail	0.885	0.813	0.780	164	43
ORIF Tibial Plate	0.886	0.890	0.740	246	41
ORIF Tibial Nail	0.964	0.960	0.881	95	16
Static Spacer	0.945	0.787	0.853	142	34
Fusion	0.995	0.547	1.000	11	3
Distal Femur Replacement Femoral Component	0.962	0.833	0.954	300	65
Distal Femur Replacement Tibial Component	0.944	0.801	0.923	301	65
ACL Femur Screw	0.981	0.895	1.000	528	108
ACL Femur Button	0.983	0.976	0.947	557	94
ACL Tibial Screw	0.935	0.743	1.000	147	33
ACL Tibial Button	0.991	0.997	0.941	233	45
ACL Femur Other Fixation	0.695	1.000	0	6	2
ACL Tibial Other Fixation	0.992	0.889	1.000	170	32

TKA: Total Knee Arthroplasty, UKA: Unicompartmental Knee Arthroplasty, PFJ: Patellofemoral Joint Arthroplasty, ORIF: Open Reduction and Internal Fixation, ACL: Anterior Cruciate Ligament

DISCUSSION

We developed a pipeline for the automatic characterization of knee radiographs for use in large-scale patient registries. This pipeline includes a domain detection step, a semantic labeling step with uncertainty quantification via conformal prediction, and an implant detection step. Each of these steps performs at > 0.98 F1 score, and the completed pipeline was able to process over one million images in a handful of hours. This pipeline is a foundational effort for performing large-scale cohort-based research with images. Additionally, an algorithm like this integrated into clinical workflows could curate specific studies from a patient for surgeons on demand.

This work follows those by Rouzrokh et al. for the establishment of a radiography registry for the hip joint, and by Bonnin et al., who also examined knee images but took a different approach [1,12]. Bonnin et al. use a series of deep neural networks each trained to perform a specific task in the categorization of an input image. Our work uses a single, multilabel model for classifying image content and a subsequent object detection model for drawing bounding boxes around different hardware types. Additionally, the application of conformal prediction in our study allows for robust estimation of uncertainty, which is critical as a source of trustworthiness and model explainability. Other groups have trained models to detect arthroplasty implants in a classification scheme or an object detection scheme [13-15]. These works present models trained only on AP images, where our study includes all views of the knee. Further, our use of embedded features to perform domain detection as a sort of “semi-self-supervised” learning is a unique application of transfer learning.

As cornerstones of orthopaedic research, registries such as the American Joint Replacement Registry, and Mayo Clinic’s Total Joint Registry, contain data on hundreds of thousands of patients and millions of procedures [16]. Neither of these registries contains images, owing to the additional complexity of including imaging data and the prohibitive cost of cataloging the contents of the images. The DL methods, like those presented in this study, may allow for images to be included in large-scale registries. This could enable population-level research on the most meaningful radiographic outcomes in knee pathology and knee reconstruction.

Most of the errors in view classification were between the AP and PA views (Figure 4). Subsequently, a market basket analysis of the prediction sets produced by the conformal predictor on the view label output demonstrated that the majority of prediction sets with multiple items returned contained both the AP and PA view labels. This demonstrates that these are the images where the view label output had the most uncertainty. Anecdotally, these are also the view class labels where a human observer has the most uncertainty, especially in the absence of radiopaque markers in the imaging field. Some errors due to the laterality of images with a single knee are likewise ameliorated by the conformal predictor. Errors in the object detection task were mostly errors of omission, where the model failed to correctly identify ORIF devices. Notably, in some images, hip ORIF devices were visible. These devices were not annotated, so it is somewhat understandable that the model would have difficulties with omission with knee ORIF devices. Additionally, the model performed poorly on categories with few training instances. Future iterations of the model could incorporate more of those rare categories to boost performance, given the extreme heterogeneity of appearance in some of the rare implants.

Figure 4:

Error samples from the semantic classifier. In the “knee presence” and “knee instrumentation” label outputs, most errors stemmed from indeterminate or obfuscated laterality. For example, in the hip-knee-ankle image, part of the right knee is visible, but not enough to be useful in downstream analysis. The majority of errors in the view label were between AP and PA labels. Anteroposterior (AP), Posteroanterior (PA)

The primary limitation associated with this study is that, except for the external OAI test set, all the data was acquired at one institution, impacting generalizability. There are, however, a large number of imaging sites within this institution, leading to some variability in acquisition. Additionally, some rare knee implant types were not included due to a lack of training data. Also, the external OAI test set does not include any postoperative knee images, so we were not able to use that dataset to test the implant detection model.

CONCLUSION

We present a multilabel classifier with domain detection and an object detection model to characterize knee radiographs according to image content on several useful parameters, including laterality, view, and the presence of a broad array of implants relevant to knee reconstruction. Conformal prediction provides uncertainty estimations, increasing transparency into when the model is unsure of a decision. This pipeline of algorithms can be used to quickly sort and categorize knee images. Employment of these models in large scale registries could open the door to more meaningful population-based research with images.