Machine Learning–Aided Diagnosis Enhances Human Detection of Perilunate Dislocations

Abstract

Background:

Perilunate/lunate injuries are frequently misdiagnosed. We hypothesize that utilization of a machine learning algorithm can improve human detection of perilunate/lunate dislocations.

Methods:

Participants from emergency medicine, hand surgery, and radiology were asked to evaluate 30 lateral wrist radiographs for the presence of a perilunate/lunate dislocation with and without the use of a machine learning algorithm, which was used to label the lunate. Human performance with and without the machine learning tool was evaluated using sensitivity, specificity, accuracy, and F1 score.

Results:

A total of 137 participants were recruited, with 55 respondents from emergency medicine, 33 from radiology, and 49 from hand surgery. Thirty-nine participants were attending physicians or fellows, and 98 were residents. Use of the machine learning tool improved specificity from 88% to 94%, accuracy from 89% to 93%, and F1 score from 0.89 to 0.92. When stratified by training level, attending physicians and fellows had an improvement in specificity from 93% to 97%. For residents, use of the machine learning tool resulted in improved accuracy from 86% to 91% and specificity from 86% to 93%. The performance of surgery and radiology residents improved when assisted by the tool to achieve similar accuracy to attendings, and their assisted diagnostic performance reaches levels similar to that of the fully automated artificial intelligence tool.

Conclusions:

Use of a machine learning tool improves resident accuracy for radiographic detection of perilunate dislocations, and improves specificity for all training levels. This may help to decrease misdiagnosis of perilunate dislocations, particularly when subspecialist evaluation is delayed.

Introduction

Perilunate injuries can cause significant morbidity when untreated, including median nerve dysfunction, complex regional pain syndrome, carpal instability, and chondral damage with progressive arthritis.^1-4 Prompt diagnosis and treatment with reduction and fixation are critical to reduce these complications and optimize outcomes.^1,5 Definitive diagnosis of perilunate and lunate dislocations depends largely on radiographs. ⁶ Whereas posteroanterior and oblique radiographs may demonstrate abnormal carpal arcs and lunate shape, the true lateral radiograph provides the most distinctive finding of disruption of the colinear relationship between the radius, lunate, and capitate. Unfortunately, perilunate/lunate dislocations are frequently misdiagnosed or missed altogether, with approximately 25% of cases missed at the initial presentation despite adequate radiographs.^1,7,8 Initial interpretation of radiographs is often performed by trainees or clinicians without subspecialty training, which may contribute to missed or delayed diagnosis. Improvement in the diagnosis of perilunate and lunate injuries could help decrease morbidity from delay in or lack of treatment.

Artificial intelligence (AI) has undergone significant advances in recent decades, including the evolution of deep learning techniques, and has immense potential to enhance clinical decision-making. Recently, several AI-based software systems have been approved to support physicians’ clinical judgment in diagnosing pathology and determining management. ⁹ Deep learning has been used in radiographic image analysis for a variety of purposes, including in diagnosis of pneumonia, liver masses, and acute neurological events, among others.^10-12 Applications in the hand and upper extremity have thus far been limited largely to fracture detection,^13-24 and in measurements of radiographic bony parameters.^25,26

Our group has previously developed a proof-of-concept AI tool for automated lunate labeling and perilunate injury detection. ²⁷ Here we propose an application of AI to the clinical workflow for integrated performance evaluation. The present study aims to validate the clinical utility of a machine learning lunate labeling tool in assisting detection of perilunate and lunate dislocation injuries by clinicians. We hypothesized that utilization of a machine learning lunate detection tool would improve diagnosis of perilunate/lunate dislocation by less experienced clinicians.

Materials and Methods

Machine Learning Algorithm

A machine learning algorithm was previously developed for automated identification and annotation of the lunate on lateral wrist radiographs, followed by an image classifier to identify presence of perilunate/lunate dislocation using the extracted region (Figure 1). ²⁷ Both perilunate and lunate dislocations were included in model development, without excluding any particular pattern. Suboptimal lateral radiographs with rotation were not excluded, as these often occur in real clinical settings. A neural network object detector was trained to segment the lunate on lateral wrist radiographs. The algorithm then placed an annotation box around its prediction of the lunate on the radiograph image. Next, in the original model, the region around the lunate is extracted and processed by a second neural network which classifies the extracted anatomic region as normal lateral wrist anatomy (lunate-capitate colinear) or pathologic (perilunate/lunate dislocation). In this study, only the object detector tool, which labels the position of the lunate with an automated bounding box on the image, was used to assist clinicians in their diagnosis of perilunate or lunate dislocation injuries. The rationale for the study design was to assess utility of the machine learning algorithm as a diagnostic aid by assisting with lunate position identification, rather than to replace clinician diagnosis by suggesting a classification of normal versus dislocated.

Figure 1.

Outline of machine learning algorithm previously developed for identification and automatic labeling of the lunate on lateral wrist radiographs.

Validation of Machine Learning Tool

A multicenter questionnaire study was performed asking participants from emergency medicine, hand surgery, and radiology to evaluate for presence of a perilunate or lunate dislocation on 30 deidentified lateral wrist radiographs with and without the use of the tool, which labeled the lunate as identified by the algorithm (Figure 2, Supplemental Table 1). Participants were recruited across 3 large academic institutions and included plastic and orthopedic surgery residents, hand surgery–trained fellows and attendings, emergency medicine residents, emergency medicine fellows and attendings, radiology residents, and specialized musculoskeletal radiology fellows and attendings. Recruitment took place through e-mail and flyers, with a reminder e-mail at 2 weeks. Responses were anonymous but participants were asked to state basic demographic information such as their specialty, level of training, and postgraduate year. All questionnaire participants provided informed consent, and consent for use of deidentified radiographs was waived. The study was approved by the Stanford University, Mass General Brigham, and University of Washington Institutional Review Boards.

Figure 2.

(a) Sample lateral wrist radiograph presented to participants for diagnosis of presence or absence of perilunate/lunate dislocation. (b) Sample lateral wrist radiograph with aid of machine learning algorithm to annotate the lunate position with a bounding box.

The participants were informed only after beginning the questionnaire that they would be asked to evaluate lateral wrist radiographs for being either normal or pathologic containing a lunate/perilunate dislocation. All participants were shown the same series of 30 radiographs and asked whether a lunate/perilunate dislocation was present. Then, following an explanation of the lunate object detection tool, they were shown the same 30 radiographs in a different order with the lunate labeled with a bounding box by the machine learning tool. Of the 30 radiographs, 15 were normal and 15 were pathologic (10 perilunate dislocations and 5 lunate dislocations; Supplemental Table 1). Perilunate dislocations consisted of carpal dislocations in which the capitolunate alignment was disrupted but the relationship between the lunate and radius was maintained, whereas lunate dislocations involved disruption of both the capitolunate and radiolunate relationships. A time limit of 15 seconds was set for each image.

Data Analysis

Human performance with and without the machine learning tool was evaluated using sensitivity, specificity, accuracy, and F1 score. The reference standard was defined as the diagnosis agreed on by two hand specialty–trained surgeons and a musculoskeletal-trained radiologist. Results were stratified by level of training and department. For the purposes of stratification, fellows were grouped with attendings because they have completed residency training, act as supervising clinicians, and frequently hold attending privileges to practice independently. A power analysis indicated that a sample size of 34 evaluators would give 0.80 power to detect a mean difference of 5% (SD 10%) in each performance parameter before and after use of the tool. The mean and confidence interval with alpha of 0.05 were calculated for each group. A two-tailed paired student’s t test was conducted to compare results. A P-value of less than .05 was defined as statistically significant.

Results

A total of 137 participants were enrolled across 3 sites. Recruitment e-mails were sent to 826 potential participants, for a 17% response rate achieved. There were 55 respondents from emergency medicine, 33 from radiology, and 49 from hand surgery. Thirty participants were attending physicians, 9 were fellows, and 98 were residents. Of residents, distribution by postgraduate training year (PGY) was as follows: 16% PGY-1, 21% PGY-2, 22% PGY-3, 30% PGY-4, 7% PGY-5, and 3% PGY-6. Specialties represented among residents were 41% in emergency medicine, 22% in radiology, 26% in plastic surgery, and 11% in orthopedic surgery.

Among all participants, use of the machine learning tool improved specificity from 88% to 94% (P < .05), accuracy from 89% to 93% (P < .05), and F1 score from 0.89 to 0.92 (P < .05). Sensitivity remained unchanged at 91% with and without the tool. Performance for each individual radiograph with and without the tool can be found in Supplemental Table 1. Participants appear to perform more poorly on normal radiographs that are rotated from a true lateral view and those with a perilunate dislocation, while performing better on those with a lunate dislocation.

When participants were stratified by training level, attending physicians and fellows had no change in accuracy or F1 score (Table 1). Specificity improved from 93% to 97% (P < .05), while sensitivity decreased from 98% to 96% (P < .05). Of note, however, subspecialty-trained attending physicians and fellows in musculoskeletal radiology and hand surgery did not have any statistically significant changes in performance with the machine learning tool. Emergency medicine attendings and fellows experienced an improvement in specificity (88% to 97%, P < .05), decrease in sensitivity (95% to 89%, P < .05), and no statistically significant change in overall accuracy (92% to 93%, P = .53).

Table 1.

Sensitivity, Specificity, Accuracy, and F1 Score of Perilunate/Lunate Dislocation by Attendings/Fellows and Residents at Baseline and With Utilization of Machine Learning Algorithm.

Group	Sensitivity, %			Specificity, %			Accuracy, %			F1 score
	Baseline (95% CI)	Assisted (95% CI)	P-value	Baseline (95% CI)	Assisted (95% CI)	P-value	Baseline (95% CI)	Assisted (95% CI)	P-value	Baseline (95% CI)	Assisted (95% CI)	P-value
Attendings and Fellows	98 (97-100)	96 (93-99)	<.05	93 (90-95)	97 (95-100)	<.05	95 (94-97)	96 (94-98)	.25	.96 (.94-.97)	.96 (.94-.98)	.41
Residents	88 (84-91)	89 (86-93)	.13	86 (83-88)	93 (91-96)	<.05	86 (83-89)	91 (89-93)	<.05	.86 (.83-.89)	.90 (.87-.93)	<.05
P-value	<.05	<.05		<.05	.06		<.05	<.05		<.05	<.05

Note. Data presented as means and 95% CIs. CI = confidence interval.

Among all residents, use of the machine learning tool resulted in improved specificity from 86% to 93% (P < .05), accuracy from 86% to 91% (P < .05), and F1 score from 0.86 to 0.90 (P < .05). There was no significant difference in sensitivity, at 88% without the tool versus 89% with the tool (P = .13). These changes with assistance of the machine learning tool held true regardless of resident specialty.

Finally, although surgery and radiology residents lagged behind subspecialized attendings and fellows in accuracy (91% vs 97%, P < .05), sensitivity (94% vs 100%, P < .05), and specificity (88% vs 95%, P < .05) at baseline, use of the machine learning tool improved their diagnostic abilities to close the gap in performance when compared with subspecialty attendings and fellows in accuracy (95% vs 98%, P = .06) and specificity (95% vs 97%, P = .33; Table 2). In this subgroup of residents, sensitivity improved from 94% to 96%, which remains inferior compared with attending and fellows’ sensitivity in diagnosis (100%, P < .05).

Table 2.

Sensitivity, Specificity, Accuracy, and F1 Score of Perilunate/Lunate Dislocation by Attendings/Fellows and Residents in Radiology/Hand or Emergency Medicine at Baseline and With Utilization of Machine Learning Algorithm.

Group	Sensitivity, %			Specificity, %			Accuracy, %			F1 score
	Baseline (95% CI)	Assisted (95% CI)	P-value	Baseline (95% CI)	Assisted (95% CI)	P-value	Baseline (95% CI)	Assisted (95% CI)	P-value	Baseline (95% CI)	Assisted (95% CI)	P-value
Attendings and fellows, radiology/hand	100 (100-100)	100 (99-100)	.33	95 (92-98)	97 (93-100)	.18	97 (96-99)	98 (96-100)	.30	.98 (.96-.99)	.99 (.97-1.0)	.16
Residents, radiology/hand	94 (90-97)	96 (93-98)	.09	88 (85-91)	95 (92-97)	<.05	91 (88-94)	95 (93-97)	<.05	.91 (.88-.94)	.95 (.93-.97)	<.05
P-value	<.05	<.05		<.05	.33		<.05	.06		<.05	<.05
Attendings and fellows, emergency medicine	95 (92-98)	89 (82-96)	<.05	88 (84-93)	97 (94-99)	<.05	92 (89-95)	93 (89-96)	.53	.92 (.89-.95)	.92 (.88-.96)	.90
Residents, emergency medicine	79 (72-86)	80 (73-87)	.54	82 (78-86)	91 (87-95)	<.05	79 (74-84)	85 (81-89)	<.05	.79 (.73-.85)	.83 (.78-.89)	<.05
P-value	<.05	.14		.07	.10		<.05	<.05		<.05	.06

Note. Data presented as means and 95% CIs. CI = confidence interval.

Among emergency medicine residents, machine learning–assisted accuracy improved significantly from 79% to 85% (P < .05), and specificity improved from 82% to 91% (P < .05). Despite these improvements, their assisted performance remained less accurate than that of their respective attendings and fellows (85% vs 93%, P < .05).

Baseline and machine learning–assisted clinician performance was then compared with overall performance of the automated image classifier tool alone (Figure 3). As reported previously, ²⁷ at the set classification threshold of 0.5, the automated tool achieves a sensitivity of 94% at a specificity of 93%, with an area under the curve of 0.986. Overall, emergency medicine residents performed below the level of the image classifier algorithm, and attendings/fellows and specialized radiology and hand surgery residents performed at or above the curve. With assistance of the tool, overall performance of emergency medicine residents improves but does not reach performance of the classifier algorithm alone. Assistance with the lunate labeling tool improves the performance of radiology and hand surgery residents as well as emergency medicine attendings; each of these groups on their own performs at the same level as the algorithm but with algorithm assistance performed above the level of the algorithm.

Figure 3.

Unassisted (baseline) and machine learning–assisted clinician performance in the study compared with overall performance of automated image classifier neural network algorithm.

Note. Clinician performance displayed as mean with 95% confidence interval. Dashed line indicates receiver operating characteristic curve of the underlying algorithm as previously published, ²⁷ with an associated area under the curve of 0.986, demonstrating performance of the algorithm across all classification thresholds. “X” marker indicates automated classifier performance at a classification threshold of 0.5, with sensitivity of 93.8% and specificity of 93.3%. All points above the curve represent clinician performance superior to that of the automated classifier model, and points below the curve represent performance inferior to that of the algorithm. EM = emergency medicine.

Discussion

The present study demonstrates that use of a machine learning tool improves diagnostic accuracy and specificity among nonspecialized subgroups of treating physicians in the radiographic detection of perilunate/lunate dislocations. Performance of less experienced trainees and emergency medicine supervising physicians improved significantly when assisted with the machine learning tool. For hand surgery and radiology residents, the machine learning tool improves accuracy and specificity of detection of perilunate/lunate dislocations to mimic that of their respective attending physicians. Not surprisingly, hand and musculoskeletal-trained attending physicians did not benefit from AI assistance. However, we demonstrate proof of concept of the utility of AI assistance for emergency physicians and less experienced surgical and radiology clinicians who lack subspecialty training.

Many clinical settings may lack clinicians with the subspecialized expertise and experience to reliably detect perilunate injuries on radiographs. Frequently in the community setting, radiographs are interpreted by physicians who lack specialized training in wrist pathology, leading to potential missed or delayed diagnosis.^28,29 Clinicians who perform the initial evaluation may include community emergency physicians and can involve orthopedic and plastic surgeons without subspecialty hand training. Although more common pathologies such as extremity fractures are rarely missed, ²⁸ misdiagnosis rates of perilunate/lunate injuries remain unacceptably high at 25%.^1,7,8 The predominant factor identified contributing to misdiagnosis of these injuries is lack of experience of the assessing clinician. ⁷ Therefore, tools to help augment diagnostic performance of these clinicians may help to decrease this initial misdiagnosis rate. Artificial intelligence offers an opportunity to use computer vision and deep learning to detect patterns that can be missed by less experienced clinicians. In the present study, we apply deep learning techniques to demonstrate proof of concept of the utility of AI-assisted diagnosis of perilunate and lunate dislocations, a different radiographic pattern of injury than previously reported applications. Indeed, the present study found that hand surgery and radiology trainees improved when assisted by the machine learning tool, resulting in more comparable performance to their more experienced counterparts. Among hand surgery and radiology residents, machine learning–assisted clinician diagnosis outperforms the same clinicians alone and appears at least on par with AI classifier performance alone. Our findings suggest that use of a machine learning tool to identify the lunate does help less experienced clinicians, as they may have difficulty reading lateral radiographs particularly when not optimally positioned.

Although machine learning assistance improves performance of emergency medicine physicians, and in particular residents, sensitivity of diagnosis among these groups remains low. This reflects a higher missed diagnosis rate, even when assisted by the lunate detector algorithm, among physicians performing the initial evaluation of these relatively uncommon injuries. Our findings on individual images (Supplemental Table 1) show that performance is worse on perilunate dislocations when compared with lunate dislocations (Mayfield stage IV), even with the lunate labeling tool. This suggests that even if clinicians are shown the lunate’s location, they must know to evaluate for capitolunate collinearity to diagnose perilunate dislocations in which the relationship between the lunate and radius is preserved. The automated classifier tool, which achieved a sensitivity of 94% while maintaining 93% specificity as shown in our previous study, ²⁷ could potentially outperform some clinicians even with assistance from the lunate detection algorithm. Therefore, one may consider showing the treating physician not only the bounding box from the lunate detector algorithm but also a preliminary diagnosis of “normal” versus “lunate/perilunate dislocation” as predicted by the automated image classifier algorithm.

Currently, potential applications of AI in the practice of surgery continue to increase. Value may be created from improved accuracy, speed, or cost, and should ultimately lead to improved patient outcomes. ³⁰ Gan et al ¹³ developed a deep learning model to detect distal radius fractures on wrist radiographs, finding superior accuracy, sensitivity, and specificity compared with radiologists and similar performance compared with orthopedists. Chung et al ¹⁶ evaluated a model in detecting and classifying proximal humerus fractures on shoulder radiographs, demonstrating superior accuracy when compared with orthopedists. Yoon et al^17,18 have also demonstrated the ability of a deep learning algorithm to detect occult scaphoid fractures missed by physicians on plain radiographs, which has the potential to reduce misdiagnosis as well as need for advanced imaging. Beyond fracture detection, AI has also been applied to automated segmentation and measurement on radiographs, for example of the second metacarpal cortical percentage. ²⁵ In the field of hand surgery, the application of AI to the diagnosis of perilunate/lunate dislocations is particularly useful due to the high misdiagnosis rate, along with the significant potential morbidity associated with a missed or delayed diagnosis. Although the diagnosis is readily apparent to specialized physicians, these diagnoses can be missed by those without specialty training. Therefore, these injuries in particular pose a clinical problem in which machine learning has the potential to affect patient outcomes as a systems-based tool to decrease human error. Baseline performance even among academic emergency medicine faculty and nonspecialized trainees, although relatively high from an absolute standpoint, still demonstrates an unacceptably high rate of inaccuracy when considering the expected ease of diagnosis.

The present study demonstrates the utility and feasibility of a machine learning algorithm as a diagnostic aid in identifying perilunate/lunate dislocation injuries. Interestingly, it is possible that the combination of clinician diagnosis with AI assistance may result in the best performance when compared with either method alone. We envision that this technology could be implemented in the near future as an automated screening tool to assist clinician interpretation of radiographs in the setting of wrist trauma. We note from subjective feedback that some participants had difficulty initially in interpreting the bounding box tool, which may have contributed to worsened performance despite AI assistance. We anticipate that as familiarity with AI tools increases, AI-assisted performance will continue to improve.

Although an in-depth discussion of the many ethical and legal considerations of AI utilization in health care is beyond the scope of this study, it is important to briefly address the role of AI assistance in clinician decision-making. We found that less experienced clinicians benefit from an automated lunate identification tool to help them make a diagnosis of a perilunate/lunate injury. However, successful use of this tool still requires clinician knowledge and experience, and the clinician makes the final determination. In contrast, overreliance on AI classification tools in some applications can detract from knowledge and judgment development particularly among less experienced clinicians, who may not critically assess AI suggestions or fully understand the principles underlying the diagnosis. The goal of the present study was therefore to use AI to assist clinicians in their diagnosis of perilunate/lunate injuries, rather than to provide them an automated diagnosis. Currently, we envision that AI can be used to provide additional diagnostic information and to augment trainee education, where humans maintain an active role in clinical decision-making and incorporate their expertise.

There are several limitations of the study. Diagnosis of perilunate injuries in the study was performed using only a lateral wrist radiograph, which provides the most clear visualization of perilunate/lunate injuries, while physicians have access to additional views in clinical practice. The response rate, while low, resulted in participants with appropriate distribution in specialty and level of training with a Cronbach’s alpha of greater than 0.7, and provides enough power to detect clinically meaningful differences through the multicenter study structure. Nonetheless, the lower response rate has the potential to introduce nonresponse bias. The study was designed such that participants were shown all images without and then with assistance of the lunate detection tool, a methodology that may introduce bias but is standard when evaluating computer-assisted detection systems. ¹⁴ The study design, which asked participants to identify presence or absence of a perilunate/lunate dislocation, may have increased baseline sensitivity of diagnosis by increasing the index of suspicion. It is possible that this contributed to lesser differences in sensitivity, where we found small but statistically insignificant improvements. The study also included a relatively high prevalence of perilunate/lunate dislocations relative to real-world clinical settings. These two factors likely spuriously improved performance of all participants. In addition, although emergency medicine physicians and nonspecialized trainees in radiology and plastic/orthopedic surgery were included, additional inclusion of community generalist physicians could provide additional evidence for the utility of this tool. These limitations likely lead to an underestimation of the true effect of the machine learning algorithm in decreasing misdiagnosis of perilunate/lunate injuries. Although our participants may be more experienced than other populations of clinicians who could additionally benefit from this tool, and therefore may have smaller differences with AI assistance, we demonstrate baseline misdiagnoses and algorithm-assisted improvements even within our study population. Ultimately, despite these limitations and acknowledging that our demonstrated effect is likely an underestimate, this study demonstrates utility of AI assistance among less experienced clinicians in the diagnosis of perilunate and lunate injuries, which continue to be missed in clinical practice at unacceptably high rates. Future directions may include a larger scale clinical implementation and validation of the machine learning tool, and to develop a more computationally complex model that can incorporate additional radiographic views.

This study demonstrates proof of concept of the clinical application and utility of a machine learning algorithm in the diagnosis of perilunate and lunate dislocation injuries. The machine learning tool has clinical value in improving diagnostic accuracy and specificity for trainees in hand surgery, radiology, and emergency medicine. This may help to improve diagnosis of perilunate/lunate dislocations, particularly by less experienced clinicians without subspecialty training. This technology could ultimately be incorporated into clinical practice as a real-time automated tool to assist in the interpretation of radiographs in the setting of wrist trauma, and it could additionally be used as an educational tool for trainees, who may have difficulty interpreting more challenging radiographic views. This has broad potential applications for machine learning–assisted diagnosis in hand surgery in both large tertiary-care centers with trainees and community locations with limited access to subspecialized clinicians.