The Reliability of Acromioclavicular Joint Dislocation Classification Systems: A Comparison Between the Rockwood and Kraus Classifications

Abstract

Background:

The Rockwood system for the classification of acute acromioclavicular (AC) joint dislocations has been associated with a lack of reliability. A novel system has been proposed (Kraus classification) that is based on dynamic posterior translation of these injuries.

Purpose:

To assess the interobserver and intraobserver reliability of the Rockwood and Kraus classification systems and also to examine the impact of surgeon experience on the assessments.

Study Design:

Cohort study (diagnosis); Level of evidence, 3.

Methods:

Included were 45 patients with acute AC joint injuries who underwent a radiographic examination using standard bilateral AP and modified Alexander views. For interobserver reliability, 6 shoulder surgeons (expert group) and 6 orthopaedic residents (novice group) reviewed the radiographs to classify injuries according to the Rockwood and Kraus classifications; for each categorization, the participants chose between nonoperative management and surgical treatment. The evaluation was repeated 6 weeks later to determine intraobserver reliability. Kappa (κ) coefficients and their 95% CIs were used to compare interobserver and intraobserver reliability for each classification system according to assessor experience. Statistical differences were considered significant when the upper and lower boundaries of the 95% CI did not overlap.

Results:

The overall interobserver agreement for diagnosis (both novice and expert groups) was moderate (κ = 0.52 [95% CI, 0.51-0.54]) for the Rockwood classification and substantial (κ = 0.62 [95% CI, 0.53-0.65]) for the Kraus classification; however, no significant differences were observed between the κ values. The overall interobserver agreement for treatment selection was substantial when using both the Rockwood (κ = 0.78 [95% CI, 0.72-0.81]) and Kraus (κ = 0.77 [95% CI, 0.66-0.87]) classifications. The overall intraobserver agreement for diagnosis was substantial using both the Rockwood (κ = 0.65 [95% CI, 0.61-0.67]) and Kraus (κ = 0.73 [95% CI, 0.69-0.75]) classifications. There were no significant differences between the novice and expert groups on any of the evaluations.

Conclusion:

The Kraus system was slightly more reliable than the Rockwood system for classifying AC joint injuries both between assessor groups and overall. The level of surgeon experience had no impact on the evaluations.

Acromioclavicular (AC) joint injuries are frequent among the young, active population and athletes who participate in contact sports, ¹³ representing approximately 10% of all shoulder injuries. ² The Rockwood classification is the most widely accepted approach to diagnosis and severity, ^11,18,23,30 which identifies 6 types of AC joint injuries. ²² This subdivision is based on certain cut-off points of the coracoclavicular distance (CCD) determined by bilateral standard anteroposterior (AP) and Zanca radiographic views. ²³

Nonoperative management is generally preferred for low-grade injuries (Rockwood types I and II), while surgical treatment is recommended for high-grade symptomatic injuries (Rockwood types IV-VI). ^25,29 However, because type III injuries can respond differently to nonoperative management ¹⁴ and because of the paucity of adequately powered randomized studies, it is difficult to establish the optimal treatment method for these injuries. ^28,29 Furthermore, several studies have questioned the reliability, especially in Rockwood type III, IV, and V injuries, ^4,20 and validity of this classification. ⁶

Vertical displacement of the clavicle has been crucial in treatment selection. ^24,25 However, postoperative residual horizontal instability has been associated with poor clinical results even after the effective surgical treatment of vertical instability. ²⁵ In addition, previous studies have shown less predictable clinical results in patients with low-grade injuries treated nonoperatively. ^14,27 These findings suggest that the severity of low-grade lesions is frequently underestimated ^14,16 potentially because of the inadequate evaluation or management of the horizontal component of the injury. ⁹ However, neither the Rockwood classification ²¹ nor conventional clinical evaluation methodologies adequately identify horizontal displacement. ⁹

Recently, Kraus et al ⁹ proposed a novel AC joint injury classification system based on the evaluation of horizontal and vertical displacement of the clavicle on bilateral Alexander and AP radiographic views (Table 1). However, no additional reliability studies or comparisons with the Rockwood system have been reported. Therefore, this study aimed to evaluate the interobserver and intraobserver reliability of the Kraus and Rockwood classifications for diagnosing and managing acute AC joint injuries as well as the influence of surgeon experience on the strength of agreement. We hypothesized that the reliability of the Kraus classification would be significantly greater than that of the Rockwood system and be unaffected by the surgeon’s experience.

Table 1.

Kraus Classification of Acromioclavicular Joint Injuries ⁹

	Anteroposterior View		Alexander View
Kraus Type	Coracoclavicular Distance Ratio	Vertical Displacement	Dynamic Horizontal Translation
IA	≤30%	Partial vertical displacement	None/partial dynamic horizontal translation
IB	≤30%	Partial vertical displacement	Complete dynamic horizontal translation
IIA	>30%	Complete vertical displacement	None/partial dynamic horizontal translation
IIB	>30%	Complete vertical displacement	Complete dynamic horizontal translation

Methods

Patients and Imaging

This retrospective analysis was authorized by the institutional review board of our hospital. Participants were not required to provide informed consent. A total of 52 consecutive patients were identified at a single center for 3 years (January 1, 2018, to December 31, 2020). Included were patients aged between 18 and 40 years with an acute AC joint injury who had bilateral AP and modified Alexander radiographs taken within 3 weeks after the original trauma. Exclusion criteria were a history of shoulder interventions or shoulder trauma, os acromiale, osteolysis or arthritis of the distal clavicle, a radiographic method that did not meet the protocol, open physes, and fracture sequelae that had not been previously recorded.

At the time of the initial injury, bilateral plain AP radiographs of patients in the standing position were taken. A carefully defined procedure was used for the modified bilateral Alexander view, ¹² which included positioning the patient, a consistent source-to-image distance, and 0° of cephalocaudal tilt (Figure 1). Additional technical factors such as collimation, exposure, and centering point were consistent in this cohort.

Figure 1.

(A) Protocol for the modified Alexander radiographic view. The ipsilateral arm is placed in the cross-body position, increasing the load on the acromioclavicular joint. The patient is placed at a 45° angle to the detector for a Y-view of the shoulder. Subsequently, the scapula is parallel to the sagittal plane of the body and drawn away from the ribs in this posture. The modified Alexander view of the (B) affected (right) shoulder and (C) unaffected (left) shoulder.

Data Collection

Overall, 12 assessors were divided into 2 groups according to their experience level: The expert group included 6 shoulder and elbow specialist surgeons from different institutions, and the novice group included 6 orthopaedic residents (first postgraduate year) from the same institution. All of the shoulder and elbow specialist surgeons had been fellowship trained in shoulder and elbow trauma and orthopaedic abnormalities, had at least 5 years of experience, and had practices involving the surgical management of AC joint injuries.

Digital Imaging and Communications in Medicine (DICOM) files were collected and deidentified by the principal investigator, who was not among the evaluators. Before the assessment, the evaluators received a copy of the original publications outlining the Rockwood ²² and Kraus ⁹ classification systems and a written document with instructions for performing the evaluation.

A self-administered computerized questionnaire was used to collect data. A survey using Google Forms with research questions and answer options was created to collect qualitative and quantitative data. The link to the form and the DICOM files was generated and distributed to all 12 assessors.

Imaging Evaluation

Each assessor performed imaging evaluations using the Horos picture archiving and communication system (Version 3.3.6; Horos Project). The purpose of the evaluators was to classify AC joint injuries using both the Rockwood and Kraus classifications and to select nonoperative management or surgical treatment according to each categorization. For each set of images, the evaluators were instructed to measure the CCD on the AP view using the digital caliper tool of the software. The CCD was defined as the vertical distance between the upper border of the coracoid process and the lower border of the clavicle on the injured and contralateral sides (Figure 2).

Figure 2.

Measurement of the bilateral coracoclavicular distance (CCD). We used the “parallel line tool” of the digital caliper for both classification systems to create 2 lines between the upper edge of the coracoid process and the lower edge of the clavicle on the injured and healthy sides. L, left; R, right.

Subsequently, the CCD ratio ({[CCD injured – CCD healthy]/CCD healthy} × 100) was calculated to (1) classify the injury using the Rockwood system and (2) select a treatment option between nonoperative management and surgical treatment. After the qualitative assessment of the Alexander radiographs, 2 additional inquiries were made: (3) to classify the injury using the Kraus system and (4) to select a treatment option accordingly. The number and order of the set of images were randomly modified in the second stage, and the 12 investigators repeated the analysis 6 weeks later using the same approach to determine intraobserver reliability. This time period was chosen to avoid visual recall by the surgeon.

Statistical Analysis

The sample was described using frequencies for categorical variables. The Fleiss kappa (κ) coefficient was used to calculate interobserver and intraobserver reliability both overall (novice and expert groups) and within each group. ⁵ The strength of agreement was categorized according to Landis and Koch ¹⁰ as follows: 0.00-0.20, slight agreement; 0.21-0.40, fair agreement; 0.41-0.60, moderate agreement; 0.61-0.80, substantial agreement; and 0.81-1.00, almost perfect agreement.

The 95% CI was calculated for significance. Statistical differences between κ values were considered significant when the upper and lower boundaries of the 95% CI did not overlap. Stata (Version 16; StataCorp) was used for statistical analysis.

The sample size was calculated using the proportions and κ values ³ reported in the original article of the Kraus classification. ⁹ With 6 evaluators in each group, an alpha of 0.05, a lower 95% CI limit of 0.6, and an upper 95% CI limit of 0.9, we determined that a minimum of 24 patients would be needed.

Results

Of the initial 52 patients identified, 3 were excluded because they had fracture sequelae in the shoulder girdle, 2 patients with osteolysis of the distal clavicle, and 2 patients with osteoarthritis of the AC joint; thus, 45 patients were included in the analysis. The mean age of the patients was 28 ± 10 years; 32 (71.1%) were men, and 13 (28.9%) were women.

Overall Interobserver Agreement

The overall interobserver agreement for diagnosis (both novice and expert groups) was moderate (κ = 0.52 [95% CI, 0.51-0.54]) for the Rockwood classification and substantial (κ = 0.62 [95% CI, 0.53-0.65]) for the Kraus classification. However, no significant differences were observed between the κ values (Figure 3A). The strength of agreement for treatment selection was substantial for both the Rockwood (κ = 0.78 [95% CI, 0.72-0.81]) and Kraus (κ = 0.77 [95% CI, 0.66-0.87]) classifications (Figure 3B).

Figure 3.

Interobserver reliability of acromioclavicular joint injury classifications for (A) diagnosis and (B) treatment selection. The mean kappa values and their 95% CIs (error bars) are shown according to assessor group and overall.

Interobserver Agreement by Assessor Group

The interobserver agreement was moderate in the novice (κ = 0.51 [95% CI, 0.50-0.54]) and expert (κ = 0.55 [95% CI, 0.51-0.56]) groups using the Rockwood classification for the diagnosis of AC joint injuries. Similarly, there were no statistically significant differences between the novice (κ = 0.72 [95% CI, 0.65-0.84]) and expert (κ = 0.65 [95% CI, 0.53-0.68]) groups for diagnostic agreement using the Kraus classification (Figure 3A).

In the novice group, the interobserver agreement on treatment selection was almost perfect (κ = 0.83 [95% CI, 0.73-0.92]), while the strength of agreement was substantial in the expert group (κ = 0.76 [95% CI, 0.71-0.80]) when using the Rockwood classification. Substantial agreement was found on treatment selection in the novice group (κ = 0.79 [95% CI, 0.77-0.80]) and in the expert group (κ = 0.64 [95% CI, 0.52-0.70]) using the Rockwood classification. No statistical differences were found between the 2 groups, regardless of the classification system used to select the treatment option (Figure 3B).

Interobserver Agreement by Injury Type

Regarding the Rockwood classification, the lowest agreement was observed for type II injuries in the novice (κ = 0.23 [95% CI, 0.15-0.31]) and expert (κ = 0.29 [95% CI, 0.22-0.36]) groups, achieving only fair agreement. In contrast, both the novice (κ = 0.75 [95% CI, 0.68-0.82]) and expert (κ = 0.74 [95% CI, 0.66-0.82]) groups demonstrated substantial interobserver agreement for type V injuries (Table 2).

Table 2.

Interobserver Agreement of Rockwood Injury Type Stratified by Assessor Group

	Novice Group		Expert Group
Rockwood Type	κ (95% CI)	Strength	κ (95% CI)	Strength
I	0.38 (0.30-0.46)	Fair	0.50 (0.42-0.58)	Moderate
II	0.23 (0.15-0.31)	Fair	0.29 (0.22-0.36)	Fair
III	0.56 (0.49-0.63)	Moderate	0.54 (0.46-0.62)	Moderate
V	0.75 (0.68-0.82)	Substantial	0.74 (0.66-0.82)	Substantial

Using the classification proposed by Kraus et al, ⁹ the agreement was slight (κ = 0.06 [95% CI, 0.02-0.14]) in the novice group and fair (κ = 0.25 [95% CI, 0.17-0.33]) in the expert group for categorizing type IB injuries. In contrast, for type IIB injuries, the novice group achieved almost perfect interobserver agreement (κ = 0.84 [95% CI, 0.76-0.92]), while the expert group had substantial agreement (κ = 0.74 [95% CI, 0.66-0.82]) (Table 3).

Table 3.

Interobserver Agreement of Kraus Injury Type Stratified by Assessor Group

	Novice Group		Expert Group
Kraus Type	κ (95% CI)	Strength	κ (95% CI)	Strength
IA	0.77 (0.69-0.85)	Substantial	0.73 (0.65-0.81)	Substantial
IB	0.06 (0.02-0.14)	Slight	0.25 (0.17-0.33)	Fair
IIA	0.52 (0.45-0.59)	Moderate	0.35 (0.27-0.43)	Fair
IIB	0.84 (0.76-0.92)	Almost perfect	0.74 (0.66-0.82)	Substantial

Intraobserver Agreement

The overall intraobserver reliability for diagnosis was substantial using the Rockwood classification (κ = 0.65 [95% CI, 0.61-0.67]) and Kraus classification (κ = 0.73 [95% CI, 0.69-0.75]). Regarding treatment selection, almost perfect agreement was observed for the Rockwood classification (novice group: κ = 0.88 [95% CI, 0.82-0.94]; expert group: κ = 0.82 [95% CI, 0.76-0.90]) and Kraus classification (novice group: κ = 0.87 [95% CI, 0.82-0.93]; expert group: κ = 0.83 [95% CI, 0.76-0.90]). No significant differences were found between the groups (Figure 4).

Figure 4.

Intraobserver reliability of acromioclavicular joint injury classifications for (A) diagnosis and (B) treatment selection. The mean kappa values and their 95% CIs (error bars) are shown according to assessor group and overall.

Discussion

The main finding of this study demonstrates that the interobserver reliability to categorize AC joint injuries using the Rockwood classification was moderate among shoulder-trained subspecialists (κ = 0.55 [95% CI, 0.51-0.56]) and between orthopaedic residents (κ = 0.51 [95% CI, 0.50-0.54]). The Kraus system showed substantial interobserver agreement for the classification of injuries in the expert (κ = 0.65 [95% CI, 0.53-0.68]) and novice (κ = 0.72 [95% CI, 0.65-0.84]) groups, also without significant differences between the groups. In addition, no statistically significant differences were found between the interobserver κ values for treatment selection when comparing the 2 groups of evaluators, regardless of the classification system used.

The Rockwood classification has recently been questioned because of a general lack of agreement. ^4,8,15,20,26 When the reliability of a classification system is tested, the degree of agreement can be influenced by various factors. Expertise and experience have been cited as significant contributors. ¹⁷ When assessing conventional AP shoulder radiographs using the Rockwood classification, an expert group of orthopaedic surgeons demonstrated moderate interobserver agreement (κ = 0.5147), ¹⁷ comparable to our results. Although moderate, orthopaedic surgeons agreed more than radiologists (κ = 0.3628) or emergency doctors (κ = 0.1894). However, the authors did not know whether the poor interdepartmental agreement reflected classification or training deficiencies. ¹⁷

Compared with our data, Cho et al ⁴ reported lower agreement using the Rockwood classification. They found fair interobserver (κ = 0.214) and moderate intraobserver (κ = 0.474) agreement, indicating a general lack of reliability. ⁴ Interestingly, they chose 10 shoulder surgeons with an average of 11 years of practice, as the level of expertise is believed to increase reliability. ⁴ In our study, the differences in experience had no noticeable effect on the results between the groups. The inclusion of a preassessment training strategy in our protocol may be one of the reasons for the better results compared to Cho et al. ⁴ As demonstrated in a previous report, ²⁶ it could reduce experience discrepancies and improve agreement.

Overall, our data were more consistent with Kraeutler et al. ⁸ In their study, 8 orthopaedic surgeons achieved moderate interobserver and intraobserver agreement (κ = 0.602 and 0.496, respectively) for the diagnosis of AC joint injuries using the Rockwood classification. ⁸ However, they reported fair agreement between surgeons on the recommendation for surgical or nonoperative treatment (κ = 0.366). ⁸ On the contrary, we found substantial overall agreement for treatment selection (κ = 0.78 [95% CI, 0.72-0.81]). Interestingly, the agreement achieved by Kraeutler et al ⁸ was reached by visual analysis of AP and axial radiographs without any quantitative evaluation. Other authors have found significant discrepancies in diagnosis between visual analyses and digital measurements between groups of different experience levels. ^15,26 In addition to our protocol’s training strategy, using a digital caliper with a consistent method for measuring the CCD could explain the similar results between groups with different experience levels.

Interestingly, Rockwood type II injuries had one of the lowest agreements in our study. Only fair reliability was observed in the novice (κ = 0.23 [95% CI, 0.15-0.31]) and expert (κ = 0.29 [95% CI, 0.22-0.36]) groups. In a series of 24 patients with acute type I or II AC joint injuries, 7 complained of activity-related discomfort, 8 reported persistent AP instability, and 12 indicated a positive cross-body test finding at 6 years of follow-up. ¹⁴ The severity of sequelae after type I and II injuries is potentially underestimated on plain AP radiographs, as horizontal instability can also occur in low-grade injuries. ^14,27

The Kraus classification incorporates the Alexander radiographic view to identify horizontal instability; however, the strength of agreement for diagnosing low-grade injuries was not significantly greater than with the Rockwood classification. ⁹ In particular, for type IB injuries, the interobserver agreement in the novice and expert groups was slight (κ = 0.06 [95% CI, 0.02-0.14]) and fair (κ = 0.25 [95% CI, 0.17-0.33]), respectively. Furthermore, moderate (κ = 0.52 [95% CI, 0.45-0.59]) and fair (κ = 0.35 [95% CI, 0.27-0.43]) agreement was found for type IIA injuries in the novice and expert groups, respectively. According to our findings, the qualitative evaluation of Alexander radiographs for posterior displacement appears to diminish the consensus in the Kraus classification.

The radiographic diagnosis of posterior displacement of the clavicle can be challenging and inacurate. ^1,19,20 A Rockwood type IV injury is a lesion in which the clavicle rests posterior to the acromion and penetrates the trapezius muscle. ²¹ However, these injuries are relatively rare, and their static condition may not be correlated with posterior displacement in cross-body positions. Rockwood ²² identified them in only 0.7% of cases.

Although the Rockwood classification has been accepted as the gold standard for grading acute AC joint injuries, ²⁶ its usefulness as a decision-making instrument in the management of lesions is uncertain. ⁶ In a retrospective case series of 77 patients with acute injuries ranging from Rockwood type I to V, Granville-Chapman et al ⁶ used the Spearman correlation coefficient (r _S) to determine clinical correlations with the type of injury. A negligible correlation was found with the visual analog scale for pain (r _S = 0.05), the functional elevation deficit (r _S = 0.09), and the Constant score (r _S = 0.01). ⁶ In contrast, Kraus et al ⁹ determined that a precise cut-off value for vertical displacement (CCD ratio >30%) was significantly correlated with clinical presentation. In addition, patients with complete posterior displacement of the clavicle had significantly worse clinical scores (P < .05). ⁹ However, no further validation of this clinical correlation has been performed.

Moreover, low unanimity in diagnosis has been reported using the Rockwood system. In the study published by Ng et al, ¹⁵ there was no single image on which the 15 orthopaedic surgeons agreed on the categorization. Similarly, in different studies, only 2 of the 28 (7.1%) ⁴ and 4 of the 50 (8.0%) ²⁰ patients were universally classified as the same type. In our series, 8 of the 45 (17.8%) patients were unanimously rated by the 12 assessors using the Rockwood classification, while 22 of the 45 (48.9%) patients were classified as the same type using the Kraus classification.

This is the first publication to evaluate the reliability of the classification system proposed by Kraus et al. ⁹ In our study, the level of agreement overall (both novice and expert groups) was substantial for interobserver agreement (κ = 0.62 [95% CI, 0.53-0.65]) and intraobserver agreement (κ = 0.73 [95% CI, 0.69-0.75]). Similarly, the reliability published by Kraus et al ⁹ showed substantial and almost perfect interobserver and intraobserver agreement (κ = 0.80 and 0.82, respectively).

The overall interobserver reliability using the Kraus classification was not substantially higher than that of the Rockwood classification. Therefore, we believe that the Kraus classification may not fully solve the limitation of the Rockwood classification in characterizing some subgroups of lesions, particularly in terms of horizontal instability. Quantitative radiographic measurements of horizontal displacement could help improve the consistency in categorizing these injuries. Previous reports have proposed various strategies for this assessment, ^7,12,31 but the necessary subsequent reliability and validity processes have been limited.

Limitations

This study has several limitations. First, because the research design contained only radiographic measures and qualitative observations, no clinical data were provided. For this reason, we were unable to assess the validity and correlate our results with the true extent of the damage and the clinical presentation. Consequently, the clinical translation of our results is limited. Another limitation is that the axillary view was not available in this study. Although it has been used in some series for the Rockwood classification, the impact of the axillary radiographic view on the reliability of AC joint injury classification systems is uncertain. Previous reports have demonstrated inaccuracy in the assessment of posterior displacement of the clavicle ¹⁹ as well as disagreement about the amount of posterior displacement needed to define a true type IV injury. ²⁰

In the same way, Kraus et al ⁹ proposed a vertical translation assessment using the weighted bilateral AP view. Again, we recognize that every AP radiograph used in the study was taken without loading the arm. However, because the goal was to measure reliability exclusively, no other consideration in terms of validity was taken. Consequently, the model used in this study is not representative of what one might encounter in a clinical setting, but it is still valid to assess the reliability of the classification system.

Moreover, both classifications have conceptual limitations. While the Rockwood classification characterizes AC joint injuries based on soft tissue damage and the Kraus system is based on dynamic posterior translation, we believe that both radiographic approaches are insufficient to confirm these bases. However, determining these assumptions is beyond the scope of this study.

Conclusion

In the current study, we demonstrated that the Kraus classification was more reliable than the Rockwood system for classifying AC joint injuries, although no significant differences were found. The level of surgeon experience had no noticeable impact on the assessments of these 2 classifications.