Inter- and Intraobserver Reliability Between Orthopaedic Surgeons for Reparability of the Anterior Cruciate Ligament Using MRI

Abstract

Background:

Anterior cruciate ligament (ACL) reconstruction is the standard of care for patients after an ACL tear, as poor historical outcomes were observed after primary ACL repair. Certain subgroups of patients, however, have been shown to have outcomes equivalent to reconstruction after undergoing ACL repair and therefore may benefit from the potential advantages offered by avoiding reconstruction. It is important to accurately and consistently identify and indicate these candidates for ACL repair.

Purpose/Hypothesis:

The purpose of this study was to determine the inter- and intraobserver reliability of magnetic resonance imaging (MRI) evaluation for the reparability of ACL tears and to identify imaging factors that may lead to surgeon uncertainty or disagreement in decision making. Our hypothesis was that the orthopaedic surgeons surveyed would not be able to reliably agree on the reparability of an ACL using MRI scans alone.

Study Design:

Cohort study (diagnosis); Level of evidence, 3.

Methods:

We administered 2 surveys to 6 fellowship-trained orthopaedic sports medicine surgeons. Each surgeon reviewed preoperative MRI scans for 20 patients and answered a series of questions, ultimately determining whether they would choose an ACL reconstruction or repair for the patient based on the imaging alone. The same survey was repeated 6 weeks later. Kappa values for inter- and intraobserver reliability of their decision making were then calculated.

Results:

The average kappa for interobserver reliability in the 2 surveys was 0.22, and the average kappa for intraobserver reliability was 0.34. Interobserver reliability among the surgeons in this group was poor to moderate; intraobserver reliability was slightly better. The choice for ACL repair was significantly correlated with proximal tear locations (r = 0.854; P < .001), good-quality ACL tissue remnant (r = 0.929; P < .001), and how many surgeons believed that the tear only involved a single bundle (r = 0.590; P = .006).

Conclusion:

The surgeons surveyed in this study did not consistently agree on candidates for ACL repair using MRI alone.

A tear of the anterior cruciate ligament (ACL) is a common orthopaedic injury, with an incidence of 68.6 per 100,000 in the overall population.¹¹ Primary suture repair of the ACL through an open arthrotomy was the early treatment of choice for these injuries, but it was largely discontinued as a result of poor outcomes including persistent instability, rerupture, and knee stiffness.^{4,7,10,12,13,16} ACL reconstruction has since become the standard of care for patients with complete ACL tears who desire to return to their preinjury athletic levels.

Recent studies have found that some patients may have equivalent outcomes after ACL repair and reconstruction, specifically those with an avulsion or proximal tear of the ACL from the femoral footprint.^1,3,13,18,21 As a result, new arthroscopic ACL repair techniques have been developed, and there is a renewed interest in ACL preservation. Advocates for ACL repair have argued that it has multiple benefits compared with reconstruction. These include preservation of native ACL proprioception, avoidance of the morbidity of autograft harvest and large bone tunnel drilling, and earlier motion with faster recovery.^{2,3,5,8,18,19}

It is important to accurately and consistently identify candidates for ACL repair versus reconstruction, as the 2 procedures require different preoperative preparation for surgeons and different preoperative counseling for their patients. In addition, future comparative studies of ACL repair and reconstruction rely on consistent identification of ideal candidates for repair. While clinical factors (i.e., age and timing of injury) can easily be identified in the office, some of the most important factors to consider when deciding between repair and reconstruction are identified on preoperative magnetic resonance imaging (MRI) scans. These imaging factors include location of the tear within the ligament, full versus partial tear, and the quality of the ACL remnant tissue.¹⁸

Despite the resurgence of ACL repair in this selected population, it is not known if MRI scans can reliably be used to preoperatively determine whether a patient is an optimal candidate for ACL repair. The purpose of this study was to determine the inter- and intraobserver reliability of MRI evaluation for the reparability of ACL tears and to identify imaging factors that may lead to surgeon uncertainty or disagreement in decision making. Our hypothesis was that the orthopaedic surgeons surveyed would not reliably agree on the reparability of an ACL using the MRI scans alone.

Methods

This study was completed through a series of 2 surveys. The surveys were created and distributed using Qualtrics software (SAP SE). Six sports medicine fellowship-trained orthopaedic surgeons, all with an interest in ACL surgery, were identified and agreed to participate in the study. The participants had a wide range of experience (1-27 years in practice), each performing between 30 and 100 ACL reconstructions annually, and all surgeons were performing ACL repairs in select patients at the time of this study. In order to create the survey, we obtained a list of all ACL tears treated at our institution by 1 investigator (C.S.A.)  between January 2016 and March 2017, which included 96 patients in total. This study was approved by our institutional review board.

The patients and imaging were reviewed by 2 authors (F.L.A., M.L.W.) who were not participating in the survey, and 20 patients were selected for evaluation. All MRI scans had been obtained within 2 weeks of injury. The patients were chosen based on the quality of the MRI itself (ie, no motion artifact, appropriate T2 sequences available, and not conducted at an outside imaging center) and the appearance of the ACL. They were selected with the goal of providing a wide range of ACL appearances (including tear location and tissue quality) in the survey. Participants were shown the coronal, sagittal, and axial views of the T2 sequences for each patient. All MRI scans were obtained via 3-T MRI, using our institution’s standard knee MRI protocol (3-mm slice thickness using a T2 fat supression fast-spin echo sequence for the selected imaging), and the images were examined using the Centricity picture archiving and communication system (GE Healthcare).

The surgeons were then asked a series of 7 questions about each patient, ultimately choosing whether they would perform an ACL repair or an ACL reconstruction based on the available imaging (Figure 1). There was no specific prestudy education for participants regarding factors thought to be important for ACL repair.

Figure 1.

Survey questions. ACL, anterior cruciate ligament; MRI, magnetic resonance imaging; PCL, posterior cruciate ligament.

Tear location in our study was described using a classification initially described by Sherman et al¹³ and modified by van der List et al.¹⁷ This classification describes tear location as proximal avulsion with distal remnant length >90% of original ligament length (type 1), proximal with 75% to 90% distal remnant remaining (type 2), intrasubstance with 25% to 75% distal remnant remaining (type 3), distal with 10% to 25% of distal remnant remaining (type 4), and distal avulsion with <10% of distal remnant remaining (type 5).^10,17 For the purposes of our study, types 1 and 2 were considered a proximal tear, types 3 and 4 were considered an intrasubstance tear, and type 5 was considered a distal tear. As discussed in previous studies, this distinction was made to improve the reliability of the classification and because treatment decisions are most likely to differ based on these distinctions.^18,20

The participants were given the survey twice; the first survey was used to calculate interobserver reliability, and the same survey was performed again 6 weeks later to calculate intraobserver reliability. Patients were shown in a different order to minimize recollection of answers from the previous survey. The survey participants were blinded to their own previous responses and those of the other participants.

Statistical Procedures

In order to assess the degree of agreement within and between observers, Cohen kappa values were calculated for questions 1 to 6 for each patient. Descriptive statistics were used for question 7. The classification for interpreting kappa statistics proposed by Landis and Koch⁶ was used to interpret the results of the statistical analysis:

• Kappa <0: No agreement

• 0.01 to 0.20: Slight agreement

• 0.21 to 0.40: Fair agreement

• 0.41 to 0.60: Moderate agreement

• 0.61 to 0.80: Substantial agreement

• 0.81 to 1.00: Almost perfect agreement

The Pearson correlation coefficient was calculated for questions 1 to 5 based on the number of surgeons who chose to perform an ACL repair for each patient in order to determine which factors were most associated with surgeon decision making.

Results

ACL Repair Versus Reconstruction

The kappa value for interobserver reliability when deciding between ACL repair and reconstruction was 0.23 in the first survey and 0.21 in the second survey, for an overall combined kappa of 0.22 (Table 1). The overall intraobserver reliability was 0.34, with a surgeon range of –0.09 to 0.74. When results of the 2 surveys were combined, there were 7 patients for whom all surgeons chose ACL reconstruction. There were no patients for whom all surgeons chose ACL repair, but there were 2 patients for whom 9 of 12 surgeons chose ACL repair. The combined distribution of surgeon decisions by patient is shown in Table 2.

TABLE 1.

Kappa Values for Inter- and Intraobserver Reliability by Survey Question^a

Question	Interobserver Reliability—Survey 1	Interobserver Reliability—Survey 2	Interobserver Reliability—Combined	Average Intraobserver Reliability
1. Tissue quality	0.22	0.11	0.17	0.17
2. Tear location	0.38	0.39	0.39	0.54
3. Single vs double bundle	0.11	0.10	0.11	0.30
4. Characteristic bone edema	0.50	0.48	0.49	0.51
5. ACL scarred to PCL	0.06	0.02	0.04	0.06
6. ACL repair vs reconstruction	0.23	0.21	0.22	0.34

^aACL, Anterior cruciate ligament; PCL, posterior cruciate ligament.

TABLE 2.

Procedure Choice and Confidence by Patient (Repair vs Reconstruction)

Patient No.	No. of Surgeons Choosing ACL Repair (Combined Surveys)	Mean % Confidence	Range % Confidence	SD
1	0/12	10.50	0.00-30.00	12.0
2	9/12	60.75	9.00-90.00	25.5
3	5/12	49.75	0.00-94.00	31.4
4	1/12	15.00	0.00-63.00	20.6
5	7/12	56.17	10.00-93.00	28.6
6	3/12	26.58	0.00-91.00	34.6
7	3/12	33.75	0.00-91.00	26.9
8	0/12	11.08	0.00-48.00	15.6
9	9/12	71.58	30.00-94.00	22.1
10	1/12	26.83	0.00-50.00	18.2
11	0/12	3.417	0.00-18.00	5.8
12	0/12	14.00	0.00-38.00	14.2
13	2/12	19.33	0.00-68.00	22.7
14	4/12	48.42	8.00-91.00	29.5
15	0/12	15.58	0.00-68.00	19.8
16	3/12	25.42	0.00-82.00	29.0
17	7/12	52.00	0.00-91.00	33.2
18	0/12	23.08	0.00-67.00	24.5
19	0/12	23.08	0.00-62.00	17.3
20	7/12	61.42	24.00-94.00	26.2

ACL Tear Characteristics

The kappa value for interobserver reliability when identifying proximal versus intrasubstance tears was 0.38 in the first survey, 0.39 in the second survey, and 0.39 when the survey results were combined. The intraobserver reliability was 0.54. The kappa values for all MRI characteristics (including tissue quality, bone edema, single vs double bundle, and scar of the ACL to the posterior cruciate ligament) appear in Table 1.

When sorted by patient, the number of surgeons who chose to repair the ACL was significantly correlated with how many surgeons identified the tear as proximal within the ligament (r = 0.854; P < .001), how many surgeons identified the ACL remnant tissue as good quality (r = 0.929; P < .001), and how many surgeons believed that the tear was single-bundle (r = 0.590; P = .006). The number of surgeons choosing ACL repair did not correlate with whether they believed that the characteristic bone edema pattern was present (r = –0.054; P = .821) or whether the ACL was scarred to the posterior cruciate ligament (r = –0.059; P = .803).

Confidence in ACL Repair Success

The average surgeon confidence in the expected success of the ACL repair ranged from 3.4% to 71% (Table 2). The number of surgeons who chose ACL repair for each patient was significantly correlated with the average percentage confidence in success for that patient (r = 0.949; P < .001). Surgeon confidence in the success of ACL repair was significantly higher among those who chose ACL repair than in those who chose ACL reconstruction in all but 3 of the patients.

Discussion

ACL reconstruction has been the standard of care for patients with a torn ACL for many years because of poor outcomes after ACL repair. More recently, groups of patients who may have equivalent outcomes with ACL repair have been identified in the literature, such as those with a proximal ACL tear. Surgeon agreement regarding the identification of these characteristics and preoperative decision making is not currently known. This is the first study of inter- and intraobserver reliability in choosing candidates for ACL repair based on MRI evaluation. Our study demonstrated that the surgeons surveyed did not consistently agree on candidates for ACL repair using MRI scans alone.

While the overall agreement for ACL repair versus reconstruction in the 20 patients was low, there was high agreement for ACL reconstruction in many individual patients. As expected, in these patients, surgeons had nearly full agreement that the tissue quality was poor and the tear location was intrasubstance or distal. ACL repair, on the other hand, was not universally agreed upon in any patients, although 75% agreement was observed in 2 patients. Interestingly, of those 2 patients, only 1 patient went on to have an ACL repair. Overall, 3 patients in this cohort underwent ACL repair (patients 5, 9, and 20), each of whom was chosen as a candidate for repair in at least 7 of 12 surveys; however, patients 2 and 17 also had similar or better results, but the treating surgeon chose to perform an ACL reconstruction. The perceived tear location and tissue quality were strongly correlated with choice of ACL repair or reconstruction among the surgeons in this study. Patients more likely to be chosen for repair had high-quality ACL tissue with a proximal or avulsion tear from the femoral insertion, which has been shown to be predictive of choosing ACL repair in another recent study as well.¹⁷ Kappa values for individual MRI characteristics were highest when identifying tear location in our study, although they were lower than previously published reliability data regarding tear location, likely a result of increased numbers of observers.²⁰ However, it is possible that reliability was lower than expected because differentiating between types 2 and 3 tears may be difficult depending on the quality and orientation of the ACL remnant. Not surprisingly, kappa values were low for some of the other MRI characteristics, as they have been low in other previously published studies as well.¹⁵

Patients of particular interest in this study were those in whom surgeons were nearly evenly split about whether ACL repair or reconstruction would be appropriate. In patients who are not perfect candidates for ACL repair, such as those with a proximal tear but stretched or frayed remnant tissue, there is growing interest in other types of ACL preservation, such as ACL repair using a hamstring augment or biologic scaffold.^8,9,14 These procedures have similar advantages to repair by maintaining the ACL remnant but are currently undergoing long-term outcome studies before being widely performed.

This study also reflected surgeon decision making based on preoperative MRI evaluation, but exact tear pattern and tissue quality are definitively assessed at the time of arthroscopy. The decision to perform repair or reconstruction may change intraoperatively, despite surgeon confidence in the appearance of the preoperative imaging, and preoperative planning with backup options remains critical. Finally, other clinical factors, such as patient age, timing of the injury, expectations, and activity level, also influence surgical decision making. This emphasizes the importance of preoperative counseling and discussion with patients regarding the procedure and possible outcomes.

The success of ACL repair surgery relies on selecting candidates for ACL repair who are most likely to have a successful outcome, and the validity of future outcome research also relies on the choice of appropriate patients for ACL repair. The poor reliability among surgeons choosing ACL repair may have been related to the lack of evidence-based research for exact indications and results for repair. Ongoing study of ACL repair and surgeon decision making should include evaluation of reliability when viewing intraoperative arthroscopic imaging, as the final decision regarding procedure choice is made intraoperatively. Most importantly, long-term outcomes of ACL repair using current surgical techniques remain unknown. It is important to obtain long-term follow-up in order to determine surgical outcomes among these patients, as well as to identify additional clinical or radiographic factors that may affect patient outcomes.

There are a few limitations of this study. First, a small group of surgeons was sampled, although the number of observers in this study was greater than that in previous studies of agreement among surgeons and radiologists when viewing the ACL. Second, surgeon decisions are multifactorial and were likely affected by their experience using ACL preservation procedures. All of the surgeons involved in this study had either performed or observed ACL repair and have an interest in the techniques; however, not all had performed repairs on their own patients at the time of the survey. This is likely similar to the experience of most of the sports medicine community so far, and in our view it makes the study more generalizable.

Conclusion

While ACL repair offers a promising alternative to ACL reconstruction for a select group of patients, surgeon decision making based on preoperative MRI evaluation remains a challenge. Tissue quality, proximal tear location, and tear of a single bundle are factors in the decision for ACL repair. This study found that surgeons consistently identify patients who are candidates for ACL reconstruction but have poor agreement regarding patients who may be candidates for ACL repair based on MRI characteristics alone. Preoperative patient counseling is essential when considering ACL repair, and careful intraoperative examination of the ligament is required before ACL repair, regardless of MRI characteristics.