Diagnostic accuracy of candidate MRI knee osteoarthritis definitions versus radiograph in an acute anterior cruciate ligament injury cohort

Abstract

Background:

We tested diagnostic accuracy of previously proposed magnetic resonance imaging (MRI) OA definitions in a cohort following acute anterior cruciate ligament (ACL) injury.

Methods:

We studied participants with posteroanterior and lateral knee radiographs and MRIs 5 years after ACL injury, scored using the Anterior Cruciate Ligament Osteoarthritis Score. Radiographic OA (ROA) was defined using Osteoarthritis Research Society International scoring of osteophytes and joint space narrowing considering medial/lateral tibiofemoral and patellofemoral compartments.

We tested three candidate MRI OA definitions that performed well in an older adult cohort. “MOST simple” required cartilage score ≥2 (range 0–6) and osteophyte score ≥2 (0–7); “MOST optional” included cartilage score ≥2 and osteophyte score ≥2, and either bone marrow lesions (BML) ≥1 (0–3) or synovitis ≥2 (0–3). The third, a Delphi panel definition, included nonzero scores for cartilage, osteophyte, BML, meniscus and other structures. We calculated sensitivity and specificity with 95% confidence intervals for each MRI definition versus ROA.

Results:

We included 113 participants (mean age 26 years, 26% female). At 5 years, 29 (26%) had ROA. “MOST simple” had a sensitivity of 52% (95% CI 33–71%), and specificity of 76% (95% CI 66–85%). Sensitivity and specificities for “MOST optional” were 28% (95% CI 29–67%) and 83% (95% CI 74–91%), respectively. The Delphi panel definition had a sensitivity of 48% (95% CI 29–67%) and specificity of 77% (95% CI 67–86%).

Conclusion:

Simple MRI-based OA definitions requiring at least cartilage damage and an osteophyte have low sensitivity and high specificity in young persons after knee injury.

INTRODUCTION

Studies have reported that 10–90% of individuals develop post-traumatic osteoarthritis (OA) over 10–20 years following acute anterior cruciate ligament (ACL) injury.¹ The wide range is due to variability in reporting of important study variables, including the definitions of the OA outcome, most of which have been based on radiographic definitions of OA. Research is urgently needed to determine effective preventative strategies for OA in young adults following knee injury. For such studies to detect potentially efficacious interventions, a validated measure of the OA outcome that relies on an imaging modality more sensitive than radiographs is needed. Using radiographs to diagnose structural OA is problematic in that radiographs are insensitive to visualize OA structural abnormalities, in particular in earlier stages, and features of structural OA are revealed only on MR images or at arthroscopy. Further, x-rays are often misleading in the depiction of joint pathology. However, despite the limitations of the modality, x-ray is the current reference standard to define structural OA in both clinical trials and observational studies.

Recently, we developed two MRI-based definitions in the MOST cohort that included (i) cartilage damage and osteophytes and (ii) cartilage damage and osteophytes with the optional inclusion of bone marrow lesions [BML] or synovitis.² Cartilage damage and osteophytes were required features as both are hallmark features of OA; MRI is more sensitive to their detection than x-rays. BMLs or synovitis, the presence of which are associated with pain and disease progression, were tested as optional features. We additionally tested a Delphi panel-based MRI definition³ which, while comprehensive in inclusion of multiple structures, had limitations including different definitions for the tibiofemoral (TF) versus patellofemoral (PF) compartments and the possibility of satisfying the OA definition without cartilage damage despite this being a hallmark feature of OA.³ The two MOST definitions had acceptable sensitivity for radiographic OA (ROA) in the range 75–95%. A simple definition consisting of cartilage damage and at least a moderate osteophyte performed similarly to the other definitions. The Delphi panel-based definition performed similarly.

The MOST cohort focuses on middle aged and older persons with or at risk of knee OA. In order to assess the diagnostic accuracy of these MRI-based OA definitions in a younger population at risk for OA, we tested their performance metrics against radiographic OA in a cohort following ACL injury within the Knee Anterior Cruciate Ligament, Nonsurgical versus Surgical Treatment (KANON) study.^4,5 Compared to the MOST cohort of older adults with or at risk for OA in which we developed the candidate OA definitions, their performance for defining OA may differ in this younger cohort, all with prior knee injury.

MATERIALS AND METHODS

Study population.

The KANON study (International Standard Randomized Controlled Trial 84752559) was a randomized controlled trial that included 121 young active adults with acute ACL tear. Participant enrollment and inclusion have been previously described.^4,6 KANON compared structural rehabilitation with early ACL reconstruction versus structural rehabilitation with optional later ACL reconstruction in adults 18–35 years old presenting within 4 weeks of ACL injury.

MRI acquisition and assessment.

In KANON MRI was performed at baseline and 5-year follow-up with a 1.5-T system (Gyroscan Intera; Phillips) with details previously described.⁷ MRI features were scored using Anterior Cruciate Ligament Osteoarthritis Score (ACLOAS)⁸, including abnormalities of cartilage and ligaments, and presence of BMLs, osteophytes, effusion-synovitis, and Hoffa-synovitis. Relevant to the present study, cartilage damage was scored with a range 0–6 with 2 indicating a focal partial thickness defect. Osteophytes were scored with a range 0–7 with 2 indicating a small, beak-like osteophyte and 3 a small or moderate osteophyte. Degenerative BMLs were scored at follow-up with a range 0–3 based on percentage of regional involvement (1 = <33% regional involvement; 2 = 33–66% regional involvement; 3 ≥ 66%). Effusion-synovitis was scored on a range 0–3 based on maximum potential capsular distention (1 = <33%; 2 = 33–66%; 3 ≥ 66%). The scoring was done by the same readers as in the study that initially developed ACLOAS. Each OA feature was scored based on the most severe lesion across all the TF and PF sub-regions. Details regarding ACLOAS scoring are included in Supplemental Table 1. All MRIs were read using the ACLOAS by a musculoskeletal radiologist (Frank W. Roemer) with experience in semiquantitative OA assessment at the time of the study. A subset of cases was evaluated by a second musculoskeletal radiologist (Ali Guermazi); intra- and inter-observer reliability had substantial or perfect agreement for most MRI features when considering all time points (cartilage, kappas 0.64–0.93; degenerative BMLs, 0.83–1.00; osteophytes, 0.68–0.87; effusion-synovitis, 0.85–0.73).⁸

Proposed MRI OA definitions.

We tested the two best performing candidate knee MRI OA definitions developed in MOST and the Delphi panel definition (Table 1). In our prior study, the sensitivity ranged from 75–95% though with an outlier for the definition of cartilage score ≥2 AND osteophyte score ≥3 and either [BML ≥2 or synovitis ≥2] (sensitivity 56.6%). The specificity ranged from 37–89%. We converted cutoffs from the Whole Organ MRI Score (WORMS)⁹, which we used for MOST, to ACLOAS for use in KANON (Supplemental Table 1). The cartilage scoring for ACLOAS is the same as for the WORMS scale and scoring for other features is similar. The main MRI feature we chose not to include in previously developed MRI-based definitions were meniscal tears or extrusion. While degenerative tears are a component of the previously proposed OA definitions, tears also occur as acute injuries in young persons with knee trauma such as ACL injury. In this case, they do not represent OA and could be misleading as to the presence of OA.

Table 1.

Candidate MRI definitions of OA previously defined in the Multicenter Osteoarthritis Study and a previously defined Delphi panel definition

	Cartilage loss		Osteophyte presence		Other features
Candidate definition #1 (MOST simple)	Partial thickness loss; WORMS or ACLOAS ≥2 (required)	and	Small, definite; WORMS or ACLOAS ≥2 (required)		n/a
Candidate definition #2 (MOST optional)	Partial thickness loss; WORMS or ACLOAS ≥2 (required)	and	Small, definite; WORMS or ACLOAS ≥2 (required)	and	BML WORMS or ACLOAS ≥1 and/or Synovitis WORMS or ACLOAS ≥1
Delphi panel definition
Tibiofemoral compartment, 1	Full thickness loss; WORMS or ACLOAS 2.5, 5, or 6 (required)	and	Small, definite; WORMS or ACLOAS ≥2 (required)		n/a
Tibiofemoral compartment, 2	Either full thickness loss; WORMS or ACLOAS 2.5, 5, or 6	or	Small, definite; WORMS or ACLOAS ≥2	and	≥2 other features*
Patellofemoral compartment	Partial or full thickness; WORMS or ACLOAS 2, 2.5, 3, 4, 5, or 6 (required)	and	Small, definite; WORMS or ACLOAS ≥2 (required)		n/a

^*including subchondral BML or cyst, meniscal subluxation, maceration, or degenerative tear, partial thickness cartilage loss, and bone attrition

Abbreviations: MOST, Multicenter Osteoarthritis Study; WORMS, Whole Organ MRI Score; BML, bone marrow lesion; ACLOAS, Anterior Cruciate Ligament Osteoarthritis Score

The first candidate definition (“MOST simple”) was comprised of a cartilage score ≥2 and an osteophyte score ≥2. The second candidate definition (“MOST optional”) was a cartilage score ≥2 and an osteophyte score ≥2, and either BML ≥1 or synovitis ≥1, or both. A knee was considered to meet the third, Delphi panel-based definition if it met either of the two TF definitions or the PF definition as follows: (1) presence of both a definite osteophyte (grade ≥2) and full thickness cartilage loss (grade 2.5, 5, or 6) in the TF compartment; (2) the presence of either a definite osteophyte or full thickness cartilage loss were met, in addition to two other features in the TF compartment (i.e., BML or bone cyst; meniscal subluxation, maceration or degenerative tear; partial thickness cartilage loss; bone attrition) or 3) the PF definition, with the presence of both a definite osteophyte and partial (grade 2, 3, 4) or full thickness cartilage loss (grade 5,6).

Reference standard.

ROA at 5-year follow-up was defined in KANON using Osteoarthritis Research Society International (OARSI) scoring for osteophytes and joint space narrowing.¹⁰ Posteroanterior radiographs with weight bearing, lateral radiographs of the TF compartment and patella skyline (Merchant) view radiographs were obtained at this timepoint. ROA was considered present if any of the following criteria were met in the TF or PF compartment: joint space narrowing grade ≥2 (TF compartment only); the sum of the two marginal osteophyte grades from the same compartment were ≥2; grade 1 joint space narrowing in combination with grade 1 osteophytes in the same compartment (TF compartment only). For the PF compartment, only scoring of osteophytes was available (skyline or Merchant view). This definition for ROA in the TF joint was different from Kellgren and Lawrence grade 2¹¹ which can be dependent on osteophyte score alone. However, the definition of ROA used in KANON is also more liberal compared to the candidate MRI definitions, which require the presence of both cartilage damage and osteophytes.

Analytic methods.

We included MRI data from baseline (for descriptive analyses) and MRI and x-ray data from 5-year follow-up (for assessment of performance metrics for MRI OA definitions). We excluded individuals with missing x-ray or MRI data at baseline and 5-year follow-up, leaving 113 individuals available for our study. Descriptive statistics were performed at baseline, and at 5 years for the proportion of knees with each score for cartilage damage, and for the presence of osteophytes, BMLs, and effusion-synovitis based on ACLOAS at 5 years.

We reported the number of KANON participants fulfilling the three potential MRI definitions at baseline in order to determine whether the potential MRI OA definitions capture individuals with recent ACL tears and concomitant prevalent MRI-defined OA. If the candidate MRI definitions are accurate, then few, if any, young persons with acute ACL injury should meet these definitions at the time of incident injury.

Then, using the three candidate MRI OA definitions, we determined the number and proportion of individuals who fulfilled the ROA definition in at least one of three compartments (hereafter, “ROA”) at 5 years. We calculated the sensitivity, specificity, positive predictive value, negative predictive value, and likelihood ratios¹² with corresponding 95% confidence intervals for each proposed MRI OA definition at 5-year follow-up versus ROA (as defined above). Secondarily, we evaluated the candidate MRI definitions for the TF compartment only.

RESULTS

We included 113 study participants from KANON, with a mean age of 26±5 years, of whom 26% were female. The mean body mass index was 24.0±2.5 kg/m². At baseline none of the participants met any of the MRI definitions. The frequency of individual MRI features by ACLOAS grade at follow-up is reported in Supplemental Table 2. At 5 years, 29 (26%) had ROA, 35 (31%) met the “MOST simple” definition, 22 (19%) met the “MOST optional” definition, while 33 (29%) met the Delphi panel definition of OA. A cross-tabulation of participants meeting each of the 3 candidate MRI definitions versus ROA is shown in Table 2. For example, of the 35 meeting “MOST simple”, 15 had ROA and 20 did not have ROA. Of the 78 that did not meet the same MRI definition, 14 had ROA and 64 did not have ROA.

Table 2.

Cross-tabulation of participants meeting candidate MRI OA definitions compared to radiographic OA (ROA) definition for n=113

	ROA	No ROA
Candidate MRI definition 1
OA on MRI	15	20
No OA on MRI	14	64
Candidate MRI definition 2
OA on MRI	8	14
No OA on MRI	21	70
Delphi panel definition
OA on MRI	14	19
No OA on MRI	15	65

The “MOST simple” definition using only cartilage and osteophyte score had the highest sensitivity of 52% (95% CI 33–71%), while the Delphi panel definition had a similar sensitivity of 48% (95% CI 29–67%) (Figure 1). The “MOST optional” definition which also required either synovitis or BMLs appeared to have a lower sensitivity of 28% (95% CI 13–47%). “MOST optional” had the highest specificity (83%, 95% CI 74–91%) and “MOST simple” the lowest (76%, 95% CI 66–85%). However, in all cases, the sample size was too small to derive conclusive results regarding differences in sensitivities and specificities between MRI definitions. Positive and negative predictive values are provided in Supplemental Table 3. When we limited to TF OA, we found similar sensitivities for these definitions (range 23–54%; Supplemental Table 3). Positive likelihood ratios ranged from 2.13 for the Delphi panel definition to 2.56 for “MOST optional” (Supplemental Table 4).

Figure 1.

Performance metrics for two candidate MRI definitions and Delphi panel definition compared to reference standard of radiographic OA at 5 years in the KANON study.

Negative predictive value (NPV), positive predictive value (PPV), sensitivity, and specificity with 95% confidence intervals are graphed for candidate definition 1 (cartilage damage and osteophyte; CO), candidate definition 2 (cartilage damage and osteophyte, and either bone marrow lesion or synovitis-effusion; CO+BE), and the Delphi panel definition (Delphi).

DISCUSSION

A consensus MRI definition of knee OA is needed for research studies including those focused on the risk of post-traumatic OA following acute ACL injury. We previously tested candidate MRI definitions in a cohort of older adults with or at risk for knee OA and compared their performance against a Delphi panel-based definition³ to determine whether a simpler definition with consistency across compartments would have similar or better performance.² In the current study, we assessed the diagnostic performance of these candidate MRI definitions in a cohort of young adults with acute ACL injuries. All three tested MRI OA definitions performed similarly, though with considerably lower sensitivity and higher specificity for ROA compared to their performance in our prior study.

When the MRI definitions were developed and tested in MOST, the specificities were low, from 35% for the Delphi panel definition to 42.3% for “MOST optional”. Specificities were higher in the current study, from 77–83%, indicating that these MRI definitions have good ability to correctly identify knees without OA. In young adults with clinical symptoms following ACL injury, this is important because optimizing specificity to reduce false positives would be ideal in prospective studies of incident post-traumatic OA, or in clinical trials for prevention.

The unexpected finding of lower sensitivities for our candidate MRI definitions in KANON compared to MOST requires further explanation. While one would generally expect that all knees with ROA should have MRI evidence of OA, given the greater sensitivity of MRI, this was not the case in the current study. From the imaging standpoint, this may be due to false positive joint space narrowing based on positioning errors related to X-ray acquisition, or by the fact that joint space narrowing is explained by meniscal extrusion or resection where no cartilage damage is present.^13,14 For the PF joint, where axial (Merchant) views were available for radiographs in KANON, a knee may fulfill the X-ray definition (based on presence of osteophytes only) but not the MRI definition particularly when there was no cartilage loss on MRI (Supplemental Figures 1 and 2).⁷ In addition, no axial sequence was available in KANON and thus, osteophytes could not be scored on such axial scans but only at two locations in the sagittal sequence, and PF cartilage loss could have been missed. Additionally, the ROA definition used in KANON, because it requires narrowing at least in the TF joint, is more lenient than Kellgren and Lawrence grade 2 which is the usual threshold to define ROA.¹⁰ Specifically, the ROA definition in KANON required either osteophytes (sum grade 2 or higher) or joint space narrowing of grade 2 or higher. The presence of both osteophytes and joint space narrowing are only required together when both are grade 1. The MRI definitions are more stringent than the x-ray definition, requiring the presence of both osteophytes and cartilage damage simultaneously. Differences in patient-level characteristics between the younger adults with ACL injury in KANON and the older adults in MOST should also be considered, particularly with regards to body mass index and physical activity levels, which could contribute to differential joint loading and different bony responses that would be apparent on x-ray.

Limitations of this study must be acknowledged. First, the sample size limits precision in our estimates, and we were unable to make further inference on which MRI definition performed best. Second, we were unable to use a survey-based OA validation criterion unlike in our prior study in MOST.¹⁵ Whether candidate definitions of OA can accurately capture symptomatic knees as well as those with structural changes is important, as OA cannot be diagnosed with structural changes alone. We used OA defined on radiographs as our reference standard despite radiographs having lower sensitivity than MRI for detecting structural changes of knee OA. Third, sensitivities are low for MRI-based definitions. This is likely due to the difference in how MRI and x-ray definitions of OA were constructed. Misclassification of non-OA as OA would lead to bias, the same knowledge gap that we aim to close with a MRI OA definition. Finally, the lower proportion of females in the study also limited sex-stratified assessment, as some studies have suggested a differential risk of post-traumatic OA after ACL injury by sex.¹⁶

Strengths of the study include the use of a cohort of young adults with acute ACL injury with knee MRIs obtained at baseline and 5-year follow-up. Use of this cohort allows us to assess the diagnostic accuracy of the candidate MRI definitions developed in a prior cohort of older adults. These findings would be generalizable to research studies of preventative interventions for post-traumatic OA.

In conclusion, in this study assessing diagnostic accuracy of candidate MRI definitions vs x-ray of knee OA in a cohort of young adults with acute ACL injury, we have found that they have lower sensitivities for ROA compared to MOST, a cohort of older adults. However, the similar performance of all three candidate definitions with each other corroborate our findings in MOST. An MRI definition of knee OA that requires cartilage damage and at least a moderate osteophyte is simple to apply.

SIGNIFICANCE AND INNOVATION.

• There is no widely accepted MRI definition of knee osteoarthritis (OA) despite the widespread use of MRI in OA research.

• In a prior study, we developed candidate MRI-based definitions of OA that included cartilage damage and osteophytes, with the optional inclusion of the presence of either bone marrow lesions or synovitis in a cohort of older adults. We tested the diagnostic accuracy of these two definitions and a third Delphi panel-based definition in a cohort of young adults after acute anterior cruciate ligament tear (KANON).

• In the KANON cohort, the MRI definition of knee OA that requires cartilage damage and at least a moderate osteophyte had low sensitivity and high specificity for identifying radiographic OA.